do parse but do not include url in sitemap
« on: August 24, 2008, 10:14:55 PM »
Hi,

Is it possible to DO SEARCH a certain url for links, include them but not include the searched URL in sitemap?
for example:
1. DO SEARCH (fetch/parse?) example.php?catid=x for links
2. INSERT these found links into sitemap
3. NOT INCLUDE example.php?catid=x in sitemap?

kind regards
Re: do parse but do not include url in sitemap
« Reply #1 on: August 25, 2008, 12:28:42 AM »
Hello,

you can put the following the following in html code of those example.php?catid=x pages (but ONLY on those pages):
Code: [Select]
<meta name="robots" content="follow,noindex">
Re: do parse but do not include url in sitemap
« Reply #2 on: August 25, 2008, 12:42:00 AM »
Hi,

thanx, but these are all dynamic pages with a header template. the site has over 1.000.000 pages and i have already excluded the 'example.php' in robots.txt but then google will list all these pages in the 'error section' which is not the most terrible thing, but it would be nice if pages that are excluded for google via robots weren't in the sitemap, plus it would save some kb's in sitemap.

maybe for future releases; 'do parse but do not include in sitemap' ?

kind regards

ps. i am really satisfied with your software, it crawls these allmost 1.000.000 pages without problems,really cool.
« Last Edit: August 25, 2008, 12:45:24 AM by erdgi »