XML Sitemaps Generator

Author Topic: do parse but do not include url in sitemap  (Read 15350 times)

erdgi

  • Registered Customer
  • Approved member
  • *
  • Posts: 2
do parse but do not include url in sitemap
« on: August 24, 2008, 09:14:55 PM »
Hi,

Is it possible to DO SEARCH a certain url for links, include them but not include the searched URL in sitemap?
for example:
1. DO SEARCH (fetch/parse?) example.php?catid=x for links
2. INSERT these found links into sitemap
3. NOT INCLUDE example.php?catid=x in sitemap?

kind regards

XML-Sitemaps Support

  • Administrator
  • Hero Member
  • *****
  • Posts: 10624
Re: do parse but do not include url in sitemap
« Reply #1 on: August 24, 2008, 11:28:42 PM »
Hello,

you can put the following the following in html code of those example.php?catid=x pages (but ONLY on those pages):
Code: [Select]
<meta name="robots" content="follow,noindex">
Oleg Ignatiuk
www.xml-sitemaps.com
Send me a Private Message

For maximum exposure and traffic for your web site check out our additional SEO Services.

erdgi

  • Registered Customer
  • Approved member
  • *
  • Posts: 2
Re: do parse but do not include url in sitemap
« Reply #2 on: August 24, 2008, 11:42:00 PM »
Hi,

thanx, but these are all dynamic pages with a header template. the site has over 1.000.000 pages and i have already excluded the 'example.php' in robots.txt but then google will list all these pages in the 'error section' which is not the most terrible thing, but it would be nice if pages that are excluded for google via robots weren't in the sitemap, plus it would save some kb's in sitemap.

maybe for future releases; 'do parse but do not include in sitemap' ?

kind regards

ps. i am really satisfied with your software, it crawls these allmost 1.000.000 pages without problems,really cool.
« Last Edit: August 24, 2008, 11:45:24 PM by erdgi »

 

SMF 2.0.12 | SMF © 2014, Simple Machines
XHTML RSS WAP2