XML Sitemaps Generator

Author Topic: How do I get it to do my whole site  (Read 45654 times)

dan

  • Registered Customer
  • Approved member
  • *
  • Posts: 3
How do I get it to do my whole site
« on: July 08, 2005, 06:50:30 AM »
I ahve installed the program and it seems to work to some degree. But it does not cover the whole site. I try to tell it not to look in the cgi-bin but it seems to have a mind of it's own. Hope someone can help.

thanks


Dan

raramuridesign

  • Registered Customer
  • Jr. Member
  • *
  • Posts: 11
Re: How do I get it to do my whole site
« Reply #1 on: July 08, 2005, 07:12:16 AM »
If you use the latest version, you can add a robots.txt file to the root and the crawler should obey what you restict in there...

hope this helps

XML-Sitemaps Support

  • Administrator
  • Hero Member
  • *****
  • Posts: 10624
Re: How do I get it to do my whole site
« Reply #2 on: July 08, 2005, 09:00:52 AM »
Hi,
I want to confirm what raramuridesign just said about robots.txt - the script works as the Google crawler does for your site.
But it's also possible to add special restrictions at the generator configuration page: "Exclude files" textarea input lets you define the exclusion substrings, like "cgi-bin/".
Oleg Ignatiuk
www.xml-sitemaps.com
Send me a Private Message

For maximum exposure and traffic for your web site check out our additional SEO Services.

dan

  • Registered Customer
  • Approved member
  • *
  • Posts: 3
Re: How do I get it to do my whole site
« Reply #3 on: July 08, 2005, 03:21:21 PM »
Hello Again:

Perhaps I didn't explain myself well. I would like to index my whole site but it doesn't. Only 107 of over 1000 files. The fact that it insists on indexing my cgi-bin is an annoyance because the program gives me the impression it can handle that

XML-Sitemaps Support

  • Administrator
  • Hero Member
  • *****
  • Posts: 10624
Re: How do I get it to do my whole site
« Reply #4 on: July 08, 2005, 08:56:59 PM »
Hi,

please let me know the url to your site either here or in PM.
Oleg Ignatiuk
www.xml-sitemaps.com
Send me a Private Message

For maximum exposure and traffic for your web site check out our additional SEO Services.

dan

  • Registered Customer
  • Approved member
  • *
  • Posts: 3
Re: How do I get it to do my whole site
« Reply #5 on: July 08, 2005, 09:32:38 PM »
Hello:

The site is [external links are visible to admins only]

XML-Sitemaps Support

  • Administrator
  • Hero Member
  • *****
  • Posts: 10624
Re: How do I get it to do my whole site
« Reply #6 on: July 08, 2005, 11:09:48 PM »
I tried the generator to crawl your site and seems to work ok. Are you sure you didn't added anything into 'Exclude URLs' field? (in this case the part of urls is not included in sitemap) Otherwise please let me know the url for your generator instance (via PM)  so I can check you confi settings.
Oleg Ignatiuk
www.xml-sitemaps.com
Send me a Private Message

For maximum exposure and traffic for your web site check out our additional SEO Services.

 

SMF 2.0.12 | SMF © 2014, Simple Machines
XHTML RSS WAP2