dan

*
  • *
  • 3
How do I get it to do my whole site
« on: July 08, 2005, 07:50:30 AM »
I ahve installed the program and it seems to work to some degree. But it does not cover the whole site. I try to tell it not to look in the cgi-bin but it seems to have a mind of it's own. Hope someone can help.

thanks


Dan
Re: How do I get it to do my whole site
« Reply #1 on: July 08, 2005, 08:12:16 AM »
If you use the latest version, you can add a robots.txt file to the root and the crawler should obey what you restict in there...

hope this helps
Re: How do I get it to do my whole site
« Reply #2 on: July 08, 2005, 10:00:52 AM »
Hi,
I want to confirm what raramuridesign just said about robots.txt - the script works as the Google crawler does for your site.
But it's also possible to add special restrictions at the generator configuration page: "Exclude files" textarea input lets you define the exclusion substrings, like "cgi-bin/".

dan

*
  • *
  • 3
Re: How do I get it to do my whole site
« Reply #3 on: July 08, 2005, 04:21:21 PM »
Hello Again:

Perhaps I didn't explain myself well. I would like to index my whole site but it doesn't. Only 107 of over 1000 files. The fact that it insists on indexing my cgi-bin is an annoyance because the program gives me the impression it can handle that

dan

*
  • *
  • 3
Re: How do I get it to do my whole site
« Reply #5 on: July 08, 2005, 10:32:38 PM »
Hello:

The site is [ External links are visible to forum administrators only ]
Re: How do I get it to do my whole site
« Reply #6 on: July 09, 2005, 12:09:48 AM »
I tried the generator to crawl your site and seems to work ok. Are you sure you didn't added anything into 'Exclude URLs' field? (in this case the part of urls is not included in sitemap) Otherwise please let me know the url for your generator instance (via PM)  so I can check you confi settings.