How do I get it to do my whole site

[dan]

I ahve installed the program and it seems to work to some degree. But it does not cover the whole site. I try to tell it not to look in the cgi-bin but it seems to have a mind of it's own. Hope someone can help.

thanks


Dan

[raramuridesign]

If you use the latest version, you can add a robots.txt file to the root and the crawler should obey what you restict in there...

hope this helps

[admin]

Hi,
I want to confirm what raramuridesign just said about robots.txt - the script works as the Google crawler does for your site.
But it's also possible to add special restrictions at the generator configuration page: "Exclude files" textarea input lets you define the exclusion substrings, like "cgi-bin/".

[dan]

Hello Again:

Perhaps I didn't explain myself well. I would like to index my whole site but it doesn't. Only 107 of over 1000 files. The fact that it insists on indexing my cgi-bin is an annoyance because the program gives me the impression it can handle that

[admin]

Hi,

please let me know the url to your site either here or in PM.

[dan]

Hello:

The site is [external links are visible to admins only]

[admin]

I tried the generator to crawl your site and seems to work ok. Are you sure you didn't added anything into 'Exclude URLs' field? (in this case the part of urls is not included in sitemap) Otherwise please let me know the url for your generator instance (via PM)  so I can check you confi settings.