Sitemap Generator Forum

Standalone XML Sitemap Generator v1.02 has been released!

2005, July 05

ChangeLog

• robots.txt protocol is supported ('*' and 'googlebot' user-agents are checked)
• view current process state when crawler is in background mode
• interrupt the crawler in background mode
• show real current path on the configuration page
• show sitemap summary block at the main (configuration) page
• check if sitemap exists on the analyze page to avoid warning messages
• split sitemaps on part per 49,999 URLs for consistency (instead of 50,000)
• redirections to external domains are not followed anymore

I from legiraffe.it

Well good work for the upgrade 1.06. The problem with ' in the URLs a remember.... But

I've read that the Standalone PHP Sitemap Generator support the Robots protocol.

For example look at[ External links are visible to forum administrators only ]. For my opinion it work good and Google, yahoo and msn spider respect it.

But the crawler of Standalone PHP Sitemap Generator in my case do not respect the robots.txt.

Why? Is it not well formatted for you.

Sorry for my bad english and best regards

Hi,

your robots.txt looks ok. Do you have some disallowed URLs crawled by generator script?
If so, please PM me your generator instance URL so I can check that.
Thanks!