• Welcome to Sitemap Generator Forum.
 

robots.txt 'Allow:' command

Started by daustin, February 04, 2013, 01:47:47 PM

Previous topic - Next topic

daustin

Hi, does Standalone Sitemap Generator respect the robots.txt 'Allow:' command for bots like google [ External links are visible to forum administrators only ]

I have an image folder under a disallowed directory but have used the 'Allow:' command to allow it. Not sure if i have done this correct but no images are included in my images site map.

Disallow: /components/
Allow: /components/product/images/
Disallow: /components/product/images/thumb/
Disallow: /includes/