Unlimited Sitemap Generator Error on Crawling
« on: August 06, 2013, 12:32:53 AM »

I tried to run the script to generate an XML sitemap for my site but get the following error,

There was an error while retrieving the URL specified: [ External links are visible to forum administrators only ]
HTTP Code:

HTTP headers:
x_csize: 0

HTTP output:

currently, my site is blocked by robots.txt because of some reason.
It can be a problem to crawl for XML Sitemap generation?

Please, let me know.

MJ Kang

Re: Unlimited Sitemap Generator Error on Crawling
« Reply #1 on: August 11, 2013, 03:53:27 PM »

yes, sitemap generator respects rules defined in robots.txt and will not crawl the site if that is disallowed.