Unlimited Sitemap Generator Error on Crawling

[mjkang1]

Hi,

I tried to run the script to generate an XML sitemap for my site but get the following error,

There was an error while retrieving the URL specified: [ External links are visible to forum administrators only ]
HTTP Code:

HTTP headers:
x_csize: 0

HTTP output:


currently, my site is blocked by robots.txt because of some reason.
It can be a problem to crawl for XML Sitemap generation?

Please, let me know.

MJ Kang

[XML-Sitemaps Support]

Hello,

yes, sitemap generator respects rules defined in robots.txt and will not crawl the site if that is disallowed.