XML Sitemaps Generator

Author Topic: Unlimited Sitemap Generator Error on Crawling  (Read 7772 times)


  • Registered Customer
  • Approved member
  • *
  • Posts: 1
Unlimited Sitemap Generator Error on Crawling
« on: August 06, 2013, 12:32:53 AM »

I tried to run the script to generate an XML sitemap for my site but get the following error,

There was an error while retrieving the URL specified: [external links are visible to admins only]
HTTP Code:

HTTP headers:
x_csize: 0

HTTP output:

currently, my site is blocked by robots.txt because of some reason.
It can be a problem to crawl for XML Sitemap generation?

Please, let me know.

MJ Kang

XML-Sitemaps Support

  • Administrator
  • Hero Member
  • *****
  • Posts: 10904
Re: Unlimited Sitemap Generator Error on Crawling
« Reply #1 on: August 11, 2013, 03:53:27 PM »

yes, sitemap generator respects rules defined in robots.txt and will not crawl the site if that is disallowed.
Oleg Ignatiuk
Send me a Private Message

For maximum exposure and traffic for your web site check out our additional SEO Services.


SMF 2.0.14 | SMF © 2017, Simple Machines