XML Sitemaps Generator

Author Topic: Unlimited Sitemap Generator Error on Crawling  (Read 6450 times)

mjkang1

  • Registered Customer
  • Approved member
  • *
  • Posts: 1
Unlimited Sitemap Generator Error on Crawling
« on: August 05, 2013, 11:32:53 PM »
Hi,

I tried to run the script to generate an XML sitemap for my site but get the following error,

There was an error while retrieving the URL specified: [external links are visible to admins only]
HTTP Code:

HTTP headers:
x_csize: 0

HTTP output:



currently, my site is blocked by robots.txt because of some reason.
It can be a problem to crawl for XML Sitemap generation?

Please, let me know.

MJ Kang



XML-Sitemaps Support

  • Administrator
  • Hero Member
  • *****
  • Posts: 10625
Re: Unlimited Sitemap Generator Error on Crawling
« Reply #1 on: August 11, 2013, 02:53:27 PM »
Hello,

yes, sitemap generator respects rules defined in robots.txt and will not crawl the site if that is disallowed.
Oleg Ignatiuk
www.xml-sitemaps.com
Send me a Private Message

For maximum exposure and traffic for your web site check out our additional SEO Services.

 

SMF 2.0.12 | SMF © 2014, Simple Machines
XHTML RSS WAP2