XML Sitemaps Generator

Author Topic: Can't start crawling my Drupal 6.x site  (Read 7811 times)

dvanrooy

  • Registered Customer
  • Approved member
  • *
  • Posts: 7
Can't start crawling my Drupal 6.x site
« on: August 23, 2011, 12:58:14 PM »
I use sitemap on another site and all's OK.  I just tried to install and set it up for a new Drupal site.  Immediately after I start the Crawl, I get the message:
Quote
There was an error while retrieving the URL specified: [external links are visible to admins only]
HTTP headers follow:

HTTP output:
where 'mysite.com' is really my real site's home page.  So it doesn't even get past the first page of my site.  I suspect Drupal has it blocked somehow -- or there's something I'm missing.

Can you help?

dvanrooy

  • Registered Customer
  • Approved member
  • *
  • Posts: 7
Re: Can't start crawling my Drupal 6.x site
« Reply #1 on: August 24, 2011, 12:30:33 AM »
 :-[  Duh!  robots.txt as installed by Drupal blocks ALL spiders.  Sorry for the noise.

XML-Sitemaps Support

  • Administrator
  • Hero Member
  • *****
  • Posts: 10624
Re: Can't start crawling my Drupal 6.x site
« Reply #2 on: August 25, 2011, 04:10:01 PM »
I'm glad you found the issue!
Oleg Ignatiuk
www.xml-sitemaps.com
Send me a Private Message

For maximum exposure and traffic for your web site check out our additional SEO Services.

 

SMF 2.0.12 | SMF © 2014, Simple Machines
XHTML RSS WAP2