Can't start crawling my Drupal 6.x site
« on: August 23, 2011, 01:58:14 PM »
I use sitemap on another site and all's OK.  I just tried to install and set it up for a new Drupal site.  Immediately after I start the Crawl, I get the message:
Quote
There was an error while retrieving the URL specified: [ External links are visible to forum administrators only ]
HTTP headers follow:

HTTP output:
where 'mysite.com' is really my real site's home page.  So it doesn't even get past the first page of my site.  I suspect Drupal has it blocked somehow -- or there's something I'm missing.

Can you help?
Re: Can't start crawling my Drupal 6.x site
« Reply #1 on: August 24, 2011, 01:30:33 AM »
 :-[  Duh!  robots.txt as installed by Drupal blocks ALL spiders.  Sorry for the noise.