Not crawling my site
« on: July 15, 2009, 11:50:16 PM »
Have installed the generator to 2 of my domains. Everything is exactly the same. On the one domain it is crawling without any dificulties. On the other it starts crawling and immediately ends with an error mesage, which I cannot read because it is only shown with a galnce. Disappears immediately.

Cannot find any difference in the installation of both sites. Can my domain be the reason for it? Can there be something which does not allow to crawl.

With this domain I as well have problems to get it crawled by Google. It still is not indexed yet. Have included a robot.txt file and google accepts the text of robot.txt.

Anybody here, who has an idea what can be wrong?
Re: Not crawling my site
« Reply #2 on: July 16, 2009, 02:11:56 PM »
I did, what you suggested. Got the answer:
There was an error while retrieving the URL specified

Now what. How can I find out, what the trouble is?
Re: Not crawling my site
« Reply #3 on: July 16, 2009, 02:32:17 PM »
Can it have anything to do with the "follow" or "non follow". May be I made a mistake with a special plugin. Do not really understand this "follow" stuff.
Re: Not crawling my site
« Reply #4 on: July 17, 2009, 12:03:46 PM »
It might be related, please let me know your website URL so that I can check.
Re: Not crawling my site
« Reply #5 on: July 18, 2009, 05:18:35 PM »
[ External links are visible to forum administrators only ]
Re: Not crawling my site
« Reply #6 on: July 18, 2009, 05:23:30 PM »
I found the mistake right now. Had the domain written in the configuration including www. after http://
Now I took the www. out and the site is being crawled.

Thank you for your help.