XML Sitemaps Generator

Author Topic: Not crawling my site  (Read 13364 times)

Renate Schedl

  • Registered Customer
  • Jr. Member
  • *
  • Posts: 13
Not crawling my site
« on: July 15, 2009, 10:50:16 PM »
Have installed the generator to 2 of my domains. Everything is exactly the same. On the one domain it is crawling without any dificulties. On the other it starts crawling and immediately ends with an error mesage, which I cannot read because it is only shown with a galnce. Disappears immediately.

Cannot find any difference in the installation of both sites. Can my domain be the reason for it? Can there be something which does not allow to crawl.

With this domain I as well have problems to get it crawled by Google. It still is not indexed yet. Have included a robot.txt file and google accepts the text of robot.txt.

Anybody here, who has an idea what can be wrong?

XML-Sitemaps Support

  • Administrator
  • Hero Member
  • *****
  • Posts: 10625
Re: Not crawling my site
« Reply #1 on: July 16, 2009, 12:53:13 AM »
Hello,

please try to check your site with our search engine crawler simulator tool: http://www.xml-sitemaps.com/se-bot-simulator.html
Oleg Ignatiuk
www.xml-sitemaps.com
Send me a Private Message

For maximum exposure and traffic for your web site check out our additional SEO Services.

Renate Schedl

  • Registered Customer
  • Jr. Member
  • *
  • Posts: 13
Re: Not crawling my site
« Reply #2 on: July 16, 2009, 01:11:56 PM »
I did, what you suggested. Got the answer:
There was an error while retrieving the URL specified

Now what. How can I find out, what the trouble is?

Renate Schedl

  • Registered Customer
  • Jr. Member
  • *
  • Posts: 13
Re: Not crawling my site
« Reply #3 on: July 16, 2009, 01:32:17 PM »
Can it have anything to do with the "follow" or "non follow". May be I made a mistake with a special plugin. Do not really understand this "follow" stuff.

XML-Sitemaps Support

  • Administrator
  • Hero Member
  • *****
  • Posts: 10625
Re: Not crawling my site
« Reply #4 on: July 17, 2009, 11:03:46 AM »
It might be related, please let me know your website URL so that I can check.
Oleg Ignatiuk
www.xml-sitemaps.com
Send me a Private Message

For maximum exposure and traffic for your web site check out our additional SEO Services.

Renate Schedl

  • Registered Customer
  • Jr. Member
  • *
  • Posts: 13
Re: Not crawling my site
« Reply #5 on: July 18, 2009, 04:18:35 PM »
[external links are visible to admins only]

Renate Schedl

  • Registered Customer
  • Jr. Member
  • *
  • Posts: 13
Re: Not crawling my site
« Reply #6 on: July 18, 2009, 04:23:30 PM »
I found the mistake right now. Had the domain written in the configuration including www. after http://
Now I took the www. out and the site is being crawled.

Thank you for your help.

 

SMF 2.0.12 | SMF © 2014, Simple Machines
XHTML RSS WAP2