Google crawler is producing 400 and 404 errors
« on: October 20, 2009, 03:19:15 PM »
I ran xml-sitemap tool to generate sitemap.xml then load it to the root of my website www.bugun.com
I did configured Google+Site Configuration+Sitemaps to use sitemap.xml

Crawler Access is setup as:
User-agent: *
Disallow: /aspnet_client/
Disallow: /Bin/
Disallow: /App_Code/
Disallow: /Common/
Allow: /*
Sitemap: http://www.bugun.com/ror.xml
Sitemap: http://www.bugun.com/sitemap.html
Sitemap: http://www.bugun.com/sitemap.xml
Sitemap: http://www.bugun.com/sitemap.xml.gz
Sitemap: http://www.bugun.com/urllist.txt

I m using Google's recommended crawle rate

There are 4 HTML (400 errors), 8 In Sitemap (404 errors), 14 Not Found (404 errors)

Thank you in advance for your help.
Re: Google crawler is producing 400 and 404 errors
« Reply #1 on: October 20, 2009, 10:19:59 PM »
I ran xml-sitemap tool to generate sitemap.xml then load it to the root of my website www.bugun.com
I did configured Google+Site Configuration+Sitemaps to use sitemap.xml

Crawler Access is setup as:
User-agent: *
Disallow: /aspnet_client/
Disallow: /Bin/
Disallow: /App_Code/
Disallow: /Common/
Allow: /*
Sitemap: http://www.bugun.com/ror.xml
Sitemap: http://www.bugun.com/sitemap.html
Sitemap: http://www.bugun.com/sitemap.xml
Sitemap: http://www.bugun.com/sitemap.xml.gz
Sitemap: http://www.bugun.com/urllist.txt

I m using Google's recommended crawle rate

There are 4 HTML (400 errors), 8 In Sitemap (404 errors), 14 Not Found (404 errors)

Thank you in advance for your help.

Re: Google crawler is producing 400 and 404 errors
« Reply #2 on: October 20, 2009, 11:21:23 PM »
Hello,

you should submit only sitemap.xml to Google.
Oleg Ignatiuk
www.xml-sitemaps.com
Send me a Private Message

For maximum exposure and traffic for your web site check out our additional SEO Services.
Re: Google crawler is producing 400 and 404 errors
« Reply #3 on: October 23, 2009, 02:45:54 PM »
As you suggessted, I changed it to have only sitemap.xml as shown below.  But nothing has changed. Still, there are 4 HTML (400 errors), 8 In Sitemap (404 errors), 14 Not Found (404 errors).  And, all of these error links are working when I click on them.

User-agent: *
Disallow: /aspnet_client/
Disallow: /Bin/
Disallow: /App_Code/
Disallow: /Common/
Allow: /*
Sitemap: http://www.bugun.com/sitemap.xml

Thank you for all your help.
Re: Google crawler is producing 400 and 404 errors
« Reply #4 on: October 25, 2009, 08:04:31 AM »
Hello,

make sure that you check errors in webmaster account->Site configuration->Sitemaps->Click opn sitemap name. Issues reported in other areas of google webmaster account are not related to sitemap files.
Oleg Ignatiuk
www.xml-sitemaps.com
Send me a Private Message

For maximum exposure and traffic for your web site check out our additional SEO Services.

DenT44

*
Re: Google crawler is producing 400 and 404 errors
« Reply #5 on: November 06, 2009, 12:24:05 AM »
Yeah, this could be totally unrelated. Funny, I sometimes get 404 errors from mozilla bot.

landdesignguy

*
Re: Google crawler is producing 400 and 404 errors
« Reply #6 on: November 06, 2009, 01:11:38 AM »
lots of error it seems