XML Sitemaps Generator

Author Topic: Not crawling subdomain properly  (Read 10103 times)

jim33

  • Registered Customer
  • Approved member
  • *
  • Posts: 2
Not crawling subdomain properly
« on: August 17, 2009, 06:17:05 PM »
I have the script installed on two sites. The first, MYSITE.com crawls properly, writes properly, pings properly, and Google likes the map (kudos!).

The second, a subdomain, MYSITE2.com has a separate install. I can reach it properly from MYSITE2.COM/THEDIRECTORY, It writes properly (to .../ MYSITE/MYSITE2/MAP) but refuses to crawl beyond the second page, even though it lists the 4 pages it finds, which in turn branch to further pages. This site is _tiny_, only 46 pages, and only to a depth of 3 (index - mainpages (3 of them) - in turn to the rest). The config. looks identical to the main site (except directories, of course). Both are on a dedicated server, no complaints about resources, even on the large MYSITE, and I have tried removing all limitations to crawling depth, memory use, etc.

Any thoughts?

XML-Sitemaps Support

  • Administrator
  • Hero Member
  • *****
  • Posts: 10623
Re: Not crawling subdomain properly
« Reply #1 on: August 17, 2009, 09:44:30 PM »
Please check with our SE bot simulator tool that your subdomain site is crawlable: http://www.xml-sitemaps.com/se-bot-simulator.html
Oleg Ignatiuk
www.xml-sitemaps.com
Send me a Private Message

For maximum exposure and traffic for your web site check out our additional SEO Services.

jim33

  • Registered Customer
  • Approved member
  • *
  • Posts: 2
Re: Not crawling subdomain properly
« Reply #2 on: August 17, 2009, 11:42:11 PM »
Oleg:

Under the ahha! category - mixed http:// www. mysite and http:mysite on the index page. Saw the error by looking at the simulator, and now the script runs fine.

Thanks!

XML-Sitemaps Support

  • Administrator
  • Hero Member
  • *****
  • Posts: 10623
Re: Not crawling subdomain properly
« Reply #3 on: August 18, 2009, 10:07:14 PM »
Great! Glad that helped :)
Oleg Ignatiuk
www.xml-sitemaps.com
Send me a Private Message

For maximum exposure and traffic for your web site check out our additional SEO Services.

peter2

  • Registered Customer
  • Jr. Member
  • *
  • Posts: 10
Re: Not crawling subdomain properly
« Reply #4 on: November 26, 2009, 01:52:51 AM »
Same problem - crawls just 4 pages

simulator result
[external links are visible to admins only]
There was an error while retrieving the URL specified

I have checked robots.txt file, the pages have <meta name="robots" content="index, follow">
 in each - the cart is os commerce set up by crecommerce.com - php.ini allows 98megs memory

pulling my hair out - It was working ok until the hoster siteground started mucking around with it (the shop) and put it a different directory

Same thing happens in pub version of xml sitemaps

thanks for any ideas
peter

XML-Sitemaps Support

  • Administrator
  • Hero Member
  • *****
  • Posts: 10623
Re: Not crawling subdomain properly
« Reply #5 on: November 26, 2009, 08:45:54 PM »

Hello,

I cannot open that URL, looks like your server is currently down.
Oleg Ignatiuk
www.xml-sitemaps.com
Send me a Private Message

For maximum exposure and traffic for your web site check out our additional SEO Services.

peter2

  • Registered Customer
  • Jr. Member
  • *
  • Posts: 10
Re: Not crawling subdomain properly
« Reply #6 on: November 26, 2009, 09:08:02 PM »
[external links are visible to admins only] is working ok right now

Peter

 

SMF 2.0.12 | SMF © 2014, Simple Machines
XHTML RSS WAP2