Not crawling subdomain properly
« on: August 17, 2009, 07:17:05 PM »
I have the script installed on two sites. The first, MYSITE.com crawls properly, writes properly, pings properly, and Google likes the map (kudos!).

The second, a subdomain, MYSITE2.com has a separate install. I can reach it properly from MYSITE2.COM/THEDIRECTORY, It writes properly (to .../ MYSITE/MYSITE2/MAP) but refuses to crawl beyond the second page, even though it lists the 4 pages it finds, which in turn branch to further pages. This site is _tiny_, only 46 pages, and only to a depth of 3 (index - mainpages (3 of them) - in turn to the rest). The config. looks identical to the main site (except directories, of course). Both are on a dedicated server, no complaints about resources, even on the large MYSITE, and I have tried removing all limitations to crawling depth, memory use, etc.

Any thoughts?
Re: Not crawling subdomain properly
« Reply #2 on: August 18, 2009, 12:42:11 AM »
Oleg:

Under the ahha! category - mixed http:// www. mysite and http:mysite on the index page. Saw the error by looking at the simulator, and now the script runs fine.

Thanks!
Re: Not crawling subdomain properly
« Reply #4 on: November 26, 2009, 01:52:51 AM »
Same problem - crawls just 4 pages

simulator result
[ External links are visible to forum administrators only ]
There was an error while retrieving the URL specified

I have checked robots.txt file, the pages have <meta name="robots" content="index, follow">
 in each - the cart is os commerce set up by crecommerce.com - php.ini allows 98megs memory

pulling my hair out - It was working ok until the hoster siteground started mucking around with it (the shop) and put it a different directory

Same thing happens in pub version of xml sitemaps

thanks for any ideas
peter
Re: Not crawling subdomain properly
« Reply #5 on: November 26, 2009, 08:45:54 PM »

Hello,

I cannot open that URL, looks like your server is currently down.
Re: Not crawling subdomain properly
« Reply #6 on: November 26, 2009, 09:08:02 PM »
[ External links are visible to forum administrators only ] is working ok right now

Peter