Crawls half the URLs on the page
« on: May 11, 2012, 03:42:14 PM »
The version I purchased crawls only half the URLs on the page, but the online version crawls all the URLS.

The URLs that it crawls are like http://www.domain.com/view/12345 and the ones that it does not crawl are like http://www.domain.com/view/12345/Some-name-here where the only difference is that the ones not crawled have an additional /Some-name-here.

For XML-sitemaps admin: This is one of the URL where the crawling problem occurs: http://www.apadment.com/apartments/rental/MA/boston . I set the URL to start crawling from to be http://www.apadment.com.

How can I fix this? I will PM you the login details
« Last Edit: May 11, 2012, 03:44:33 PM by decemberflow »
Re: Crawls half the URLs on the page
« Reply #1 on: May 14, 2012, 11:12:38 AM »
Hello,

I've updated your crawler settings, please check now.
Oleg Ignatiuk
www.xml-sitemaps.com
Send me a Private Message

For maximum exposure and traffic for your web site check out our additional SEO Services.
Re: Crawls half the URLs on the page
« Reply #2 on: May 14, 2012, 10:11:12 PM »
Thanks, its crawling those now. Can you explain to me why adding the substring like the way you did will cause it to crawl these pages?

I also notice that those pages (../view/some-number/some-string) that were not crawled and are not being crawled appears in the generated sitemap after all those with only (../view/some-number) have been listed. Just wondering why both of them are not listed one after the other, like (../view/some-number-1/some-string1) with (../view/some-number-2/some-string2).
Re: Crawls half the URLs on the page
« Reply #3 on: May 15, 2012, 09:26:31 PM »
They are added directly in sitemap if specified in "Do not parse" setting.
Oleg Ignatiuk
www.xml-sitemaps.com
Send me a Private Message

For maximum exposure and traffic for your web site check out our additional SEO Services.