Crawls half the URLs on the page
« on: May 11, 2012, 03:42:14 PM »
The version I purchased crawls only half the URLs on the page, but the online version crawls all the URLS.

The URLs that it crawls are like [ External links are visible to forum administrators only ] and the ones that it does not crawl are like [ External links are visible to forum administrators only ]e where the only difference is that the ones not crawled have an additional /Some-name-here.

For XML-sitemaps admin: This is one of the URL where the crawling problem occurs: [ External links are visible to forum administrators only ] . I set the URL to start crawling from to be [ External links are visible to forum administrators only ].

How can I fix this? I will PM you the login details
« Last Edit: May 11, 2012, 03:44:33 PM by decemberflow »
Re: Crawls half the URLs on the page
« Reply #2 on: May 14, 2012, 10:11:12 PM »
Thanks, its crawling those now. Can you explain to me why adding the substring like the way you did will cause it to crawl these pages?

I also notice that those pages (../view/some-number/some-string) that were not crawled and are not being crawled appears in the generated sitemap after all those with only (../view/some-number) have been listed. Just wondering why both of them are not listed one after the other, like (../view/some-number-1/some-string1) with (../view/some-number-2/some-string2).