XML Sitemaps Generator

Author Topic: Crawls half the URLs on the page  (Read 7470 times)

decemberflow

  • Registered Customer
  • Approved member
  • *
  • Posts: 5
Crawls half the URLs on the page
« on: May 11, 2012, 02:42:14 PM »
The version I purchased crawls only half the URLs on the page, but the online version crawls all the URLS.

The URLs that it crawls are like [external links are visible to admins only] and the ones that it does not crawl are like [external links are visible to admins only]e where the only difference is that the ones not crawled have an additional /Some-name-here.

For XML-sitemaps admin: This is one of the URL where the crawling problem occurs: [external links are visible to admins only] . I set the URL to start crawling from to be [external links are visible to admins only].

How can I fix this? I will PM you the login details
« Last Edit: May 11, 2012, 02:44:33 PM by decemberflow »

XML-Sitemaps Support

  • Administrator
  • Hero Member
  • *****
  • Posts: 10621
Re: Crawls half the URLs on the page
« Reply #1 on: May 14, 2012, 10:12:38 AM »
Hello,

I've updated your crawler settings, please check now.
Oleg Ignatiuk
www.xml-sitemaps.com
Send me a Private Message

For maximum exposure and traffic for your web site check out our additional SEO Services.

decemberflow

  • Registered Customer
  • Approved member
  • *
  • Posts: 5
Re: Crawls half the URLs on the page
« Reply #2 on: May 14, 2012, 09:11:12 PM »
Thanks, its crawling those now. Can you explain to me why adding the substring like the way you did will cause it to crawl these pages?

I also notice that those pages (../view/some-number/some-string) that were not crawled and are not being crawled appears in the generated sitemap after all those with only (../view/some-number) have been listed. Just wondering why both of them are not listed one after the other, like (../view/some-number-1/some-string1) with (../view/some-number-2/some-string2).

XML-Sitemaps Support

  • Administrator
  • Hero Member
  • *****
  • Posts: 10621
Re: Crawls half the URLs on the page
« Reply #3 on: May 15, 2012, 08:26:31 PM »
They are added directly in sitemap if specified in "Do not parse" setting.
Oleg Ignatiuk
www.xml-sitemaps.com
Send me a Private Message

For maximum exposure and traffic for your web site check out our additional SEO Services.

 

SMF 2.0.12 | SMF © 2014, Simple Machines
XHTML RSS WAP2