XML Sitemaps Generator

Author Topic: Does not crawl all the pages  (Read 3900 times)

wilf86

  • Registered Customer
  • Approved member
  • *
  • Posts: 2
Does not crawl all the pages
« on: December 17, 2015, 05:31:17 AM »
We have CMS website (developed using DotNetNuke). Whenever we run the crawler it generates sitemap.xml for 500+ pages.

We expect the pages to be more than 9000+

What configuration should be done, please help.

XML-Sitemaps Support

  • Administrator
  • Hero Member
  • *****
  • Posts: 10624
Re: Does not crawl all the pages
« Reply #1 on: December 17, 2015, 06:24:35 AM »
Hello,

could you please PM me your generator URL and an example URL that is not included in sitemap and how it can be reached starting from homepage?
 
Oleg Ignatiuk
www.xml-sitemaps.com
Send me a Private Message

For maximum exposure and traffic for your web site check out our additional SEO Services.

wilf86

  • Registered Customer
  • Approved member
  • *
  • Posts: 2
Re: Does not crawl all the pages
« Reply #2 on: December 17, 2015, 07:39:24 AM »
example url
[external links are visible to admins only]

We are using standalone sitemap generator configured in our server. using the credentials we login in and then go to Crawling tab (attached screen grab) and click run

starting from home page
1. [external links are visible to admins only]
2. Click on Get Ideas from the top menu
3. In the find - select make -> Porsche , model -> Cayenne then click GO
4. Click on the first picture AudioMobile will reach you to the above page.


XML-Sitemaps Support

  • Administrator
  • Hero Member
  • *****
  • Posts: 10624
Re: Does not crawl all the pages
« Reply #3 on: December 18, 2015, 07:12:04 AM »
Hello,

there must be a way to reach the pages by clicking links, so that crawler bot can find them. In this case it's required submitting the form (selecting item from dropdown list) hence it's not visible to a bot.
Oleg Ignatiuk
www.xml-sitemaps.com
Send me a Private Message

For maximum exposure and traffic for your web site check out our additional SEO Services.

 

SMF 2.0.12 | SMF © 2014, Simple Machines
XHTML RSS WAP2