Does not crawl all the pages
« on: December 17, 2015, 05:31:17 AM »
We have CMS website (developed using DotNetNuke). Whenever we run the crawler it generates sitemap.xml for 500+ pages.

We expect the pages to be more than 9000+

What configuration should be done, please help.
Re: Does not crawl all the pages
« Reply #1 on: December 17, 2015, 06:24:35 AM »
Hello,

could you please PM me your generator URL and an example URL that is not included in sitemap and how it can be reached starting from homepage?
 
Oleg Ignatiuk
www.xml-sitemaps.com
Send me a Private Message

For maximum exposure and traffic for your web site check out our additional SEO Services.
Re: Does not crawl all the pages
« Reply #2 on: December 17, 2015, 07:39:24 AM »
example url
http://www.chariotz.com/projectdetail/Audio-Mobile/PorscheCayenne2008/2008-Porsche-Cayenne-GTS/4654

We are using standalone sitemap generator configured in our server. using the credentials we login in and then go to Crawling tab (attached screen grab) and click run

starting from home page
1. www.chariotz.com
2. Click on Get Ideas from the top menu
3. In the find - select make -> Porsche , model -> Cayenne then click GO
4. Click on the first picture AudioMobile will reach you to the above page.

Re: Does not crawl all the pages
« Reply #3 on: December 18, 2015, 07:12:04 AM »
Hello,

there must be a way to reach the pages by clicking links, so that crawler bot can find them. In this case it's required submitting the form (selecting item from dropdown list) hence it's not visible to a bot.
Oleg Ignatiuk
www.xml-sitemaps.com
Send me a Private Message

For maximum exposure and traffic for your web site check out our additional SEO Services.