Site has over 14000 pages but only 300 showing in sitemap
« on: March 19, 2014, 11:15:05 AM »
Hi,
My site has over 14000 pages and the sitemap generated shows only ~300+ links in it. I noticed that the pages are being skipped. When the crawler is executing, I can see all those pages being crawled, however they are not getting added in the sitemap.

Issue #2:
I want the crawler to ignore a certain type of query. I have tried blocking Disallow: /? and also tried the generator exclude url. Both works fine. However, the crawled URL's are now even less.

Would appreciate if you could help me fix/ look into what is going wrong in my case.
Re: Site has over 14000 pages but only 300 showing in sitemap
« Reply #1 on: March 20, 2014, 10:13:20 AM »
Hello,

could you please PM me your generator URL and an example URL that is not included in sitemap and how it can be reached starting from homepage?
 
Re: Site has over 14000 pages but only 300 showing in sitemap
« Reply #2 on: March 20, 2014, 10:34:36 AM »
Hi,
I just sent you a PM with all the information. Please check.
Re: Site has over 14000 pages but only 300 showing in sitemap
« Reply #3 on: March 20, 2014, 10:39:09 AM »
Hi,
Please also note that the images and videos are not crawled at all. The images are stored in AWS S3.
Re: Site has over 14000 pages but only 300 showing in sitemap
« Reply #5 on: March 20, 2014, 03:12:01 PM »
Yes. The url can be reached. Please see my reply too in your PM
Re: Site has over 14000 pages but only 300 showing in sitemap
« Reply #6 on: March 21, 2014, 01:21:39 PM »
Hi,
Have you figured out why the sitemaps are not generated? Awaiting for your response and fix for this issue.
Re: Site has over 14000 pages but only 300 showing in sitemap
« Reply #7 on: March 22, 2014, 04:32:58 AM »
Hi,
I just noticed that when I use your free sitemap tool for the same site I use your unlimited sitemap generator, all the skipped url's are perfectly being crawled and added to the sitemap. I'm wondering why your unlimited generator is skipping my URL's when the free tool can generate properly.
Re: Site has over 14000 pages but only 300 showing in sitemap
« Reply #8 on: March 22, 2014, 05:54:40 AM »
Sorry..it's the same.. I looked at the wrong counter.
Re: Site has over 14000 pages but only 300 showing in sitemap
« Reply #9 on: March 23, 2014, 04:18:03 PM »
Hello,

looks like all "pin/" links are loaded with ajax request which is not visible to bots. It must comply with "ajax crawling" specifications: https://developers.google.com/webmasters/ajax-crawling/docs/getting-started