XML Sitemaps Generator

Author Topic: Site has over 14000 pages but only 300 showing in sitemap  (Read 8686 times)

sales1296

  • Registered Customer
  • Approved member
  • *
  • Posts: 9
Site has over 14000 pages but only 300 showing in sitemap
« on: March 19, 2014, 11:15:05 AM »
Hi,
My site has over 14000 pages and the sitemap generated shows only ~300+ links in it. I noticed that the pages are being skipped. When the crawler is executing, I can see all those pages being crawled, however they are not getting added in the sitemap.

Issue #2:
I want the crawler to ignore a certain type of query. I have tried blocking Disallow: /? and also tried the generator exclude url. Both works fine. However, the crawled URL's are now even less.

Would appreciate if you could help me fix/ look into what is going wrong in my case.

XML-Sitemaps Support

  • Administrator
  • Hero Member
  • *****
  • Posts: 10624
Re: Site has over 14000 pages but only 300 showing in sitemap
« Reply #1 on: March 20, 2014, 10:13:20 AM »
Hello,

could you please PM me your generator URL and an example URL that is not included in sitemap and how it can be reached starting from homepage?
 
Oleg Ignatiuk
www.xml-sitemaps.com
Send me a Private Message

For maximum exposure and traffic for your web site check out our additional SEO Services.

sales1296

  • Registered Customer
  • Approved member
  • *
  • Posts: 9
Re: Site has over 14000 pages but only 300 showing in sitemap
« Reply #2 on: March 20, 2014, 10:34:36 AM »
Hi,
I just sent you a PM with all the information. Please check.

sales1296

  • Registered Customer
  • Approved member
  • *
  • Posts: 9
Re: Site has over 14000 pages but only 300 showing in sitemap
« Reply #3 on: March 20, 2014, 10:39:09 AM »
Hi,
Please also note that the images and videos are not crawled at all. The images are stored in AWS S3.

XML-Sitemaps Support

  • Administrator
  • Hero Member
  • *****
  • Posts: 10624
Re: Site has over 14000 pages but only 300 showing in sitemap
« Reply #4 on: March 20, 2014, 02:57:01 PM »
Replied to your PM.
Oleg Ignatiuk
www.xml-sitemaps.com
Send me a Private Message

For maximum exposure and traffic for your web site check out our additional SEO Services.

sales1296

  • Registered Customer
  • Approved member
  • *
  • Posts: 9
Re: Site has over 14000 pages but only 300 showing in sitemap
« Reply #5 on: March 20, 2014, 03:12:01 PM »
Yes. The url can be reached. Please see my reply too in your PM

sales1296

  • Registered Customer
  • Approved member
  • *
  • Posts: 9
Re: Site has over 14000 pages but only 300 showing in sitemap
« Reply #6 on: March 21, 2014, 01:21:39 PM »
Hi,
Have you figured out why the sitemaps are not generated? Awaiting for your response and fix for this issue.

sales1296

  • Registered Customer
  • Approved member
  • *
  • Posts: 9
Re: Site has over 14000 pages but only 300 showing in sitemap
« Reply #7 on: March 22, 2014, 04:32:58 AM »
Hi,
I just noticed that when I use your free sitemap tool for the same site I use your unlimited sitemap generator, all the skipped url's are perfectly being crawled and added to the sitemap. I'm wondering why your unlimited generator is skipping my URL's when the free tool can generate properly.

sales1296

  • Registered Customer
  • Approved member
  • *
  • Posts: 9
Re: Site has over 14000 pages but only 300 showing in sitemap
« Reply #8 on: March 22, 2014, 05:54:40 AM »
Sorry..it's the same.. I looked at the wrong counter.

XML-Sitemaps Support

  • Administrator
  • Hero Member
  • *****
  • Posts: 10624
Re: Site has over 14000 pages but only 300 showing in sitemap
« Reply #9 on: March 23, 2014, 04:18:03 PM »
Hello,

looks like all "pin/" links are loaded with ajax request which is not visible to bots. It must comply with "ajax crawling" specifications: https://developers.google.com/webmasters/ajax-crawling/docs/getting-started
Oleg Ignatiuk
www.xml-sitemaps.com
Send me a Private Message

For maximum exposure and traffic for your web site check out our additional SEO Services.

 

SMF 2.0.12 | SMF © 2014, Simple Machines
XHTML RSS WAP2