XML Sitemaps Generator

Author Topic: URL's are crawled but not added in sitemap  (Read 7944 times)

office83

  • Registered Customer
  • Approved member
  • *
  • Posts: 3
URL's are crawled but not added in sitemap
« on: May 02, 2015, 09:52:26 PM »
Hi, can someone explain to me way is not adding to sitemap the urls?
Is there something wrong in my configuration?
Pages crawled: 9000 (482 added in sitemap), Queued: 7518, Depth level: 3
Thank you
« Last Edit: May 02, 2015, 10:08:15 PM by office83 »

XML-Sitemaps Support

  • Administrator
  • Hero Member
  • *****
  • Posts: 10625
Re: URL's are crawled but not added in sitemap
« Reply #1 on: May 03, 2015, 10:29:06 AM »
Hello,

possibly those URLs are redirected to other pages or they have noindex robots meta tag.
Oleg Ignatiuk
www.xml-sitemaps.com
Send me a Private Message

For maximum exposure and traffic for your web site check out our additional SEO Services.

office83

  • Registered Customer
  • Approved member
  • *
  • Posts: 3
Re: URL's are crawled but not added in sitemap
« Reply #2 on: May 03, 2015, 11:31:59 AM »
Hi,
you are right in the log file i get this;
[external links are visible to admins only] - lu - [external links are visible to admins only]

but i am crawling the https site not the http and all shop is redirected to https from http.

Can you let me know if i can do something.
Thank you

XML-Sitemaps Support

  • Administrator
  • Hero Member
  • *****
  • Posts: 10625
Re: URL's are crawled but not added in sitemap
« Reply #3 on: May 04, 2015, 05:19:15 AM »
Hello,

make sure that "Allow subdomains" setting is disabled, otherwise generator will crawl both http and https links.
Oleg Ignatiuk
www.xml-sitemaps.com
Send me a Private Message

For maximum exposure and traffic for your web site check out our additional SEO Services.

office83

  • Registered Customer
  • Approved member
  • *
  • Posts: 3
Re: URL's are crawled but not added in sitemap
« Reply #4 on: May 04, 2015, 06:02:09 AM »
I have try that, now seams to be ok, but still i have a 8000 page limit and i can generate that sitemap. Other limits are set to 0.
The Shop is in OC and there are a loot of url's to crawl but i can't even crawl this 8000 pages.
Can you please take a look it is possible that i have some bad settings :(.

Thank you

XML-Sitemaps Support

  • Administrator
  • Hero Member
  • *****
  • Posts: 10625
Re: URL's are crawled but not added in sitemap
« Reply #5 on: May 04, 2015, 10:53:29 AM »
Hello,

please let me know your generator URL/login in private message to check this.
Oleg Ignatiuk
www.xml-sitemaps.com
Send me a Private Message

For maximum exposure and traffic for your web site check out our additional SEO Services.

 

SMF 2.0.12 | SMF © 2014, Simple Machines
XHTML RSS WAP2