URL's are crawled but not added in sitemap
« on: May 02, 2015, 10:52:26 PM »
Hi, can someone explain to me way is not adding to sitemap the urls?
Is there something wrong in my configuration?
Pages crawled: 9000 (482 added in sitemap), Queued: 7518, Depth level: 3
Thank you
« Last Edit: May 02, 2015, 11:08:15 PM by office83 »
Re: URL's are crawled but not added in sitemap
« Reply #1 on: May 03, 2015, 11:29:06 AM »
Hello,

possibly those URLs are redirected to other pages or they have noindex robots meta tag.
Re: URL's are crawled but not added in sitemap
« Reply #2 on: May 03, 2015, 12:31:59 PM »
Hi,
you are right in the log file i get this;
[ External links are visible to forum administrators only ] - lu - [ External links are visible to forum administrators only ]

but i am crawling the https site not the http and all shop is redirected to https from http.

Can you let me know if i can do something.
Thank you
Re: URL's are crawled but not added in sitemap
« Reply #3 on: May 04, 2015, 06:19:15 AM »
Hello,

make sure that "Allow subdomains" setting is disabled, otherwise generator will crawl both http and https links.
Re: URL's are crawled but not added in sitemap
« Reply #4 on: May 04, 2015, 07:02:09 AM »
I have try that, now seams to be ok, but still i have a 8000 page limit and i can generate that sitemap. Other limits are set to 0.
The Shop is in OC and there are a loot of url's to crawl but i can't even crawl this 8000 pages.
Can you please take a look it is possible that i have some bad settings :(.

Thank you
Re: URL's are crawled but not added in sitemap
« Reply #5 on: May 04, 2015, 11:53:29 AM »
Hello,

please let me know your generator URL/login in private message to check this.