Unlimited Sitemap Generator v8.0 doesn't crawl all pages
« on: June 06, 2018, 12:40:20 PM »
Hi!

First I want to say that I love your product! But I'm facing a problem:

Generator v8.0
Use server's response
Number of URLs: 40000
URLs per file: 10
Maximum pages: 0
Maximum depth level: 150
Maximum execution time, seconds: 0
Maximum memory usage, MB: 512
Use robots.txt file: yes

My sitemap:
https://www.artimol.pt/sitemap.xml

The sitemap generates with no errors, but doesn't do a ton of pages, I don't understand why, it always stops at 2787 links, for example he completely ignores this link:

https://www.artimol.pt/placas-de-derivados-de-madeira-e-outros/4272-4630067-5607322395156.html

You can search the sitemap for 5607322395156 and it's not there, please help, thanks!
Re: Unlimited Sitemap Generator v8.0 doesn't crawl all pages
« Reply #1 on: June 06, 2018, 01:35:33 PM »
Hello,
please let me know how that page can be reached, starting from homepage.
Oleg Ignatiuk
www.xml-sitemaps.com
Send me a Private Message

For maximum exposure and traffic for your web site check out our additional SEO Services.
Re: Unlimited Sitemap Generator v8.0 doesn't crawl all pages
« Reply #2 on: June 06, 2018, 03:47:07 PM »
Just like others that are included in the sitemap, that's the strange part, anyway:

https://www.artimol.pt/ (left side categories, click "Placas de derivados de madeira e outros")
https://www.artimol.pt/21-placas-de-derivados-de-madeira-e-outros (you have several pages)
https://www.artimol.pt/21-placas-de-derivados-de-madeira-e-outros?p=15 (is at page 15, enter it)
https://www.artimol.pt/placas-de-derivados-de-madeira-e-outros/4272-4630067-5607322395156.html (done!)

The strange part is that the sitemap has products from this category, just search for "placas":

https://www.artimol.pt/sitemap.xml

Link 456, 457, 458 and others, I don't get it! Thanks for reply and please help!
Re: Unlimited Sitemap Generator v8.0 doesn't crawl all pages
« Reply #3 on: June 06, 2018, 05:28:19 PM »
Hello,

the  pagination links have rel=nofollow attribute on your website:
Code: [Select]
<a rel="nofollow" href="/21-placas-de-derivados-de-madeira-e-outros?p=2">
Oleg Ignatiuk
www.xml-sitemaps.com
Send me a Private Message

For maximum exposure and traffic for your web site check out our additional SEO Services.
Re: Unlimited Sitemap Generator v8.0 doesn't crawl all pages
« Reply #4 on: June 06, 2018, 05:55:44 PM »
I'm going to investigate asap, will reply when I have more news, thanks.
Re: Unlimited Sitemap Generator v8.0 doesn't crawl all pages
« Reply #5 on: June 07, 2018, 03:17:03 PM »
Here is my report.

I contacted my provider and after a couple of tests we came to the conclusion that unfortunately your generator doesn't work properly with Prestashop (not sure for other online stores)

It works and adds all links, you should consider updating your script, thanks again.
« Last Edit: June 08, 2018, 04:41:26 AM by XML-Sitemaps Support »
Re: Unlimited Sitemap Generator v8.0 doesn't crawl all pages
« Reply #6 on: June 08, 2018, 04:42:33 AM »
Hello,

as long as rel=nofollow attribute is removed from links on your website it should work fine.

Also, you can disable "Support robots" setting in generator configuration to ignore those robots attributes.
Oleg Ignatiuk
www.xml-sitemaps.com
Send me a Private Message

For maximum exposure and traffic for your web site check out our additional SEO Services.
Re: Unlimited Sitemap Generator v8.0 doesn't crawl all pages
« Reply #7 on: June 08, 2018, 10:55:59 AM »
I can't change source code of Prestashop, it's not practical, and like I said, the other crawler does all the pages.

I'm still gonna use both, but you guys should consider revising your code, that said can you please check your inbox? Thanks again!
Re: Unlimited Sitemap Generator v8.0 doesn't crawl all pages
« Reply #8 on: June 08, 2018, 01:27:18 PM »
You can disable the setting as described above if you want to instruct generator bot to ignore the directive specified in the link on your website.
Oleg Ignatiuk
www.xml-sitemaps.com
Send me a Private Message

For maximum exposure and traffic for your web site check out our additional SEO Services.