19k page scanned
« on: July 18, 2019, 05:16:35 PM »

i run the script and (13 hours already) and until now didn't finish.
its already scanned around 20k paged and added to sitemap only 945 page!!! its logic or something wrong?
Re: 19k page scanned
« Reply #1 on: July 18, 2019, 06:39:22 PM »

some pages could be skipped when there is a noindex robots meta tag or canonical meta tag points to a different URL.
Re: 19k page scanned
« Reply #2 on: July 29, 2019, 04:28:13 PM »
all the part of the currency convert was scanned but wasn't added to site maps
for example the below link.

[ External links are visible to forum administrators only ]

what is the reason?

Re: 19k page scanned
« Reply #3 on: July 30, 2019, 06:40:12 AM »

canonical meta tag on that page points to a different URL:
Code: [Select]
<link rel="canonical" href="https://www.domain.com/rates/currencies-converter/USD/EUR" />