Only a random sample of URLs is being crawled
« on: January 07, 2022, 03:19:53 PM »

Hello there,

since we have a large number of URLs I excluded several directories via the "Exclude URLs" option in the "Crawler Rules" tab. So far this seems working, but the remaining URLs that are being crawled and added to the sitemap are far too less.

Could you help us up with looking for a more correct setup?

Best regards,
CS

Re: Only a random sample of URLs is being crawled
« Reply #1 on: January 10, 2022, 06:40:35 PM »
Hello,

please let me know your generator URL/login and example of missing URL in private message to check this.
Re: Only a random sample of URLs is being crawled
« Reply #2 on: January 12, 2022, 03:42:03 PM »

Hello, thanks for your reply.

I wrote you a PM and I sincerely hope you received it :)

Looking forward to receiving some help!

Best, CS