when doing unlimited crawl, it always comes up with different totals

[rgcderuiter]

Hey Oleg,

I have purchased your stand alone version of the generator. The script seems to work fine, however, every time i run it for unlimited pages, it comes up with an entirely different total number of indexed pages.

When i check each generated sitemap, it looks like it skips the urls it already indexed before, but I of course need one big sitemap with ALL urls. I have about 7000 urls on my site, according to google. At least id like to see the result to be the same outcome every time i run the crawler, so i know its working properly.

Could you please have a look at our config / settings - what should we do to make it work properly?

I pm'ed you our location of the generator etc.

Hope to hear from you soon.

[XML-Sitemaps Support]

Hello,

please try to enable Changelog option and track which pages are removed/added to sitemap.

[rgcderuiter]

i noticed you ticked that function on

ok awesome

i did another crawl and it added not even 500 pages.. and it removed over 600 pages.. you can check it out in the changelog

what can cause the removal of these urls?

i just want them all to be listed, every single one of them

thanks for your feedback so far, hope to hear from you soon again

[rgcderuiter]

got any further news bro?

its kinda urgent here. google has punishing us, we need to hurry

[XML-Sitemaps Support]

Hello,

looks like your site locks incoming requests in case if there are many continued pages loaded from the same IP, I've added a delay after every request to avoid that. Please check now.

[rgcderuiter]

thank you for your excellent help, everything is sorted now!

thanks!

[XML-Sitemaps Support]

You are welcome!