Hey Oleg,

I have purchased your stand alone version of the generator. The script seems to work fine, however, every time i run it for unlimited pages, it comes up with an entirely different total number of indexed pages.

When i check each generated sitemap, it looks like it skips the urls it already indexed before, but I of course need one big sitemap with ALL urls. I have about 7000 urls on my site, according to google. At least id like to see the result to be the same outcome every time i run the crawler, so i know its working properly.

Could you please have a look at our config / settings - what should we do to make it work properly?

I pm'ed you our location of the generator etc.

Hope to hear from you soon.
Re: when doing unlimited crawl, it always comes up with different totals
« Reply #2 on: January 23, 2009, 12:46:19 AM »
i noticed you ticked that function on ;)

ok awesome

i did another crawl and it added not even 500 pages.. and it removed over 600 pages.. you can check it out in the changelog

what can cause the removal of these urls?

i just want them all to be listed, every single one of them ;)

thanks for your feedback so far, hope to hear from you soon again
Re: when doing unlimited crawl, it always comes up with different totals
« Reply #3 on: January 23, 2009, 02:11:46 PM »
got any further news bro?

its kinda urgent here. google has punishing us, we need to hurry
Re: when doing unlimited crawl, it always comes up with different totals
« Reply #4 on: January 23, 2009, 07:04:36 PM »
Hello,

looks like your site locks incoming requests in case if there are many continued pages loaded from the same IP, I've added a delay after every request to avoid that. Please check now.
Re: when doing unlimited crawl, it always comes up with different totals
« Reply #5 on: January 27, 2009, 04:37:22 PM »
thank you for your excellent help, everything is sorted now!

thanks!