XML Sitemaps Generator

Author Topic: when doing unlimited crawl, it always comes up with different totals  (Read 17298 times)

rgcderuiter

  • Registered Customer
  • Approved member
  • *
  • Posts: 4
Hey Oleg,

I have purchased your stand alone version of the generator. The script seems to work fine, however, every time i run it for unlimited pages, it comes up with an entirely different total number of indexed pages.

When i check each generated sitemap, it looks like it skips the urls it already indexed before, but I of course need one big sitemap with ALL urls. I have about 7000 urls on my site, according to google. At least id like to see the result to be the same outcome every time i run the crawler, so i know its working properly.

Could you please have a look at our config / settings - what should we do to make it work properly?

I pm'ed you our location of the generator etc.

Hope to hear from you soon.

XML-Sitemaps Support

  • Administrator
  • Hero Member
  • *****
  • Posts: 10624
Re: when doing unlimited crawl, it always comes up with different totals
« Reply #1 on: January 22, 2009, 11:10:08 PM »
Hello,

please try to enable Changelog option and track which pages are removed/added to sitemap.
Oleg Ignatiuk
www.xml-sitemaps.com
Send me a Private Message

For maximum exposure and traffic for your web site check out our additional SEO Services.

rgcderuiter

  • Registered Customer
  • Approved member
  • *
  • Posts: 4
Re: when doing unlimited crawl, it always comes up with different totals
« Reply #2 on: January 23, 2009, 12:46:19 AM »
i noticed you ticked that function on ;)

ok awesome

i did another crawl and it added not even 500 pages.. and it removed over 600 pages.. you can check it out in the changelog

what can cause the removal of these urls?

i just want them all to be listed, every single one of them ;)

thanks for your feedback so far, hope to hear from you soon again

rgcderuiter

  • Registered Customer
  • Approved member
  • *
  • Posts: 4
Re: when doing unlimited crawl, it always comes up with different totals
« Reply #3 on: January 23, 2009, 02:11:46 PM »
got any further news bro?

its kinda urgent here. google has punishing us, we need to hurry

XML-Sitemaps Support

  • Administrator
  • Hero Member
  • *****
  • Posts: 10624
Re: when doing unlimited crawl, it always comes up with different totals
« Reply #4 on: January 23, 2009, 07:04:36 PM »
Hello,

looks like your site locks incoming requests in case if there are many continued pages loaded from the same IP, I've added a delay after every request to avoid that. Please check now.
Oleg Ignatiuk
www.xml-sitemaps.com
Send me a Private Message

For maximum exposure and traffic for your web site check out our additional SEO Services.

rgcderuiter

  • Registered Customer
  • Approved member
  • *
  • Posts: 4
Re: when doing unlimited crawl, it always comes up with different totals
« Reply #5 on: January 27, 2009, 04:37:22 PM »
thank you for your excellent help, everything is sorted now!

thanks!

XML-Sitemaps Support

  • Administrator
  • Hero Member
  • *****
  • Posts: 10624
Re: when doing unlimited crawl, it always comes up with different totals
« Reply #6 on: January 27, 2009, 11:34:48 PM »
You are welcome!
Oleg Ignatiuk
www.xml-sitemaps.com
Send me a Private Message

For maximum exposure and traffic for your web site check out our additional SEO Services.

 

SMF 2.0.12 | SMF © 2014, Simple Machines
XHTML RSS WAP2