XML Sitemaps Generator

Author Topic: Very Slow all of a sudden  (Read 10399 times)

premium

  • Registered Customer
  • Approved member
  • *
  • Posts: 2
Very Slow all of a sudden
« on: October 23, 2009, 09:41:53 AM »
Hello,

I have encountered a very strange problem. The sitemap generator used to process the nearly 60,000 pages in just over 150 minutes but it is currently crawling at 1000 pages per hour. No setting has been changed on the generator or the server. It seems that something on the server is slowing things down. I have tried rebooting it but nothing has changed. Usually when I run the crawler the loads spike a little to say 1.5 - 2.0 but now they seem stuck at below 1 and it also does not use the fully memory. Does anyone have any experience with this? I am at a complete loss.

Thanks,

Wouter

XML-Sitemaps Support

  • Administrator
  • Hero Member
  • *****
  • Posts: 10621
Re: Very Slow all of a sudden
« Reply #1 on: October 23, 2009, 12:58:57 PM »
Hello,

sitemap generator crawls your site similar to normal visitors, i.e. opening every page of the site to find nested links. So, working time is mostly related to the time it takes to open a page on your site. i.e. if your pages (without images/css/etc) are opened in average of 1 second, crawler will get 3,600 pages in hour. Correspondingly, if your site serves pages faster it can crawl more pages. Please check how long (approximately) it takes to load pages on your site, may be it just started working slower.
Oleg Ignatiuk
www.xml-sitemaps.com
Send me a Private Message

For maximum exposure and traffic for your web site check out our additional SEO Services.

premium

  • Registered Customer
  • Approved member
  • *
  • Posts: 2
Re: Very Slow all of a sudden
« Reply #2 on: October 23, 2009, 01:30:36 PM »
I have not noticed a drop in page load times. It seems like something on the server is capping the script off at very slow speed.

XML-Sitemaps Support

  • Administrator
  • Hero Member
  • *****
  • Posts: 10621
Re: Very Slow all of a sudden
« Reply #3 on: October 23, 2009, 01:32:43 PM »
Check that you don't have "Make delay after each X request" enabled in generator configuration. Also, what is your website URL? (you can PM it to me)
Oleg Ignatiuk
www.xml-sitemaps.com
Send me a Private Message

For maximum exposure and traffic for your web site check out our additional SEO Services.

 

SMF 2.0.12 | SMF © 2014, Simple Machines
XHTML RSS WAP2