XML Sitemaps Generator

Author Topic: Crawling one page every 1 or 2 seconds  (Read 8543 times)

william1

  • Registered Customer
  • Approved member
  • *
  • Posts: 3
Crawling one page every 1 or 2 seconds
« on: March 19, 2010, 12:15:20 AM »
Hi All

My crawls are taking forever and after looking at teh access logs i see that a page is being requested every 1 or 2 seconds. how can i speed this up?

Ww

XML-Sitemaps Support

  • Administrator
  • Hero Member
  • *****
  • Posts: 10625
Re: Crawling one page every 1 or 2 seconds
« Reply #1 on: March 19, 2010, 12:30:39 PM »
Hello,

sitemap generator has a built-in crawler and it requests the page after the previous one has been processed, so in case if page load time is about 1 second, then that's a time period between requests.
Oleg Ignatiuk
www.xml-sitemaps.com
Send me a Private Message

For maximum exposure and traffic for your web site check out our additional SEO Services.

william1

  • Registered Customer
  • Approved member
  • *
  • Posts: 3
Re: Crawling one page every 1 or 2 seconds
« Reply #2 on: March 19, 2010, 03:26:32 PM »
Thanks for the pointers, i will see what i can do to improve me site speed.

Is there a way of disabling this as my server has very little load on it at the moment.

Ww

XML-Sitemaps Support

  • Administrator
  • Hero Member
  • *****
  • Posts: 10625
Re: Crawling one page every 1 or 2 seconds
« Reply #3 on: March 20, 2010, 08:17:03 AM »
That is the way generator uses to find all pages on the site, it cannot be disabled.
Oleg Ignatiuk
www.xml-sitemaps.com
Send me a Private Message

For maximum exposure and traffic for your web site check out our additional SEO Services.

 

SMF 2.0.12 | SMF © 2014, Simple Machines
XHTML RSS WAP2