How to make crawl slower?
« on: July 31, 2009, 02:18:59 AM »
Hi..i want to make crawl process slower cause it very slows my site (takes trafic, requests and you know..).. do i have to use

Make a delay between requests, X seconds after each N requests: function? and how should i know how much requests it does?

thanks, your sitemap tool is the best.
Re: How to make crawl slower?
« Reply #1 on: July 31, 2009, 01:32:02 PM »
Exactly, this option is used to slow down the crawling process. One request is basically one URL added to your sitemap. You can try something like 1 second delay after each 10 or 20 requests and see how it works for you.