Resource Hog
« on: August 01, 2008, 08:45:31 PM »
Boy, if I knew that this program was such a resource hog...

Maybe if you had a function to turn off the queue, you would save on memory usage. I am on a shared server and cannot get over 1100 links in my SiteMap.

I can't imagine the load it would take if I spidered all 500,000 of my URLs  ???

« Last Edit: August 01, 2008, 08:53:36 PM by chris44 »
Re: Resource Hog
« Reply #1 on: August 02, 2008, 05:05:49 PM »

if you experience the increased server load due to intensive crawling of the site, it is recommended to use "Make delay after each X request" option that will let generator slow down the crawling process.