Am I crazy?
« on: March 25, 2007, 02:57:26 PM »
Is it my imagination or does the script use more resources and get slower as it crawls...

My site is about 1.5 million pages.   I am on 200,000 pages in my crawl and it is going MUCH slower than in the early pages...

Am I right?
Re: Am I crazy?
« Reply #1 on: March 25, 2007, 11:33:55 PM »

yes, it is possible that crawling becomes slower when significant amount of pages are already crawled, the actual decrease in speed depends on the number of pages and server configuration.