Links depth: 5 error
« on: January 23, 2008, 12:15:54 PM »
I am trying to run a crawl on a site with about 20,000 identified pages but it keeps freezing at Links depth: 5  (13000 pages scanned and 8000 queued). i have run and re-run the crawl on a few occasions and it keeps freezing at this point. I have also set the crawl to delay for 60s after each  1000 requests if that helps?
Re: Links depth: 5 error
« Reply #1 on: January 23, 2008, 09:58:52 PM »

do you have "resume generation" checkbox when running the generator again?
There is no need in so long delays, something like 5 seconds every 200 request would be better.
Re: Links depth: 5 error
« Reply #2 on: January 24, 2008, 12:59:47 PM »

Yes i have the 'resume generation' turned on and i have changed the delay to 6 seconds but it is still pausing at Link depth 5. I have tested the software to make sure that it is working and it seems to create sitemaps for them with no problem (they are a lot smaller though).