Links depth: 5 error

[kevin9]

I am trying to run a crawl on a site with about 20,000 identified pages but it keeps freezing at Links depth: 5  (13000 pages scanned and 8000 queued). i have run and re-run the crawl on a few occasions and it keeps freezing at this point. I have also set the crawl to delay for 60s after each  1000 requests if that helps?

[admin]

Hello,

do you have "resume generation" checkbox when running the generator again?
There is no need in so long delays, something like 5 seconds every 200 request would be better.

[kevin9]

Hi

Yes i have the 'resume generation' turned on and i have changed the delay to 6 seconds but it is still pausing at Link depth 5. I have tested the software to make sure that it is working and it seems to create sitemaps for them with no problem (they are a lot smaller though).

[admin]

Hello,

please let me know your generator URL in private message.