Our site has aproximately 90,000 pages. After 8 hours and some 35,000 pages crawling suddenly stops.
Trying to resume I select:
"Continue the interrupted session (2011-08-18 06:50:15, URLs added: 35685, estimated URLs left in a queue: 20148)"
Yet, crawling does not continue.
At this point crawl_dump.log is already 34 MB in size.
sitemap.xml and sitemap.xml.gz are set to 666 and data folder is set to 777.
I have enclosed my configuration file and teh screen shots of the two crawling pages.