Thank you, this seems to be an improvement. The crawl_dump.log file has reached 164Mb in just a few hours with no hanging. I will monitor it and see what happens.
Meanwhile the html access page index.php will not load. I would like to monitor progress. Can you help me analyze the crawl_state.log file?
The html display from a previous crawl attempt showed
Links depth: 5
Current page: detail.php?in_npi=1467466706&state=CA
Pages added to sitemap: 36852
Pages scanned: 90700 (4,347,643.9 KB)
Pages left: 630350 (+ 46121 queued for the next depth level)
Time passed: 667:48
Time left: 4641:11
Memory usage: 255,449.1 Kb
Current snapshot of crawl_state.log shows
0 => 29203.245185852,
1 => 'detail.php?in_npi=1508846221&state=CA',
2 => 629941,
3 => 99620,
4 => 4571292527,
5 => 5,
6 => '269,618.0 Kb',
7 => 92568,
8 => 354,
Some of the items are obvious, some are not. Can you list the elements of this array?