I just installed 2.9 yesterday in hopes of correcting this same issue. The new version doesn't work any differently. I never get an error msg. It just stops at random.
Links depth: 2
Current page: Art/59897/185/Wedding-Favor-Selection-Some-Considerations.html
Pages added to sitemap: 1463
Pages scanned: 1480 (44,802.2 KB)
Pages left: 36 (+ 7127 queued for the next depth level)
Time passed: 30:28
Time left: 0:44
Memory usage: -
It will stop between 500 and 1500 pages. Restarting means it will crawl another 1000pages +- before it gets stuck again.
I tried at at least 20 times with different settings and I put the IP address in the box at the bottom of the config screen.