During crawling, the generator freezes.
Links depth: 2
Current page: thoroughbred-horseracing-news.php?id=1097&mo=12&yr=2006
Pages added to sitemap: 1476
Pages scanned: 1500 (67,928.6 Kb)
Pages left: 47 (+ 4307 queued for the next depth level)
Time passed: 1:52
Time left: 0:03
Memory usage: 2,962.9 Kb
I tried setting up on another url and the crawling freezes 60 pages in.
No idea what to do, please help?