I have the same problem. I run a price comparison system [external links are visible to admins only], so I have several hundert thousand products/links in my website. The crawler always stops after one thousand links.
I already tried to adjust the server-settings. But Its not my server (just a webspace), so I am not fully free with the adjustments.
Is there another solution? What about a cronjob?
Thaks for you help