Generator cannot manage large number of links

[etienne_stephane]

My generator stop running after about crawling 40-50k pages (site has about half million pages) and when I try to start it again, I don't receive any message, just thrown back to ssh command line.

I've increase memory to 1GB and running crawler from ssh.

[XML-Sitemaps Support]

Hello,

with website of this size the best option is to create a limited sitemap - with "Maximum depth" or "Maximume URLs" option limited so that it would gather about 200-300,000 URLs, which would be main pages representing "roadmap" sitemap for search engines.

[etienne_stephane]

Ok. But how to get 200-300k links when generator stops working on 40-50k and whatever I try, it just got back to command line without any report.

P.S. This 40-50k links are in depth of 3.

[XML-Sitemaps Support]

Hello,

please let me know your generator URL/login in private message to check this.