Generator cannot manage large number of links
« on: December 20, 2014, 04:42:59 PM »
My generator stop running after about crawling 40-50k pages (site has about half million pages) and when I try to start it again, I don't receive any message, just thrown back to ssh command line.

I've increase memory to 1GB and running crawler from ssh.
Re: Generator cannot manage large number of links
« Reply #1 on: December 20, 2014, 07:47:43 PM »
Hello,

with website of this size the best option is to create a limited sitemap - with "Maximum depth" or "Maximume URLs" option limited so that it would gather about 200-300,000 URLs, which would be main pages representing "roadmap" sitemap for search engines.
Re: Generator cannot manage large number of links
« Reply #2 on: December 21, 2014, 02:41:12 PM »
Ok. But how to get 200-300k links when generator stops working on 40-50k and whatever I try, it just got back to command line without any report.

P.S. This 40-50k links are in depth of 3.
Re: Generator cannot manage large number of links
« Reply #3 on: December 22, 2014, 10:16:29 AM »
Hello,

please let me know your generator URL/login in private message to check this.