The script stops before completing the job
« on: May 27, 2014, 05:08:18 AM »
My site has more than 2 million pages and I'm facing some issue letting the application complete the job. I've already tested it on a smaller site so I know it works. What parameters should I use? I set pretty much everything to unlimited and memory at 8GB.

Any advises?
Re: The script stops before completing the job
« Reply #1 on: May 27, 2014, 05:57:05 AM »
Hello,

with website of this size the best option is to create a limited sitemap - with "Maximum depth" or "Maximume URLs" option limited so that it would gather about 200-300,000 URLs, which would be main pages representing "roadmap" sitemap for search engines.

Re: The script stops before completing the job
« Reply #2 on: May 27, 2014, 06:15:16 AM »
Aha that makes sense! Thanks for the suggestion!