crawl keeps hanging
« on: October 24, 2014, 10:02:20 PM »
Hi, my crawl script hangs all the time, no particular number of pages can be 20,000 or 75,000 - sometimes crawls for 6hrs others 6mins,

please advise
Re: crawl keeps hanging
« Reply #1 on: October 25, 2014, 02:49:38 PM »
Hello,

it looks like your server configuration doesn't allow to run the script long enough to create full sitemap. Please try to increase memory_limit and max_execution_time settings in php configuration at your host (php.ini file) or contact hosting support regarding this.
 
Re: crawl keeps hanging
« Reply #2 on: October 25, 2014, 04:54:52 PM »
that would sound right if it wasn't for the fact that the script has previously created a sitemap with 150k pages.
Re: crawl keeps hanging
« Reply #3 on: October 25, 2014, 10:07:16 PM »
memory is set at 512 and execution time 90 seconds - hang iscompletely random - can run for 20k pages or 6 pages and then stops.

please advise
Re: crawl keeps hanging
« Reply #4 on: October 26, 2014, 04:45:27 PM »
any chance of some advice on this - current crawl hangs just after 148935 pages  and will not go any further - just stops
Re: crawl keeps hanging
« Reply #5 on: October 26, 2014, 04:55:44 PM »
some further info on this is that i have checked the debug output and it hangs every time it tries to saves it's progress.
Re: crawl keeps hanging
« Reply #6 on: October 27, 2014, 04:43:07 AM »
Hello,

looks like you would need to increase memory_limit even more then, please try 800M.
Also,  in this case I would recommend to run generator in command line if you have ssh access to your server.