crawl keeps hanging

[jojor2004]

Hi, my crawl script hangs all the time, no particular number of pages can be 20,000 or 75,000 - sometimes crawls for 6hrs others 6mins,

please advise

[XML-Sitemaps Support]

Hello,

it looks like your server configuration doesn't allow to run the script long enough to create full sitemap. Please try to increase memory_limit and max_execution_time settings in php configuration at your host (php.ini file) or contact hosting support regarding this.
 

[jojor2004]

that would sound right if it wasn't for the fact that the script has previously created a sitemap with 150k pages.

[jojor2004]

memory is set at 512 and execution time 90 seconds - hang iscompletely random - can run for 20k pages or 6 pages and then stops.

please advise

[jojor2004]

any chance of some advice on this - current crawl hangs just after 148935 pages  and will not go any further - just stops

[jojor2004]

some further info on this is that i have checked the debug output and it hangs every time it tries to saves it's progress.

[XML-Sitemaps Support]

Hello,

looks like you would need to increase memory_limit even more then, please try 800M.
Also,  in this case I would recommend to run generator in command line if you have ssh access to your server.