Crawler interrupted even with "Run in background" selected.
« on: November 18, 2009, 06:00:58 PM »
Hi all,

I have recently installed the standalone generator.  All seems fine so far, however as the title suggests the crawler is interrupted every few minutes as the browser times outs.  The pages are being indexed during each crawl attempt.  Currently indexed 7000+ pages, with another 7000+ to go.  Even though I select the "run in background" option it is interrupted. 

Another issue which may be related - I never see the information of the crawl status.  The page never loads, and all I see is:

"Please wait. Sitemap generation in progress...
Please note! The script will run in the background until completion, even if browser window is closed.
"

Nothing else :(

Any ideas, it is really slow having to restart the crawl every few minutes.

Cheers.

-pb
Re: Crawler interrupted even with "Run in background" selected.
« Reply #1 on: November 18, 2009, 10:16:20 PM »
Hello,

it looks like your server configuration doesn't allow to run the script long enough to create full sitemap. Please try to increase memory_limit and max_execution_time settings in php configuration at your host (php.ini file) or contact hosting support regarding this.
Re: Crawler interrupted even with "Run in background" selected.
« Reply #2 on: November 19, 2009, 09:56:45 AM »
Thanks for the prompt reply.  Unfortunately I'm on shared hosting, not sure how willing they will be to change the config :(