Crawling is stopping after a while
« on: August 14, 2014, 03:03:33 PM »
I have a problem with the script, I installed it correctly and started the crawling. But everytime its stopped, I waited now a long time, but nothing happens...

The error
Quote
Links depth: 3
Current page: smartphone-samsung/discussions/review?id=43
Pages added to sitemap: 669
Pages scanned: 975 (70,709.1 KB)
Pages left: 18 (+ 16 queued for the next depth level)
Time passed: 0:13:29
Time left: 0:00:14
Memory usage: 2,174.8 Kb
Internal Server Error

The server encountered an internal error or misconfiguration and was unable to complete your request.

Please contact the server administrator at webmaster@allsmartphone.it to inform them of the time this error occurred, and the actions you performed just before this error.

More information about this error may be available in the server error log.

Additionally, a 404 Not Found error was encountered while trying to use an ErrorDocument to handle the request.
Quote

I have in the file php.ini this setting:
safe_mode=Off
memory_limit=128M
max_execution_time=30
post_max_size=64M
upload_max_filesize=64M

Can you help me?
Thank you very much
kind regards
Andrea
Re: Crawling is stopping after a while
« Reply #1 on: August 14, 2014, 09:04:15 PM »
Hello,

it looks like your server configuration doesn't allow to run the script long enough to create full sitemap. Please try to increase memory_limit and max_execution_time settings in php configuration at your host (php.ini file) or contact hosting support regarding this.
 
Re: Crawling is stopping after a while
« Reply #2 on: August 14, 2014, 10:00:06 PM »
Hi, thank you very much for your support!

parameter max_execution_time to what I should set it to run the script correctly?
While the memory_limit parameter is set to 128 MB

Thank you very much
kind regards
Andrea