Compression Error
« on: March 07, 2016, 11:48:22 AM »
Hello,
We are using the Standalone Generator since V2.3 form PHP 5.2 until now on 5.6. The problem is after upgrading our server to PHP5.6, Apache 2.4 with FastCGI, the generator runs about 7-8 minutes then throws a Compression Error (In Firefox, Internal Server Error on IE) and stop running. 
I configured high time (9000sec) for PHP and FastCGI max execution. The error is the same in Version 7.1 and 6.1. The cron jobs also stops.
Please advise. Thank you.
Re: Compression Error
« Reply #1 on: March 07, 2016, 02:25:19 PM »
Hello,

it looks like your server configuration doesn't allow to run the script long enough to create full sitemap. Please try to increase memory_limit and max_execution_time settings in php configuration at your host (php.ini file) or contact hosting support regarding this.
 

in this case I would recommend to run generator in command line if you have ssh access to your server.
Re: Compression Error
« Reply #2 on: March 07, 2016, 04:44:50 PM »
Same error with:
Resource Limits max_execution_time  9000
Resource Limits max_input_time  9000
Resource Limits memory_limit  512M
Re: Compression Error
« Reply #3 on: March 08, 2016, 04:25:28 AM »
Hello,
in this case I would recommend to run generator in command line if you have ssh access to your server.
Re: Compression Error
« Reply #4 on: March 09, 2016, 10:50:03 AM »
Same command as cron job ?
Re: Compression Error
« Reply #6 on: March 09, 2016, 03:44:02 PM »
19 | 88 | 730.1 | 0:00:22 | 0:01:42 | 1 | 1,790.7 Kb | 17 | 344 | 36
Killed
Re: Compression Error
« Reply #7 on: March 09, 2016, 03:50:27 PM »
But

If i run the command again:
h4>Already in progress. Current process state is displayed:</h4>
18 | 89 | 673.1 | 0:00:22 | 0:01:48 | 1 | 1,754.2 Kb | 16 | 310 | 1754

But does not run data/crawl_state.log freezes

Re: Compression Error
« Reply #8 on: March 10, 2016, 03:46:17 AM »
The process gets "killed" by the server in this case.

It's possible that the server couldn't handle the number of requests generator crawler is sending. You can try to use "Make delay" setting to slow down crawling and avoid overloading server.
Re: Compression Error
« Reply #9 on: March 11, 2016, 08:56:20 AM »
Solved. The process was killed by CSF Firewall, increasing PT_USERMEM = "150MB" and PT_USERTIME = "600s" solved the problem. Thank You !