Well, I decided to make it public as well, cause it seems your software doesn't work on all servers and you don't respond all my questions in PM.
We have not too huge vb forums. From our PMs you know already that crawler cannot complete the job properly and sitemap doesn't include all URLs. I requested our VPS tech support (siteground). 27 apr crawler completed the work and generated sitemap [ External links are visible to forum administrators only ]
Before it has been done only partially.
Server specs:
2 CPU Cores, CentOS, 4GB RAM, 40GB SSD, cPanel, Cloud Storage, Management Services
Detailed:
Total processors: 2
Processor #1
Vendor
GenuineIntel
Name
Intel(R) Xeon(R) CPU E5-2690 v2 @ 3.00GHz
Speed
3000.023 MHz
Cache
25600 KB
Processor #2
Vendor
GenuineIntel
Name
Intel(R) Xeon(R) CPU E5-2690 v2 @ 3.00GHz
Speed
3000.023 MHz
Cache
25600 KB
Response from our tech support:
"I have done excessive checks on the server and I increased a lot of limits in order to get the sitemap process completed, but to no avail.
At certain point during the process the server is loaded and this leads to the process been killed:
Code:
Apr 29 08:08:54 c54312: Killed long proc(9014) owner(***) :: /usr/local/php70/bin/php-cli /home/***/public_html/generator/runcrawl.php
Apr 29 08:22:32 c54312: Killed long proc(14546) owner(***) :: /usr/local/php70/bin/php-cli /home/***/public_html/generator/runcrawl.php
This might relate to problem in the script which causes the large load or lack of free CPU as your Cloud server has only 2 CPU cores.
The script times out.
You should revise that with the sitemap developers and check whether they would be able to provide more insights on the matter or try modifying it so that the process can take less time on the server. You can also consider having the sitemap generated on batches and not as one process.
Upgrading your resources is an alternative, however it depends on the script and in case there is a problem in it, the upgrade would not make any difference.
The process itself generates a lot of IOPS ([ External links are visible to forum administrators only ]) which actually are read and write requests.
Code:
date container cpu io_ops_read io_ops_write procs
2019-04-29 07:46:12.964538-05 c54312 17.98 37380 2223 278
2019-04-29 07:45:13.039444-05 c54312 9.57 20880 2009 275
2019-04-29 07:22:14.241641-05 c54312 74.64 11241 11397 288
2019-04-29 07:21:14.357071-05 c54312 76.60 10238 9072 288
Please advise.
Guest