XML Sitemaps Generator

Author Topic: Scripts kills server (Plesk Onyx 17.0.17, CentOS Linux 7.3.1611 (Core), PHP 7.1)  (Read 1822 times)

somebody that I used to know

  • Registered Customer
  • Jr. Member
  • *
  • Posts: 11
Hi,

We have tested it several times after moving to new server. We have deleted all, re-installed and re-configured it again and again, but it does not help.

After start of Sitemaps crawling it does not show progress as before and after about 300 to 400 pages it kills server. All domains and websites go down. You only can restart server via Plesk to make it work again. Afterwards you can go to index.php?op=crawl and then XML Sitemaps Generator shows that script is running and even shows the progress, but does not communicate. So any update of progress must be forced with 30 seconds updates or 60 seconds update:

Already in progress. Current process state is displayed:

Links depth: 2
Current page: XXXXXXXXXXX
Pages added to sitemap: 330
Pages scanned: 412 (26,934.0 KB)
Pages left: 36 (+ 1949 queued for the next depth level)
Time passed: 0:36:11
Time left: 0:03:09
Memory usage: 6,058.0 Kb

We also changed:
memory_limit from 128M to 512M
max_execution_time from 30 to 9000
but this does not change anything.

Server is a strong cloud server and has no issues with anything else.

Do you have any idea? Does XML Sitemaps Generator needs a new update to make it working again on most recent servers? Any help would be appreciated.

Best,
Wolfgang

XML-Sitemaps Support

  • Administrator
  • Hero Member
  • *****
  • Posts: 10793
Hello,

it's possible that the server couldn't handle the number of requests generator crawler is sending. You can try to use "Make delay" setting to slow down crawling and avoid overloading server.
Oleg Ignatiuk
www.xml-sitemaps.com
Send me a Private Message

For maximum exposure and traffic for your web site check out our additional SEO Services.

somebody that I used to know

  • Registered Customer
  • Jr. Member
  • *
  • Posts: 11
Thank you. I am trying.

How to make current process state displayed again? It is not showing during crawl.
How many requests needs XML Sitemaps Generator?

XML-Sitemaps Support

  • Administrator
  • Hero Member
  • *****
  • Posts: 10793
Hello,

if you are not getting progress updates it must be related to buffering of script output on server side, this can be affected by different server settings, you can find description here:
http://muzso.hu/2008/02/19/php-output-buffering

The number of requests corresponds to the number of pages on the website.
Oleg Ignatiuk
www.xml-sitemaps.com
Send me a Private Message

For maximum exposure and traffic for your web site check out our additional SEO Services.

somebody that I used to know

  • Registered Customer
  • Jr. Member
  • *
  • Posts: 11
Thank you very much, Oleg, for your kind support. We will work on your suggestion.

somebody that I used to know

  • Registered Customer
  • Jr. Member
  • *
  • Posts: 11
Oleg, We and the hosting company tried everything.

phpinfo is now showing:
zlib.output_compression off off
output_buffering    0  0

The flush-test still shows:
started: 2017.03.16 04:39:18
finished: 2017.03.16 04:39:24
Note: if flush() works in PHP, then the difference between start and end dates should be approx. 0-1 second. If flush() fails (PHP output is buffered), then the difference will be >= 5 seconds.

Flush failed. :-(

Do you have any idea what we can do to make XML Sitemaps Script showing output again?

PS: I've updated right now to new version v7.2 - the same problem afterwards. No output.
« Last Edit: March 16, 2017, 04:53:56 PM by somebody that I used to know »

XML-Sitemaps Support

  • Administrator
  • Hero Member
  • *****
  • Posts: 10793
Hello,  in this case I would recommend to run generator in command line if you have ssh access to your server.
Oleg Ignatiuk
www.xml-sitemaps.com
Send me a Private Message

For maximum exposure and traffic for your web site check out our additional SEO Services.

somebody that I used to know

  • Registered Customer
  • Jr. Member
  • *
  • Posts: 11
Our cloud server has ssh, but I am preferring browser. Thank you and thanks for the new PHP 7 compatibility! Works great. No freezing of server anymore. No delay between crawls necessary!

 

SMF 2.0.14 | SMF © 2017, Simple Machines
XHTML RSS WAP2