XML Sitemaps Generator

Author Topic: Program stops after a while and does not complete crawling  (Read 9460 times)

paypal258

  • Registered Customer
  • Jr. Member
  • *
  • Posts: 11
Program stops after a while and does not complete crawling
« on: November 19, 2012, 10:53:53 AM »
We recently changed our hosting server. I used to work with Version 5 of XML Sitemap, but it doesn't work on this new server (see attch V5_ERROR)

So, I decided to upgrade to new Version 6. I unpacked and installed it on a different folder and tried it.
At first it seems to run properly, but, after a while it stops with no vital signs. (attch V6_STUCK)

Furthermore I noticed that when Version5 worked fine, there were some details about "safety routine" with time elapsed since last update to prevent the process to stuck.

Any help solving this?

I'm obviuosly more interested in getting version 6 working fine!

Thanks a lot
Massimo

XML-Sitemaps Support

  • Administrator
  • Hero Member
  • *****
  • Posts: 10624
Re: Program stops after a while and does not complete crawling
« Reply #1 on: November 19, 2012, 04:08:51 PM »
Hello,

in case of Windows server it might be needed to increase script timeout in IIS configuration as described on:
http://forums.iis.net/p/1076662/1585343.aspx#1585343
http://www.iisadmin.co.uk/?p=7
 
Oleg Ignatiuk
www.xml-sitemaps.com
Send me a Private Message

For maximum exposure and traffic for your web site check out our additional SEO Services.

paypal258

  • Registered Customer
  • Jr. Member
  • *
  • Posts: 11
Re: Program stops after a while and does not complete crawling
« Reply #2 on: November 20, 2012, 03:03:24 PM »
Hello Oleg,

I've passed the information you gave me to our hosting technicians. They set the fastcgi timeout at 1hour and the max_execution_time at 180 seconds. (see below)

But the program still gets stuck!
I've got the sitemaps of 6 domains updated back to the 31st of October!!


======================================
c:\Windows\System32\inetsrv>appcmd list config -section:system.webServer/fastCgi

<system.webServer>
<fastCgi>
<application fullPath="C:\Program Files (x86)\Parallels\Plesk\admin\bin\php-cgi.exe" maxInstances="12" idleTimeout="1800" activityTimeout="3600">
<environmentVariables>
</environmentVariables>
</application>
<application fullPath="C:\Program Files (x86)\Parallels\Plesk\Additional\PleskPHP5\php-cgi.exe">
<environmentVariables>
</environmentVariables>
</application>
</fastCgi>
</system.webServer>
======================================


XML-Sitemaps Support

  • Administrator
  • Hero Member
  • *****
  • Posts: 10624
Re: Program stops after a while and does not complete crawling
« Reply #3 on: November 20, 2012, 04:55:25 PM »
Hello,

in case if timeout is set to 180 seconds, it would be stopped after that time and you 'll need to run generator again, resuming the process.
Oleg Ignatiuk
www.xml-sitemaps.com
Send me a Private Message

For maximum exposure and traffic for your web site check out our additional SEO Services.

paypal258

  • Registered Customer
  • Jr. Member
  • *
  • Posts: 11
Re: Program stops after a while and does not complete crawling
« Reply #4 on: November 21, 2012, 07:41:29 AM »
Thank you Oleg for your answer,

I resumed the crawling process, but it got stuck once again!
Here's the report of the crawling page:
-------
Links depth: 3
Current page: Austria_Immacolata_Tirolo_Innsbruck_Hotel_Bon_Alpina_Igls_offerte_FID8_AID11_CID73_RID117.html
Pages added to sitemap: 903
Pages scanned: 904 (38,473.0 KB)
Pages left: 618 (+ 5 queued for the next depth level)
Time passed: 0:03:30
Time left: 0:02:23
Memory usage: 2,284.6 Kb
Resuming the last session (last updated: 2012-11-20 14:52:21)
-------

Even if it worked, I cannot resume the crawling process every time it stops, since I need to set a chron job to do it daily.

In the previous installation, there was sort of a utility which re-ran the script after a while, I suppose to prevent scripting time-out: is it possible to activate it?

XML-Sitemaps Support

  • Administrator
  • Hero Member
  • *****
  • Posts: 10624
Re: Program stops after a while and does not complete crawling
« Reply #5 on: November 21, 2012, 10:43:51 PM »
There is a setting to monitor crawling state in generator configuration.
Oleg Ignatiuk
www.xml-sitemaps.com
Send me a Private Message

For maximum exposure and traffic for your web site check out our additional SEO Services.

matthew.deren

  • Registered Customer
  • Approved member
  • *
  • Posts: 1
Re: Program stops after a while and does not complete crawling
« Reply #6 on: December 06, 2012, 08:41:16 PM »
I am having this same issue as well. 

 

SMF 2.0.12 | SMF © 2014, Simple Machines
XHTML RSS WAP2