XML Sitemaps Generator

Author Topic: crawler problem (Timeout ?)  (Read 8138 times)

info1545

  • Registered Customer
  • Approved member
  • *
  • Posts: 3
crawler problem (Timeout ?)
« on: May 31, 2011, 03:00:43 PM »
Hi,

if we start the crawljob we see in the browser window
"Please wait. Sitemap generation in progress...". We switch to the server and goes into the /generator/data/ directory and see the crawl_dump file.

The Browser Windows show us the crawling process ist working but on the server the script hang up or stopped after 300kb or after 30 minutes.

It was no sitemap.xml file generated

we get no error messages.

Server=Windows2008
Joomla 1.521
XML Sitemaps=Standalone Sitemap Generator (PHP) v5.0, 2011-01-25

Please can any give us support ? or a workflow.

Regards
Michael


info1545

  • Registered Customer
  • Approved member
  • *
  • Posts: 3
Re: crawler problem (Timeout ?)
« Reply #1 on: May 31, 2011, 05:13:46 PM »
Here our php.ini and the fast-cgi entry

**********php.ini
max_execution_time = 6000     ; Maximum execution time of each script, in seconds
max_input_time = 6000   ; Maximum amount of time each script may spend parsing request data
;max_input_nesting_level = 64 ; Maximum input variable nesting level
memory_limit = 128M      ; Maximum amount of memory a script may consume (128MB)

***********IIS FASTCGI
RequestTimeout=90
ActivityTimeout=1000
BR
Michael




XML-Sitemaps Support

  • Administrator
  • Hero Member
  • *****
  • Posts: 10625
Re: crawler problem (Timeout ?)
« Reply #2 on: May 31, 2011, 07:39:36 PM »
Hello,

please try to increase RequestTimeout to 1000 as well.
Oleg Ignatiuk
www.xml-sitemaps.com
Send me a Private Message

For maximum exposure and traffic for your web site check out our additional SEO Services.

info1545

  • Registered Customer
  • Approved member
  • *
  • Posts: 3
Re: crawler problem (Timeout ?)
« Reply #3 on: June 03, 2011, 05:15:13 PM »
Hi
thanks for your answer. we have change the parameter like your response.

But it is no better function.

we have done many test over the last day and have found, that we can crawl until 1500 sites. if we give more like 2000 sites the crawler script is hanging.

if you have paraemter file with setups for us it was very nice. it is for me no problem to give you over PM access to our server.

best regards
michael



XML-Sitemaps Support

  • Administrator
  • Hero Member
  • *****
  • Posts: 10625
Re: crawler problem (Timeout ?)
« Reply #4 on: June 05, 2011, 09:30:14 PM »
Yes, please PM me details to check that.
Oleg Ignatiuk
www.xml-sitemaps.com
Send me a Private Message

For maximum exposure and traffic for your web site check out our additional SEO Services.

 

SMF 2.0.12 | SMF © 2014, Simple Machines
XHTML RSS WAP2