crawler problem (Timeout ?)
« on: May 31, 2011, 04:00:43 PM »

if we start the crawljob we see in the browser window
"Please wait. Sitemap generation in progress...". We switch to the server and goes into the /generator/data/ directory and see the crawl_dump file.

The Browser Windows show us the crawling process ist working but on the server the script hang up or stopped after 300kb or after 30 minutes.

It was no sitemap.xml file generated

we get no error messages.

Joomla 1.521
XML Sitemaps=Standalone Sitemap Generator (PHP) v5.0, 2011-01-25

Please can any give us support ? or a workflow.


Re: crawler problem (Timeout ?)
« Reply #1 on: May 31, 2011, 06:13:46 PM »
Here our php.ini and the fast-cgi entry

max_execution_time = 6000     ; Maximum execution time of each script, in seconds
max_input_time = 6000   ; Maximum amount of time each script may spend parsing request data
;max_input_nesting_level = 64 ; Maximum input variable nesting level
memory_limit = 128M      ; Maximum amount of memory a script may consume (128MB)

***********IIS FASTCGI

Re: crawler problem (Timeout ?)
« Reply #3 on: June 03, 2011, 06:15:13 PM »
thanks for your answer. we have change the parameter like your response.

But it is no better function.

we have done many test over the last day and have found, that we can crawl until 1500 sites. if we give more like 2000 sites the crawler script is hanging.

if you have paraemter file with setups for us it was very nice. it is for me no problem to give you over PM access to our server.

best regards