Crawl stalls????
« on: June 25, 2008, 12:22:38 PM »
I've been using the sitemap generator with no problem - that was untill yesterday...

The crawl stalls intermitantly... any clues???

This was the latest result:

Links depth: 3
Current page: cart.php?act=reg&redir=L2luZGV4LnBocD9hY3Q9dmlld1Byb2QmYW1wO3Byb2R1Y3RJZD0xMjc=
Pages added to sitemap: 519
Pages scanned: 520 (8,111.0 KB)
Pages left: 130 (+ 392 queued for the next depth level)
Time passed: 2:55
Time left: 0:43
Memory usage: -

Paul.
Re: Crawl stalls????
« Reply #1 on: June 25, 2008, 01:05:10 PM »
I installed the unlimited yesterday and it stalls more or less the same way.
at times it does not report nothing, at times it says states a number of pages and dept level, but never ends
url: [ External links are visible to forum administrators only ]
it just stopped with the following message.
The error message is new.
Links depth: 2
Current page: prods/Inmigracion,-estado-y-derecho-r21295.php?lb=productos/libros novedades.php
Pages added to sitemap: 80
Pages scanned: 80 (2,413.5 KB)
Pages left: 453 (+ 230 queued for the next depth level)
Time passed: 2:03
Time left: 11:39
Memory usage: 1,001.7 Kb
Resuming the last session (last updated: 1970-01-01 01:00:00)
OK
The server encountered an internal error or misconfiguration and was unable to complete your request.

Please contact the server administrator, xxxxxxxxxxxxxxxxx  and inform them of the time the error occurred, and anything you might have done that may have caused the error.

More information about this error may be available in the server error log.

Additionally, a 404 Not Found error was encountered while trying to use an ErrorDocument to handle the request.
« Last Edit: June 25, 2008, 01:13:38 PM by finam »
Re: Crawl stalls????
« Reply #2 on: June 25, 2008, 06:16:39 PM »
I just installed Sitemap and have exactly the same problem as in in POST # 1.
Re: Crawl stalls????
« Reply #3 on: June 25, 2008, 11:20:03 PM »
Hello,

it looks like your server configuration doesn't allow to run the script long enough to create full sitemap. Please try to increase memory_limit and max_execution_time settings in php configuration at your host (php.ini file) or contact hosting support regarding this.

Also, in many cases it's possible to configure sitemap generator (Exclude URLs/Do not parse options) to significantly improve crawler performance.
Re: Crawl stalls????
« Reply #4 on: July 14, 2008, 01:26:24 PM »
Hi,

I spoken with our hosting support, and as we are on a shared server, we cant change the config of the ph.ini file.

You mention that its possible to configure the sitemap generator to significantyl improve the performance - how do I do this????
Re: Crawl stalls????
« Reply #5 on: July 16, 2008, 03:05:34 AM »
It can be done with "Exclude URLs"/"Do not parse URLs" options that allow to avoid indexing/crawling of "noise content" pages that you don't want to include in sitemap. If you need assistance with that, please PM me your generator URL.
Re: Crawl stalls????
« Reply #6 on: July 16, 2008, 10:26:06 AM »
Still having problems...

The generator can be found at [ External links are visible to forum administrators only ]

Cheers

ct

*
  • *
  • 1
Re: Crawl stalls????
« Reply #7 on: July 17, 2008, 05:35:43 PM »
I am having the same problem w/ Post #1 too.  It just ran a few hundreds of pages then stalled.  I had to click [View Sitemap] (or other tabs) -> [Crawling] -> check 'run in background' & 'resume last session'  to continue the crawling.  This is not happened for previous version I was using.  Now I am running v2.9 (2008-06-15). Previous one was very smooth.  Is there any log that I can generate to send it back for investigation?  Thanks

CT
Re: Crawl stalls????
« Reply #8 on: July 17, 2008, 07:55:55 PM »
Your generator requires login, please PM me details so that I can check.
Still having problems...


Cheers
Re: Crawl stalls????
« Reply #9 on: July 17, 2008, 07:56:28 PM »
How many pages do you have in total indexed with previous version?
Re: Crawl stalls????
« Reply #10 on: July 18, 2008, 11:42:39 AM »
Ignore that - just pressed the wrong button. I've changed my password - so I'll PM it to you now!
Re: Crawl stalls????
« Reply #12 on: July 22, 2008, 09:12:59 AM »
Thank you - what had you done differently?
Re: Crawl stalls????
« Reply #13 on: July 22, 2008, 04:34:25 PM »
You are welcome!
The crawler settings were optimized to avoid timing out.
Re: Crawl stalls????
« Reply #14 on: October 15, 2008, 11:53:41 PM »
Hi Oleg,
I've been having problems with the generator for a few months now (since my hosting provider moved me to a different server) and so I reinstalled the generator tonight (v2.9).  I've checked my permissions, but the crawl only runs part way before it stalls with:

Internal Server Error
The server encountered an internal error or misconfiguration and was unable to complete your request.

Please contact the server administrator, webmaster@mydomain.co.uk and inform them of the time the error occurred, and anything you might have done that may have caused the error.

More information about this error may be available in the server error log.

Additionally, a 404 Not Found error was encountered while trying to use an ErrorDocument to handle the request.

--------------------------------------------------------------------------------
Apache/2.2.9 (Unix) mod_ssl/2.2.9 OpenSSL/0.9.8b mod_bwlimited/1.4 Server at [ External links are visible to forum administrators only ] Port 80

If I try to crawl again then I can pick up a saved session, but it's very slow (and no progress is displayed) - not as I experienced with previous versions.

I've set my maximum execution time and memory limit to match the settings returned by phpinfo but can't seem to get any further.

Please can you help?