Crawl stalls????

[paul31]

I've been using the sitemap generator with no problem - that was untill yesterday...

The crawl stalls intermitantly... any clues???

This was the latest result:

Links depth: 3
Current page: cart.php?act=reg&redir=L2luZGV4LnBocD9hY3Q9dmlld1Byb2QmYW1wO3Byb2R1Y3RJZD0xMjc=
Pages added to sitemap: 519
Pages scanned: 520 (8,111.0 KB)
Pages left: 130 (+ 392 queued for the next depth level)
Time passed: 2:55
Time left: 0:43
Memory usage: -

Paul.

[finam]

I installed the unlimited yesterday and it stalls more or less the same way.
at times it does not report nothing, at times it says states a number of pages and dept level, but never ends
url: [ External links are visible to forum administrators only ]
it just stopped with the following message.
The error message is new.
Links depth: 2
Current page: prods/Inmigracion,-estado-y-derecho-r21295.php?lb=productos/libros novedades.php
Pages added to sitemap: 80
Pages scanned: 80 (2,413.5 KB)
Pages left: 453 (+ 230 queued for the next depth level)
Time passed: 2:03
Time left: 11:39
Memory usage: 1,001.7 Kb
Resuming the last session (last updated: 1970-01-01 01:00:00)
OK
The server encountered an internal error or misconfiguration and was unable to complete your request.

Please contact the server administrator, xxxxxxxxxxxxxxxxx  and inform them of the time the error occurred, and anything you might have done that may have caused the error.

More information about this error may be available in the server error log.

Additionally, a 404 Not Found error was encountered while trying to use an ErrorDocument to handle the request.

[clasicwhite]

I just installed Sitemap and have exactly the same problem as in in POST # 1.

[XML-Sitemaps Support]

Hello,

it looks like your server configuration doesn't allow to run the script long enough to create full sitemap. Please try to increase memory_limit and max_execution_time settings in php configuration at your host (php.ini file) or contact hosting support regarding this.

Also, in many cases it's possible to configure sitemap generator (Exclude URLs/Do not parse options) to significantly improve crawler performance.

[paul31]

Hi,

I spoken with our hosting support, and as we are on a shared server, we cant change the config of the ph.ini file.

You mention that its possible to configure the sitemap generator to significantyl improve the performance - how do I do this?

[XML-Sitemaps Support]

It can be done with "Exclude URLs"/"Do not parse URLs" options that allow to avoid indexing/crawling of "noise content" pages that you don't want to include in sitemap. If you need assistance with that, please PM me your generator URL.

[paul31]

Still having problems...

The generator can be found at [ External links are visible to forum administrators only ]

Cheers

[ct]

I am having the same problem w/ Post #1 too.  It just ran a few hundreds of pages then stalled.  I had to click [View Sitemap] (or other tabs) -> [Crawling] -> check 'run in background' & 'resume last session'  to continue the crawling.  This is not happened for previous version I was using.  Now I am running v2.9 (2008-06-15). Previous one was very smooth.  Is there any log that I can generate to send it back for investigation?  Thanks

CT

[XML-Sitemaps Support]

Your generator requires login, please PM me details so that I can check.

Still having problems...


Cheers

[XML-Sitemaps Support]

How many pages do you have in total indexed with previous version?

[paul31]

Ignore that - just pressed the wrong button. I've changed my password - so I'll PM it to you now!

[XML-Sitemaps Support]

Replied to your PM, sitemap has been created successfully

[paul31]

Thank you - what had you done differently?

[XML-Sitemaps Support]

You are welcome!
The crawler settings were optimized to avoid timing out.

[Mark Wilson]

Hi Oleg,
I've been having problems with the generator for a few months now (since my hosting provider moved me to a different server) and so I reinstalled the generator tonight (v2.9).  I've checked my permissions, but the crawl only runs part way before it stalls with:

Internal Server Error
The server encountered an internal error or misconfiguration and was unable to complete your request.

Please contact the server administrator, [ External links are visible to forum administrators only ] and inform them of the time the error occurred, and anything you might have done that may have caused the error.

More information about this error may be available in the server error log.

Additionally, a 404 Not Found error was encountered while trying to use an ErrorDocument to handle the request.

--------------------------------------------------------------------------------
Apache/2.2.9 (Unix) mod_ssl/2.2.9 OpenSSL/0.9.8b mod_bwlimited/1.4 Server at [ External links are visible to forum administrators only ] Port 80

If I try to crawl again then I can pick up a saved session, but it's very slow (and no progress is displayed) - not as I experienced with previous versions.

I've set my maximum execution time and memory limit to match the settings returned by phpinfo but can't seem to get any further.

Please can you help?