Unlimited is Stuck
« on: June 30, 2010, 03:51:02 PM »
Hi

We bought the Unlimited version after playing with the Demo and finding this product to be great for it's speed and results.  I have been messing with this for three days and my boss is going to have a cow if I don't get this done and start the other projects on my plate soon, so I hope there is a fix. We generate a HUGE google base feed via php and have never run into memory or timeout issues, so I'm not sure the issue is actually memory or exec time.

I searched the forums and double checked my permissions and php.ini, which have the following settings:

max_execution_time = 18000
max_input_time = 600
memory_limit = 512M
upload_max_filesize = 256M

The first time it ran, it got stuck after 6000 links.  I manually restarted it over and over until it got down to 7 links left, then it refused to finish.  The Crawler just sits at whatever point it's stuck at and returns back to the resume/run page at random intervals.  I figured it was a fluke so I reloaded the entire package and started from scratch again, making sure all the permissions were correct just to be on the safe side.

I let it run and it got stuck at only 400 links.  I tried running it as a CRON and it still sticks.  I blew it out and ran it again, babysitting it all night and resuming whenever it would return to the resume/run screen.  This morning it is completely stuck and no longer updates anything but the mod time on the crawl_state.log file.  The crawl_dump.log still has the mod time of the last failure.

Links depth: 4
Current page: sierra/sierra-1537C
Pages added to sitemap: 10232
Pages scanned: 18840 (849,266.3 KB)
Pages left: 20 (+ 16166 queued for the next depth level)
Time passed: 2:16:30
Time left: 0:00:08
Memory usage: 20,762.5 Kb

I searched the forums and tried all recommended fixes, but most of these issues were resolved via PM and never have a posted solution.  If someone can get this to run, I'll owe you a case of beer because my boss is going to kill me when he finds out how much time I have wasted on the simple task of generating a sitemap.  I'll save a bit of time and PM my login, since that seems to be the most common next step :)
Re: Unlimited is Stuck
« Reply #1 on: June 30, 2010, 07:06:46 PM »
Now it's just sitting on this:

Links depth: -
Current page: -
Pages added to sitemap: -
Pages scanned: - (- KB)
Pages left: - (+ - queued for the next depth level)
Time passed: -
Time left: -
Memory usage: -

Resuming the last session (last updated: 2010-06-30 17:14:00)
Re: Unlimited is Stuck
« Reply #3 on: June 30, 2010, 08:21:55 PM »
Got PM & PM back

No ssh access just Job Scheduling via CRON. 

Any idea as to how to fix this??  See anything on our config?