Sitemap generation always stops. and Crawling page broke
« on: February 09, 2009, 01:54:02 AM »
Everyting is setup fine but the crawler always stops after 10 minutes or so.

Now my Crawling tab only displays the Run in background checkbox.

Config seems fine and it does crawl for a while. I always resume but it does not work for very long.

The site i'm trying to crawl has 50-75K pages so.....

I also get this occasionally:
Internal Server Error
The server encountered an internal error or misconfiguration and was unable to complete your request.


Ideas?

Ed

« Last Edit: February 09, 2009, 02:01:19 AM by ed11 »
Re: Sitemap generation always stops. and Crawling page broke
« Reply #1 on: February 09, 2009, 11:21:16 PM »
Hello,

it looks like your server configuration doesn't allow to run the script long enough to create full sitemap. Please try to increase memory_limit and max_execution_time settings in php configuration at your host (php.ini file) or contact hosting support regarding this.
Re: Sitemap generation always stops. and Crawling page broke
« Reply #2 on: February 10, 2009, 03:25:03 AM »
What do you recommend for the settings. My php max_execution_time is set to 30 and the memory limit is set to 64M

thanks for your help. I will have to put in a support ticket to have this changed.

Ed
Re: Sitemap generation always stops. and Crawling page broke
« Reply #4 on: February 14, 2009, 04:59:25 AM »
I have made the changes and it still runs for less than 5 minutes.

Also, I host at HostGator.com if that helps any.

Ed
Re: Sitemap generation always stops. and Crawling page broke
« Reply #5 on: February 15, 2009, 09:24:37 PM »
Are you sure the new settings were applied? Try to create file named phpinfo.php in generator folder with:
Code: [Select]
<?php
phpinfo
();
?>
and open it in browser to see your PHP settings.
Re: Sitemap generation always stops. and Crawling page broke
« Reply #6 on: February 26, 2009, 02:37:24 AM »
Interesting. It is set in my php.ini QuickConfig but when I check with phpinfo.php, it is not set there.

I've opened a ticket with HostGator to fix this for me.

I'll follow up when it is working.

Ed
Re: Sitemap generation always stops. and Crawling page broke
« Reply #8 on: February 27, 2009, 03:38:25 AM »
It still stops. Seems like it runs longer. The crawling tab doesn't fully load anymore. I attached a screen shot. I now can't do anything. I checked my running processes on HostGator and it is not running.
Re: Sitemap generation always stops. and Crawling page broke
« Reply #10 on: March 02, 2009, 01:49:30 AM »
Oleg,

I've sent you the info...

Ed
Re: Sitemap generation always stops. and Crawling page broke
« Reply #11 on: June 12, 2009, 04:19:03 AM »
Oleg, I have the same problem, can you help me it. I have followed the instruction written, but it is still the same.

Thank you!
Re: Sitemap generation always stops. and Crawling page broke
« Reply #13 on: June 26, 2009, 10:27:49 AM »
Hi,

Was there any resolution to this, as I'm now suffering from the same issue?  My crawl will last 10 mins and then just stop - resuming does nothing.

My host server settings I'm unable to change - however on a test site with the max_memory and max_execution_time limits set lower than my ISP, it completes satisfactorily!

Any ideas?
Re: Sitemap generation always stops. and Crawling page broke
« Reply #14 on: June 26, 2009, 10:28:16 PM »
Please PM me your generator URL, crawler settings usually should be configured depending on the site.