XML Sitemaps Generator

Author Topic: Sitemap generation always stops. and Crawling page broke  (Read 24499 times)

ed11

  • Registered Customer
  • Approved member
  • *
  • Posts: 9
Sitemap generation always stops. and Crawling page broke
« on: February 09, 2009, 01:54:02 AM »
Everyting is setup fine but the crawler always stops after 10 minutes or so.

Now my Crawling tab only displays the Run in background checkbox.

Config seems fine and it does crawl for a while. I always resume but it does not work for very long.

The site i'm trying to crawl has 50-75K pages so.....

I also get this occasionally:
Internal Server Error
The server encountered an internal error or misconfiguration and was unable to complete your request.


Ideas?

Ed

« Last Edit: February 09, 2009, 02:01:19 AM by ed11 »

XML-Sitemaps Support

  • Administrator
  • Hero Member
  • *****
  • Posts: 10621
Re: Sitemap generation always stops. and Crawling page broke
« Reply #1 on: February 09, 2009, 11:21:16 PM »
Hello,

it looks like your server configuration doesn't allow to run the script long enough to create full sitemap. Please try to increase memory_limit and max_execution_time settings in php configuration at your host (php.ini file) or contact hosting support regarding this.
Oleg Ignatiuk
www.xml-sitemaps.com
Send me a Private Message

For maximum exposure and traffic for your web site check out our additional SEO Services.

ed11

  • Registered Customer
  • Approved member
  • *
  • Posts: 9
Re: Sitemap generation always stops. and Crawling page broke
« Reply #2 on: February 10, 2009, 03:25:03 AM »
What do you recommend for the settings. My php max_execution_time is set to 30 and the memory limit is set to 64M

thanks for your help. I will have to put in a support ticket to have this changed.

Ed

XML-Sitemaps Support

  • Administrator
  • Hero Member
  • *****
  • Posts: 10621
Re: Sitemap generation always stops. and Crawling page broke
« Reply #3 on: February 11, 2009, 03:05:01 AM »
I'd try to increase to 1200 and 256M correspondingly.
Oleg Ignatiuk
www.xml-sitemaps.com
Send me a Private Message

For maximum exposure and traffic for your web site check out our additional SEO Services.

ed11

  • Registered Customer
  • Approved member
  • *
  • Posts: 9
Re: Sitemap generation always stops. and Crawling page broke
« Reply #4 on: February 14, 2009, 04:59:25 AM »
I have made the changes and it still runs for less than 5 minutes.

Also, I host at HostGator.com if that helps any.

Ed

XML-Sitemaps Support

  • Administrator
  • Hero Member
  • *****
  • Posts: 10621
Re: Sitemap generation always stops. and Crawling page broke
« Reply #5 on: February 15, 2009, 09:24:37 PM »
Are you sure the new settings were applied? Try to create file named phpinfo.php in generator folder with:
Code: [Select]
<?php
phpinfo
();
?>
and open it in browser to see your PHP settings.
Oleg Ignatiuk
www.xml-sitemaps.com
Send me a Private Message

For maximum exposure and traffic for your web site check out our additional SEO Services.

ed11

  • Registered Customer
  • Approved member
  • *
  • Posts: 9
Re: Sitemap generation always stops. and Crawling page broke
« Reply #6 on: February 26, 2009, 02:37:24 AM »
Interesting. It is set in my php.ini QuickConfig but when I check with phpinfo.php, it is not set there.

I've opened a ticket with HostGator to fix this for me.

I'll follow up when it is working.

Ed

XML-Sitemaps Support

  • Administrator
  • Hero Member
  • *****
  • Posts: 10621
Re: Sitemap generation always stops. and Crawling page broke
« Reply #7 on: February 26, 2009, 03:06:14 AM »
Ok, let us know how it goes.
Oleg Ignatiuk
www.xml-sitemaps.com
Send me a Private Message

For maximum exposure and traffic for your web site check out our additional SEO Services.

ed11

  • Registered Customer
  • Approved member
  • *
  • Posts: 9
Re: Sitemap generation always stops. and Crawling page broke
« Reply #8 on: February 27, 2009, 03:38:25 AM »
It still stops. Seems like it runs longer. The crawling tab doesn't fully load anymore. I attached a screen shot. I now can't do anything. I checked my running processes on HostGator and it is not running.

XML-Sitemaps Support

  • Administrator
  • Hero Member
  • *****
  • Posts: 10621
Re: Sitemap generation always stops. and Crawling page broke
« Reply #9 on: March 01, 2009, 01:38:20 PM »
What are your new PHP settings values? Please PM me the link to your phpinfo() page.
Oleg Ignatiuk
www.xml-sitemaps.com
Send me a Private Message

For maximum exposure and traffic for your web site check out our additional SEO Services.

ed11

  • Registered Customer
  • Approved member
  • *
  • Posts: 9
Re: Sitemap generation always stops. and Crawling page broke
« Reply #10 on: March 02, 2009, 01:49:30 AM »
Oleg,

I've sent you the info...

Ed

vera

  • Registered Customer
  • Approved member
  • *
  • Posts: 1
Re: Sitemap generation always stops. and Crawling page broke
« Reply #11 on: June 12, 2009, 03:19:03 AM »
Oleg, I have the same problem, can you help me it. I have followed the instruction written, but it is still the same.

Thank you!

XML-Sitemaps Support

  • Administrator
  • Hero Member
  • *****
  • Posts: 10621
Re: Sitemap generation always stops. and Crawling page broke
« Reply #12 on: June 12, 2009, 10:44:17 AM »
Please PM me your generator URL as well.
Oleg Ignatiuk
www.xml-sitemaps.com
Send me a Private Message

For maximum exposure and traffic for your web site check out our additional SEO Services.

alan9

  • Registered Customer
  • Approved member
  • *
  • Posts: 4
Re: Sitemap generation always stops. and Crawling page broke
« Reply #13 on: June 26, 2009, 09:27:49 AM »
Hi,

Was there any resolution to this, as I'm now suffering from the same issue?  My crawl will last 10 mins and then just stop - resuming does nothing.

My host server settings I'm unable to change - however on a test site with the max_memory and max_execution_time limits set lower than my ISP, it completes satisfactorily!

Any ideas?

XML-Sitemaps Support

  • Administrator
  • Hero Member
  • *****
  • Posts: 10621
Re: Sitemap generation always stops. and Crawling page broke
« Reply #14 on: June 26, 2009, 09:28:16 PM »
Please PM me your generator URL, crawler settings usually should be configured depending on the site.
Oleg Ignatiuk
www.xml-sitemaps.com
Send me a Private Message

For maximum exposure and traffic for your web site check out our additional SEO Services.

 

SMF 2.0.12 | SMF © 2014, Simple Machines
XHTML RSS WAP2