XML Sitemaps Generator

Author Topic: Script Overloaded my Server and Memory  (Read 15038 times)

IntegraGsrBalla

  • Registered Customer
  • Approved member
  • *
  • Posts: 3
Script Overloaded my Server and Memory
« on: July 05, 2008, 07:32:10 PM »
I placed the script on one of my servers for a very-very large website that I have (with lots of data). I ran the script and it took up my entire servers virtual memory and overloaded the server. I had to (it took time because the server had 99% of it's virtual memory full) kill the process and restart the server.

Keep in mind that I have around 40 million pages that need to be crawled. Is there anything in the setting where it allows me to specify the maximum amount of memory the script is allowed to use on the server.

How long do you think this would take to crawl and build a sitemap for 40,000,000  links/pages? Days? Weeks?

XML-Sitemaps Support

  • Administrator
  • Hero Member
  • *****
  • Posts: 10623
Re: Script Overloaded my Server and Memory
« Reply #1 on: July 06, 2008, 02:33:13 AM »
It might take weeks to create sitemap of that size with the crawling method - sitemap generator crawls your site similar to normal visitors, so you should multiple the number of URL with the average time that it takes to open the page on your server (with no images/css). It would make more sense to create a smaller sitemap with the most important URLs included.
Oleg Ignatiuk
www.xml-sitemaps.com
Send me a Private Message

For maximum exposure and traffic for your web site check out our additional SEO Services.

IntegraGsrBalla

  • Registered Customer
  • Approved member
  • *
  • Posts: 3
Re: Script Overloaded my Server and Memory
« Reply #2 on: July 06, 2008, 04:14:29 PM »
Thanks,

The main problem was the script overloading my server and taking up a good 90% of the virtual memory. Is there a file where I can specify a memory limit, etc.?

XML-Sitemaps Support

  • Administrator
  • Hero Member
  • *****
  • Posts: 10623
Re: Script Overloaded my Server and Memory
« Reply #3 on: July 07, 2008, 05:59:09 AM »
You can specify the maximum memory_limit in PHP configuration (php.ini).
Oleg Ignatiuk
www.xml-sitemaps.com
Send me a Private Message

For maximum exposure and traffic for your web site check out our additional SEO Services.

 

SMF 2.0.12 | SMF © 2014, Simple Machines
XHTML RSS WAP2