XML Sitemaps Generator

Author Topic: Memory Exausted  (Read 88277 times)

bug-software

  • Registered Customer
  • Jr. Member
  • *
  • Posts: 15
Memory Exausted
« on: July 21, 2005, 05:09:24 PM »
Hi,

Is there any workaround for Memory Exausted error?

I have a huge site and haven't yet been able to create the sitemap

I've increated php's memory to 64 MB (default was 8) but after 1 day I still get that error.

Any thoughts?

XML-Sitemaps Support

  • Administrator
  • Hero Member
  • *****
  • Posts: 10624
Re: Memory Exausted
« Reply #1 on: July 21, 2005, 05:14:52 PM »
Hi,

the workaround for this problem for the moment is further increasing of memory limit or diabling the memory_limit in php at all.
We're working on the optimization of memory usage for the next Generator version.

Thanks!
Oleg Ignatiuk
www.xml-sitemaps.com
Send me a Private Message

For maximum exposure and traffic for your web site check out our additional SEO Services.

bug-software

  • Registered Customer
  • Jr. Member
  • *
  • Posts: 15
Re: Memory Exausted
« Reply #2 on: July 22, 2005, 08:50:22 AM »
Any change you can do a resume job of some sort?

Ex: Save a log file (as you do now) and if the file exists you resume rather than start over again?

XML-Sitemaps Support

  • Administrator
  • Hero Member
  • *****
  • Posts: 10624
Re: Memory Exausted
« Reply #3 on: July 22, 2005, 10:00:16 AM »
Hi,

this is one of the features that is considered for the next version.
Oleg Ignatiuk
www.xml-sitemaps.com
Send me a Private Message

For maximum exposure and traffic for your web site check out our additional SEO Services.

bug-software

  • Registered Customer
  • Jr. Member
  • *
  • Posts: 15
Re: Memory Exausted
« Reply #4 on: July 22, 2005, 02:15:06 PM »
Cool. Thanks

I've removed the php safe mode & memory limits, waiting to see how it goes :)

Thanks

XML-Sitemaps Support

  • Administrator
  • Hero Member
  • *****
  • Posts: 10624
Re: Memory Exausted
« Reply #5 on: July 22, 2005, 06:13:21 PM »
Cool. Thanks

I've removed the php safe mode & memory limits, waiting to see how it goes :)

Thanks
Ok, great. Let me know the results please :)
Thanks!
Oleg Ignatiuk
www.xml-sitemaps.com
Send me a Private Message

For maximum exposure and traffic for your web site check out our additional SEO Services.

bug-software

  • Registered Customer
  • Jr. Member
  • *
  • Posts: 15
Re: Memory Exausted
« Reply #6 on: July 25, 2005, 01:08:26 PM »
Still nothing  :'(


'213,700.6',
'787:41',
'3121:16',
'5',
'121,270.4 Kb',
'54620',
'1751'
);</script>
<br />
<b>Fatal error</b>:  Maximum execution time of 9000 seconds exceeded in <b>/home/httpd/vhosts/XXXXXXXXXXXX/httpdocs/google/pages/class.grab.inc.php(2) : eval()'d code(1) : eval()'d code(1) : eval()'d code</b> on line <b>141</b><br />

After 1 day and 120MB of memory used it crashed with this.

Pay attentiont hat I've also disabled maximum execution time in php.ini althought it says that 9000 seconds exceeded

Any more thoughts ?

I relly need to index this site  :'(

XML-Sitemaps Support

  • Administrator
  • Hero Member
  • *****
  • Posts: 10624
Re: Memory Exausted
« Reply #7 on: July 25, 2005, 03:45:40 PM »
Hi,

it looks like you forgot to disable maximum execution time at generator configurations screen. Please set it to "0" to have unlimited time (default is 9000 seconds).
Oleg Ignatiuk
www.xml-sitemaps.com
Send me a Private Message

For maximum exposure and traffic for your web site check out our additional SEO Services.

bug-software

  • Registered Customer
  • Jr. Member
  • *
  • Posts: 15
Re: Memory Exausted
« Reply #8 on: July 26, 2005, 12:17:16 PM »
Missed that. Running the script again and going.

<script>fupdate('directory/airconditioningrepair30601
/',
'214515',
'56540',
'217,551.7',
'889:50',
'3376:04',
'5',
'122,413.1 Kb',
'56540',
'3769'
);</script>

Do you know how much memory it will "eat" ? I have 1GB of RAM and if it get's about 700 it will most likly fail.

Do you have estimates on how long does it take to process?



XML-Sitemaps Support

  • Administrator
  • Hero Member
  • *****
  • Posts: 10624
Re: Memory Exausted
« Reply #9 on: July 26, 2005, 02:15:19 PM »
Hi,
the script's running time depends mostly on the speed of your site, i.e. page generation time.
The amount of memory required also depends on the site (how many links is generated for each "depth level" and for each page, and even the length of the URL strings).
Some of the examples of big db-driven websites:
about 35,000 URLs indexed - 1h 40min total generation time
about 200,000 URLs indexed - 38hours total generation time
Oleg Ignatiuk
www.xml-sitemaps.com
Send me a Private Message

For maximum exposure and traffic for your web site check out our additional SEO Services.

bug-software

  • Registered Customer
  • Jr. Member
  • *
  • Posts: 15
Re: Memory Exausted
« Reply #10 on: July 26, 2005, 04:38:45 PM »
fupdate('directory/automobilerental81122
/',
'201575',
'69480',
'243,583.4',
'1152:02',
'3342:18',
'5',
'130,902.0 Kb',
'69480',
'17534'
);</script>

And going...

Will see how it goes :)

bug-software

  • Registered Customer
  • Jr. Member
  • *
  • Posts: 15
Re: Memory Exausted
« Reply #11 on: July 27, 2005, 09:22:21 AM »
2 days and going...

fupdate('directory/hdtv14001
/',
'152215',
'118840',
'349,393.0',
'2155:17',
'2760:35',
'5',
'166,126.3 Kb',
'118840',
'82693'
);</script>

Didn't stoped yet  ;D

bug-software

  • Registered Customer
  • Jr. Member
  • *
  • Posts: 15
Re: Memory Exausted
« Reply #12 on: July 28, 2005, 10:31:52 AM »
And going...

/',
'95575',
'175480',
'470,987.3',
'3664:09',
'1995:40',
'5',
'207,319.3 Kb',
'175480',
'157955'
);</script>
<script>fupdate('directory/petgrooming12208
/',
'95555',
'175500',
'471,028.1',
'3664:49',
'1995:23',
'5',
'207,332.0 Kb',
'175500',
'157976'
);</script>

Hoping it will last  ;D

XML-Sitemaps Support

  • Administrator
  • Hero Member
  • *****
  • Posts: 10624
Re: Memory Exausted
« Reply #13 on: July 28, 2005, 02:25:24 PM »
Hi,

wow, that is long, we had a bigger site crawled in less time. Hopefully it will be done successfully :)
Anyway, you may want to consider optimizing your site to reduce the page generation time :) (if possible)
Oleg Ignatiuk
www.xml-sitemaps.com
Send me a Private Message

For maximum exposure and traffic for your web site check out our additional SEO Services.

bug-software

  • Registered Customer
  • Jr. Member
  • *
  • Posts: 15
Re: Memory Exausted
« Reply #14 on: August 02, 2005, 11:04:34 AM »
Still going :(

[root@server2 root]# tail -f nohup.out
'170195',
'350760',
'833,621.4',
'10886:15',
'5282:12',
'6',
'312,963.0 Kb',
'350753',
'84980'
);</script>

I can't optimize the pages cause we need all the links for this site :(


 

SMF 2.0.12 | SMF © 2014, Simple Machines
XHTML RSS WAP2