Memory Exausted

[bug-software]

Hi,

Is there any workaround for Memory Exausted error?

I have a huge site and haven't yet been able to create the sitemap

I've increated php's memory to 64 MB (default was but after 1 day I still get that error.

Any thoughts?

[admin]

Hi,

the workaround for this problem for the moment is further increasing of memory limit or diabling the memory_limit in php at all.
We're working on the optimization of memory usage for the next Generator version.

Thanks!

[bug-software]

Any change you can do a resume job of some sort?

Ex: Save a log file (as you do now) and if the file exists you resume rather than start over again?

[admin]

Hi,

this is one of the features that is considered for the next version.

[bug-software]

Cool. Thanks

I've removed the php safe mode & memory limits, waiting to see how it goes

Thanks

[admin]

Cool. Thanks

I've removed the php safe mode & memory limits, waiting to see how it goes

Thanks

Ok, great. Let me know the results please
Thanks!

[bug-software]

Still nothing 


'213,700.6',
'787:41',
'3121:16',
'5',
'121,270.4 Kb',
'54620',
'1751'
);</script>
<br />
<b>Fatal error</b>:  Maximum execution time of 9000 seconds exceeded in <b>/home/httpd/vhosts/XXXXXXXXXXXX/httpdocs/google/pages/class.grab.inc.php(2) : eval()'d code(1) : eval()'d code(1) : eval()'d code</b> on line <b>141</b><br />

After 1 day and 120MB of memory used it crashed with this.

Pay attentiont hat I've also disabled maximum execution time in php.ini althought it says that 9000 seconds exceeded

Any more thoughts ?

I relly need to index this site 

[admin]

Hi,

it looks like you forgot to disable maximum execution time at generator configurations screen. Please set it to "0" to have unlimited time (default is 9000 seconds).

[bug-software]

Missed that. Running the script again and going.

<script>fupdate('directory/airconditioningrepair30601
/',
'214515',
'56540',
'217,551.7',
'889:50',
'3376:04',
'5',
'122,413.1 Kb',
'56540',
'3769'
);</script>

Do you know how much memory it will "eat" ? I have 1GB of RAM and if it get's about 700 it will most likly fail.

Do you have estimates on how long does it take to process?

[admin]

Hi,
the script's running time depends mostly on the speed of your site, i.e. page generation time.
The amount of memory required also depends on the site (how many links is generated for each "depth level" and for each page, and even the length of the URL strings).
Some of the examples of big db-driven websites:
about 35,000 URLs indexed - 1h 40min total generation time
about 200,000 URLs indexed - 38hours total generation time

[bug-software]

fupdate('directory/automobilerental81122
/',
'201575',
'69480',
'243,583.4',
'1152:02',
'3342:18',
'5',
'130,902.0 Kb',
'69480',
'17534'
);</script>

And going...

Will see how it goes

[bug-software]

2 days and going...

fupdate('directory/hdtv14001
/',
'152215',
'118840',
'349,393.0',
'2155:17',
'2760:35',
'5',
'166,126.3 Kb',
'118840',
'82693'
);</script>

Didn't stoped yet 

[bug-software]

And going...

/',
'95575',
'175480',
'470,987.3',
'3664:09',
'1995:40',
'5',
'207,319.3 Kb',
'175480',
'157955'
);</script>
<script>fupdate('directory/petgrooming12208
/',
'95555',
'175500',
'471,028.1',
'3664:49',
'1995:23',
'5',
'207,332.0 Kb',
'175500',
'157976'
);</script>

Hoping it will last 

[admin]

Hi,

wow, that is long, we had a bigger site crawled in less time. Hopefully it will be done successfully
Anyway, you may want to consider optimizing your site to reduce the page generation time (if possible)

[bug-software]

Still going

[root@server2 root]# tail -f nohup.out
'170195',
'350760',
'833,621.4',
'10886:15',
'5282:12',
'6',
'312,963.0 Kb',
'350753',
'84980'
);</script>

I can't optimize the pages cause we need all the links for this site