Fatal error: Allowed memory size

[in3sting]

Hello,
I get this error, after it scans about 10000 pages:

Fatal error: Allowed memory size of 50331648 bytes exhausted (tried to allocate 7077889 bytes) in /home/******/public_html/generator/pages/class.utils.inc.php(2) : eval()'d code on line 6

thanks.

[admin]

Hello,

it looks like your server configuration doesn't allow to run the script long enough to create full sitemap. Please try to increase memory_limit and max_execution_time settings in php configuration at your host (php.ini file) or contact hosting support regarding this.

[in3sting]

My host asks:

How much memory the process require?

Thanks

[admin]

Hello,

the memory requirements depend on how many total URLs are there on your site, do you have an estimation on that number?

[kev1]

I am having a similar problem, I keep increasing the memory limit which gets me a bit further each time, however I am yet to complete a site map.

Is there any way of estimating the memory limit i will need based on the number of pages? It is a large site and as an esitmation i would guess there is somewhere in the region of upwards of 50,000 pages.

[admin]

Hello,

you can check the current memory usage in progress window and the number of URLs crawled so far to estimate required memory limit.

[in3sting]

The host said they allocated they're best, 64Mb... but I stii get this error:

Fatal error: Allowed memory size of 67108864 bytes exhausted (tried to allocate 17039477 bytes) in /home/in3sting/public_html/generator/pages/class.utils.inc.php(2) : eval()'d code on line 6

Pages added to sitemap: 14630
Pages scanned: 15300 (70,993.9 Kb)
Pages left: 6250 (+ 7639 queued for the next depth level)

What now?

[admin]

Hello,

please PM me your generator URL. If it's not possible to increase memory limit even more, we'll try to optimize crawler setting for your specific website to get it finished.

[in3sting]

Ok,
I've sent you a PM. Please have a look!

[admin]

An update:

your gallery had infinite loops of "User login" and "User password recovery" links that caused huge number of "noise pages". I've excluded them in sitemap generator configuration and full sitemap has been created successfully now.

[vliegveld]

I seem to be having the same problem, but I cannot see where the site loops...
[external links are visible to admins only]

I already blocked reclame.php and report.php

[admin]

Hello,

I've added ".html" into "Do not parse" option (they are still included in sitemap though) and sitemap has been created successfully.