I tried several times to generate sitemap, but it's never generate properly... Our site have over 500k pages, script generated sitemap and after 59006 added pages, generator stopped adding...
Got an error message:
Resuming the last session (last updated: 2011-05-02 17:45:50)
Fatal error: Allowed memory size of 268435456 bytes exhausted (tried to allocate 32791358 bytes) in /var/www/website.com/httpdocs/generator/pages/class.utils.inc.php(2) : eval()'d code on line 27
Please let me know how to fix this issue. Also soon we'll have over 5M pages, so we need best configuration to add all pages to sitemap.
Also when generator is running google can't access sitemap, please advice how to avoid such problems ?
One more thing, we have cluster of servers, and generator is installed on our main server, which is generating sitemap by subdomain: main.website.com , and in configuration all links showing as main.website.com/content.html and so on... I changed in index config file that generator would generate [external links are visible to admins only], not main.website.com, but in url list and html sitemap generator still adding main.website.com, how to avoid this issue too ?