Large Site Wont Add more then 1,500 pages
« on: May 02, 2008, 05:10:48 AM »
I need quick help, trying to get this sitemap to run on a large website that has probably 20k pages. I run the scan and have excluded out all of the variables that dont need to be followed but yet it gets to 1,500 pages and stops. So then I go back in and try to resume it and it runs but when it stops again and i go back to crawling it still says 1,500 pages.

I am not for sure where to proceed next. I have the latest 2.8 version that came out in december.
Re: Large Site Wont Add more then 1,500 pages
« Reply #1 on: May 03, 2008, 09:36:27 PM »

it looks like your server configuration doesn't allow to run the script long enough to create full sitemap. Please try to increase memory_limit and max_execution_time settings in php configuration at your host (php.ini file) or contact hosting support regarding this.
Re: Large Site Wont Add more then 1,500 pages
« Reply #2 on: May 13, 2008, 02:29:26 AM »
Thanks for the info. I updated both fields significantly. Now the only problem is, once I run it, it processes about 9k pages. Then when i go to resume again, it runs and runs and runs and then when i finally get back to the crawling page it shows me just to resume but there is nothing below it saying how many pages or even the submit button. So I get stuck on a page with a checkbox but no option to submit or see the stats on how many pages were processed or remaining.

Any ideas please?
Re: Large Site Wont Add more then 1,500 pages
« Reply #3 on: May 14, 2008, 12:56:43 AM »

it means that memory_limit setting is exceeded while generator is trying to read your dump log to display current progress state.
It's possible that there is an endless loop of links on your site that doesn't allow to finish sitemap generation, please PM me your generator URL to check that.