My sitemap generator is at [external links are visible to admins only]. This a +17K classifieds site and I am getting an 'Internal Server Error' message at about 3-5mins after starting the crawl (regardless if I check the 'run in the background' box. the sitemap crawling seems to stop at 154 pages and about 9-10MB.
I have excluded the duplicate content ( print/) and only included the html classifieds detail pages ( category/ ). I would like to include other pages available in the nav menu and the homepage also.
Thanks in advance for your help,