I've got really large site, which consist of over 10 million pages and growing in number every day.
I ran sitemap generator that loads approximately 20 - 30 thousand pages per day. Complete Sitemap will be ready approximately in a year.
I have set the generator to make the xml files with 20 000 pages. (Number of URLs per file in XML sitemap: 20 000)
But crawling is stil running, over 200 000 pages was loaded, but there is no saved xml sitemap file yet and the temp file is getting realy large.
Is it possible to somehow set the generator to keep save xml files continuously with already loaded sites? For example after 20 thousands sites?