What is the process for generation and writing to sitemap files?
« on: September 06, 2020, 09:01:58 AM »
What is the process for generation and writing to sitemap files?

Am I right in thinking that the generator gleans all the information first, stores it temporarily, and then writes it to the sitemap files once done?

Also, once it's generated the site map for the first time, and runs again on a daily cron job, does it just look for changes and overwrites the sitemap files if changes to the website have occurred, rather than do a complete new generation process?

Currently my first sitemap generation is into 'Links depth 7' and has been running all night, but the site map files are empty.
Re: What is the process for generation and writing to sitemap files?
« Reply #1 on: September 06, 2020, 06:14:57 PM »
Hello,

1. yes, sitemap is cleared.

2. crawling is performed from the scratch each time.

3. sitemap files are created after the crawling is finished.
Re: What is the process for generation and writing to sitemap files?
« Reply #2 on: September 06, 2020, 06:25:41 PM »
So if the website is updated, for instance forum posts, and the Cron job is set to run daily, then if it takes over 24hours for the generator to crawl the site, then the sitemaps will be emptied each time and never get the chance to be written long enough for the Bots to read them?

Please explain a bit further as this would render a 24hr Cron job on our site useless
Re: What is the process for generation and writing to sitemap files?
« Reply #3 on: September 07, 2020, 08:30:59 AM »
Hello,

the cron task will resume the process and will not clear the previously crawled progress until it's finished.
Re: What is the process for generation and writing to sitemap files?
« Reply #4 on: September 07, 2020, 09:08:27 AM »
Thank you