Number of pages
« on: September 17, 2022, 10:00:42 AM »
I have beed using this for some time now (years)
I changed the config a few months ago to crawl the entire site instead of just products pages
At one time it crawled 100k pages but never finished the crawl

So now I have it to set run once a week with a max of 10K per file
I see many files
I have 31,000 unique products, but am seeing only 10k in a file
I get a weekly notification that my sitemap has been updated

So my question is:
1. Is the entire site getting crawled in that one week or just one file being updated with 10k links?

How often do I need to run the script (cron) in order to update for the entire site? Or am I good?



Re: Number of pages
« Reply #2 on: September 21, 2022, 10:14:33 AM »
well, somthing is not right
It is completing every day with 111 pages
The site has 30,000 product pages alone which is the main thing I am trying to index
ALSO, I am getting this error and there is not such path on my site

An error occured
Sitemap file is not writable: /home/stampden/public_html/sitemapgen/data/sitemap.html
This is an addon domain and should be
/home/stampfair.com/stampden/sitemapgen/data/sitemap.html