ok I have bought the sitemap genarator, love the layou and easy of use and all the nice little features BUT its going to take forever to finish my sitemap?
I estamat that there will be aprox 1 to 1.5 millions pages to index on my site and at the current rate will be lucky if it can do it in a week, not only that but allowing it to do it is draging the server load down also so I am left to interupt the generator during the day and only let it run in the background at night, so now take into account it is only going to be running for apox 1/3 of the day means its going to take 3 weeks for it to genrate my sitemap.
thing I dont understand is the strain it is putting on the server, I mean im getting that the http process for this genrator can take upto 90% of the cpu ussage (figures from top in ssh) now the what I dont get is that it should'nt be anything near that becuase at pressent it only seems to be doing 20 pages every 10 seconds? now I can goto my site and click refresh 20 times in 10 seconds and it wouldnt put anywhere near that amount of strain on the server?
it seems the further it goes the through the site the longer it is taking, I have autosave (or whatever the feature is called) enabled to save the data every 3 minutes (180) and it is saving the nice big file in the data folder, so im guesing that the problem is it is taking far to long to add data to this file becuase of the size (90+ meg and growing every 3 mins)?
am I right in this? also of course it is set to create more than one sitemap (each map = 50,000 pages) so i HOPE that when the current state hits 50,000 pages added to sitemap it will actualy createt he first map and then empty the dump file and begin with the next file? if it doesnt then I doiubt that this sitemap will ever get finished becuase like I said it seems to slow down the more pages it scans.
ps would have posted this in the top forum but for some reason unable to create new topic in that forum so this seemed to be the next best choice, also would have sent a private message to the admin direct but once again there seems to be no option for me to do that? I did click the admin profile but no options to send private message.
I am leaving the sitemap going now in the background until tomorow morning, I will then interup the process and let you know whats happening.