Sitemap Generator Way Slow

[john]

I just moved my site from an aging dedicated server to a cloud server.  I have total server control and can change anything I need.

After moving I noticed maybe one page every 30 seconds was being indexed and the process would stop after a few hours.  On the old server it would scan my 50,000 pages fast, stop at 40,000, but then I would restart it and it would finish up.  Total time would be maybe 6 hours.

I read through the and it seems the PHP.ini file needs to be tweaked.  I changed my settings to:

max_execution_time = 86400
max_input_time = 86400
memory_limit = 1024

But this had absolutely no effect.  All settings in the program are exactly the same as before.  Any ideas?

[XML-Sitemaps Support]

Hello,


The crawling time itself depends on the website page generation time mainly, since it crawls the site similar to search engine bots.
For instance, if it it takes 1 second to retrieve every page, then 1000 pages will be crawled in about 16 minutes.

Some of the real-world examples of big db-driven websites:
about 35,000 URLs indexed - 1h 40min total generation time
about 200,000 URLs indexed - 38hours total generation time

[john]

I reset the items in my php.ini and it is not stopping but it is so slow.  I restarted it from where it left off this morning and here are the stats:

Links depth: 3
Current page: articles.detail.php/16160/209/Finance/3/If_You_Want_to_Make_Money
Pages added to sitemap: 18289
Pages scanned: 29180 (466,023.5 KB)
Pages left: 31282 (+ 27393 queued for the next depth level)
Time passed: 227:08
Time left: 243:29
Memory usage: 84,780.0 Kb

On my last, much slower server, it would have been finished adding about 49,000 pages to the site map hours ago.  What can I do to speed it up?  The site is not slow at all.  According to my Pingdom server monitoring my average page load time is 572 ms

[XML-Sitemaps Support]

Hello,

please let me know your generator URL/login in private message to check this.

[john]

It is still going.  I'll get you the log in details but after many many hours it seems close to finished.

Links depth: 7
Current page: articles.detail.php/3126/189/Motivational/Publishing/45/E-Bay_Gets_from_Sellers
Pages added to sitemap: 49878
Pages scanned: 114520 (1,312,408.4 KB)
Pages left: 2645 (+ 13149 queued for the next depth level)
Time passed: 1676:00
Time left: 38:42
Memory usage: 137,108.9 Kb

[john]

I don't know if you did anything but my site maps fly now:


Links depth: 4
Current page: articles.detail.php/1802/180/Home_Business/Business/1/Success...Bottoms-Up
Pages added to sitemap: 60679
Pages scanned: 60680 (93,919.2 KB)
Pages left: 12277 (+ 0 queued for the next depth level)
Time passed: 74:20
Time left: 15:02
Memory usage: 103,453.3 Kb

1 hour 14 minutes ans 60,679 pages added.  nice

[XML-Sitemaps Support]

Great, I'm glad it works better now!