john

*
  • *
  • 16
Sitemap Generator Way Slow
« on: March 31, 2010, 09:38:19 PM »
I just moved my site from an aging dedicated server to a cloud server.  I have total server control and can change anything I need.

After moving I noticed maybe one page every 30 seconds was being indexed and the process would stop after a few hours.  On the old server it would scan my 50,000 pages fast, stop at 40,000, but then I would restart it and it would finish up.  Total time would be maybe 6 hours.

I read through the and it seems the PHP.ini file needs to be tweaked.  I changed my settings to:

max_execution_time = 86400
max_input_time = 86400
memory_limit = 1024

But this had absolutely no effect.  All settings in the program are exactly the same as before.  Any ideas?

Re: Sitemap Generator Way Slow
« Reply #1 on: March 31, 2010, 11:25:36 PM »
Hello,


The crawling time itself depends on the website page generation time mainly, since it crawls the site similar to search engine bots.
For instance, if it it takes 1 second to retrieve every page, then 1000 pages will be crawled in about 16 minutes.

Some of the real-world examples of big db-driven websites:
about 35,000 URLs indexed - 1h 40min total generation time
about 200,000 URLs indexed - 38hours total generation time

john

*
  • *
  • 16
Re: Sitemap Generator Way Slow
« Reply #2 on: April 03, 2010, 05:05:35 AM »
I reset the items in my php.ini and it is not stopping but it is so slow.  I restarted it from where it left off this morning and here are the stats:

Links depth: 3
Current page: articles.detail.php/16160/209/Finance/3/If_You_Want_to_Make_Money
Pages added to sitemap: 18289
Pages scanned: 29180 (466,023.5 KB)
Pages left: 31282 (+ 27393 queued for the next depth level)
Time passed: 227:08
Time left: 243:29
Memory usage: 84,780.0 Kb

On my last, much slower server, it would have been finished adding about 49,000 pages to the site map hours ago.  What can I do to speed it up?  The site is not slow at all.  According to my Pingdom server monitoring my average page load time is 572 ms
Re: Sitemap Generator Way Slow
« Reply #3 on: April 03, 2010, 04:54:45 PM »
Hello,

please let me know your generator URL/login in private message to check this.

john

*
  • *
  • 16
Re: Sitemap Generator Way Slow
« Reply #4 on: April 04, 2010, 05:05:57 AM »
It is still going.  I'll get you the log in details but after many many hours it seems close to finished.

Links depth: 7
Current page: articles.detail.php/3126/189/Motivational/Publishing/45/E-Bay_Gets_from_Sellers
Pages added to sitemap: 49878
Pages scanned: 114520 (1,312,408.4 KB)
Pages left: 2645 (+ 13149 queued for the next depth level)
Time passed: 1676:00
Time left: 38:42
Memory usage: 137,108.9 Kb

john

*
  • *
  • 16
Re: Sitemap Generator Way Slow
« Reply #5 on: April 08, 2010, 06:14:13 AM »
I don't know if you did anything but my site maps fly now:


Links depth: 4
Current page: articles.detail.php/1802/180/Home_Business/Business/1/Success...Bottoms-Up
Pages added to sitemap: 60679
Pages scanned: 60680 (93,919.2 KB)
Pages left: 12277 (+ 0 queued for the next depth level)
Time passed: 74:20
Time left: 15:02
Memory usage: 103,453.3 Kb

1 hour 14 minutes ans 60,679 pages added.  nice