fun

*
  • *
  • 2
Slow?
« on: July 11, 2006, 09:05:06 AM »
Hello!
I noticed that SG is slowing down when indexing more pages. For 4 days is scanned about 40K pages, but for last day only 500. I would like to index over 200K pages, but i don't know when it finish indexing.

Pages added to sitemap: 40375
Pages scanned: 40540 (847,164.3 Kb)
Pages left: 103579 (+ 44164 queued for the next depth level)
Time passed: 4288:38
Time left: 10957:25
Memory usage: 64,821.3 Kb


Also very often i have problem to access Crawling page.
In addition i noticed that the script slowing down whole server.

btw. Have you thought about database version, where links could be stored?

Yours Jack
« Last Edit: July 11, 2006, 09:06:53 AM by fun »
Re: Slow?
« Reply #1 on: July 12, 2006, 10:38:53 PM »
Hello,

since sitemap generator crawls your site with http requests, the main speed factor is how fast your pages are generated at the server.
You can find out more details here:
How long it will take to crawl my large website