Very Very Slow crawling 3819 pages en 24 h
« on: September 05, 2013, 04:58:28 PM »
Already in progress. Current process state is displayed:
Links depth: 93
Current page: behance.net.html
Pages added to sitemap: 3819
Pages scanned: 3821 (54,169.1 KB)
Pages left: 24 (+ 0 queued for the next depth level)
Time passed: 1:27:33
Time left: 0:00:33
Memory usage: 1,646.2 Kb

going very slow so I will not be able to create my sitemap or 10 years. since my website has over 50,000 pages.
I need your help,

thanks

Muchas Gracias !
Re: Very Very Slow crawling 3819 pages en 24 h
« Reply #1 on: September 06, 2013, 08:36:47 PM »
memory_limit   256M

max_execution_time   9000


2500 pages  en 24 H.

i try every thing  but can,t speed up .

Help !!!


Muchas gracias!
Re: Very Very Slow crawling 3819 pages en 24 h
« Reply #2 on: September 08, 2013, 08:22:58 AM »
Hello,

The crawling time itself depends on the website page generation time mainly, since it crawls the site similar to search engine bots.
For instance, if it it takes 1 second to retrieve every page, then 1000 pages will be crawled in about 16 minutes.

In your case I see that it has crawled 3821 pages in 1h27min:
Pages scanned: 3821 (54,169.1 KB)
Pages left: 24 (+ 0 queued for the next depth level)
Time passed: 1:27:33
Re: Very Very Slow crawling 3819 pages en 24 h
« Reply #3 on: May 19, 2017, 02:58:42 AM »
I am getting 130000 pages in 23 minutes. Used to having 130000 pages in days.
Hint was in Narrow Indexed Pages Set. Try Exclusion preset. I am now using this software without a hassle.