Very Very Slow crawling 3819 pages en 24 h

[info2402]

Already in progress. Current process state is displayed:
Links depth: 93
Current page: behance.net.html
Pages added to sitemap: 3819
Pages scanned: 3821 (54,169.1 KB)
Pages left: 24 (+ 0 queued for the next depth level)
Time passed: 1:27:33
Time left: 0:00:33
Memory usage: 1,646.2 Kb

going very slow so I will not be able to create my sitemap or 10 years. since my website has over 50,000 pages.
I need your help,

thanks

Muchas Gracias !

[info2402]

memory_limit   256M

max_execution_time   9000


2500 pages  en 24 H.

i try every thing  but can,t speed up .

Help !!!


Muchas gracias!

[XML-Sitemaps Support]

Hello,

The crawling time itself depends on the website page generation time mainly, since it crawls the site similar to search engine bots.
For instance, if it it takes 1 second to retrieve every page, then 1000 pages will be crawled in about 16 minutes.

In your case I see that it has crawled 3821 pages in 1h27min:
Pages scanned: 3821 (54,169.1 KB)
Pages left: 24 (+ 0 queued for the next depth level)
Time passed: 1:27:33

[jaykjayk4]

I am getting 130000 pages in 23 minutes. Used to having 130000 pages in days.
Hint was in Narrow Indexed Pages Set. Try Exclusion preset. I am now using this software without a hassle.