XML Sitemaps Generator

Author Topic: Very Very Slow crawling 3819 pages en 24 h  (Read 5987 times)

info2402

  • Registered Customer
  • Approved member
  • *
  • Posts: 2
Very Very Slow crawling 3819 pages en 24 h
« on: September 05, 2013, 03:58:28 PM »
Already in progress. Current process state is displayed:
Links depth: 93
Current page: behance.net.html
Pages added to sitemap: 3819
Pages scanned: 3821 (54,169.1 KB)
Pages left: 24 (+ 0 queued for the next depth level)
Time passed: 1:27:33
Time left: 0:00:33
Memory usage: 1,646.2 Kb

going very slow so I will not be able to create my sitemap or 10 years. since my website has over 50,000 pages.
I need your help,

thanks

Muchas Gracias !

info2402

  • Registered Customer
  • Approved member
  • *
  • Posts: 2
Re: Very Very Slow crawling 3819 pages en 24 h
« Reply #1 on: September 06, 2013, 07:36:47 PM »
memory_limit   256M

max_execution_time   9000


2500 pages  en 24 H.

i try every thing  but can,t speed up .

Help !!!


Muchas gracias!

XML-Sitemaps Support

  • Administrator
  • Hero Member
  • *****
  • Posts: 10624
Re: Very Very Slow crawling 3819 pages en 24 h
« Reply #2 on: September 08, 2013, 07:22:58 AM »
Hello,

The crawling time itself depends on the website page generation time mainly, since it crawls the site similar to search engine bots.
For instance, if it it takes 1 second to retrieve every page, then 1000 pages will be crawled in about 16 minutes.

In your case I see that it has crawled 3821 pages in 1h27min:
Pages scanned: 3821 (54,169.1 KB)
Pages left: 24 (+ 0 queued for the next depth level)
Time passed: 1:27:33
Oleg Ignatiuk
www.xml-sitemaps.com
Send me a Private Message

For maximum exposure and traffic for your web site check out our additional SEO Services.

 

SMF 2.0.12 | SMF © 2014, Simple Machines
XHTML RSS WAP2