XML Sitemaps Generator

Author Topic: Sitemap.xml crawling very slow  (Read 6954 times)

sunderlandz25

  • Registered Customer
  • Approved member
  • *
  • Posts: 4
Sitemap.xml crawling very slow
« on: August 31, 2013, 11:10:20 AM »
Can anyone explain why it takes a lot of time for crawling?
I'm already on it for 3 hours and still lacking about 6 hours to complete.
And I have to do the manual crowling every 15 minutes!
 >:(


Pages added to sitemap: 1637
Pages scanned: 4146 (303,328.7 KB)
Pages left: 7940 (+ 2686 queued for the next depth level)
Time passed: 3:07:20
Time left: 5:58:45
Memory usage: 14,559.8 Kb
Resuming the last session (last updated: 2013-08-31 10:59:15)

XML-Sitemaps Support

  • Administrator
  • Hero Member
  • *****
  • Posts: 10624
Re: Sitemap.xml crawling very slow
« Reply #1 on: August 31, 2013, 12:41:57 PM »
Hello,

The crawling time itself depends on the website page generation time mainly, since it crawls the site similar to search engine bots.
For instance, if it it takes 1 second to retrieve every page, then 1000 pages will be crawled in about 16 minutes.
Oleg Ignatiuk
www.xml-sitemaps.com
Send me a Private Message

For maximum exposure and traffic for your web site check out our additional SEO Services.

sunderlandz25

  • Registered Customer
  • Approved member
  • *
  • Posts: 4
Re: Sitemap.xml crawling very slow
« Reply #2 on: August 31, 2013, 01:08:34 PM »
Thanks for your reply...
What does this mean? (2686 + queued for the next depth level)
By now has almost 5000 in queue!
And Is always increasing.
Regards...

sunderlandz25

  • Registered Customer
  • Approved member
  • *
  • Posts: 4
Re: Sitemap.xml crawling very slow
« Reply #3 on: August 31, 2013, 01:23:42 PM »
Now I can not crawling and appears every time this information!
Can you help me please ...

Already in progress. Current process state is displayed:

Crawling already in progress.
Last log access time: 2013-08-31 13:13:08
Click here to interrupt it.
-------------------------------------------------------------------------


Pages added to sitemap: 2138
Pages scanned: 5650 (400,052.0 KB)
Pages left: 6436 (+ 5253 queued for the next depth level)
Time passed: 4:16:00
Time left: 4:51:36
Memory usage: 16,491.8 Kb

sunderlandz25

  • Registered Customer
  • Approved member
  • *
  • Posts: 4
Re: Sitemap.xml crawling very slow
« Reply #4 on: September 01, 2013, 11:52:22 AM »
I have just finished one depth and now I have another to complete!
I'm in the 4th depht, when it ends?
I'm still waiting for a reply!
Regards...

XML-Sitemaps Support

  • Administrator
  • Hero Member
  • *****
  • Posts: 10624
Re: Sitemap.xml crawling very slow
« Reply #5 on: September 04, 2013, 08:12:31 PM »
Hello,

sitemap is created when crawling is completely finished (i.e. all pages found on your site are visited by generator bot).
Oleg Ignatiuk
www.xml-sitemaps.com
Send me a Private Message

For maximum exposure and traffic for your web site check out our additional SEO Services.

 

SMF 2.0.12 | SMF © 2014, Simple Machines
XHTML RSS WAP2