Sitemap.xml crawling very slow
« on: August 31, 2013, 12:10:20 PM »
Can anyone explain why it takes a lot of time for crawling?
I'm already on it for 3 hours and still lacking about 6 hours to complete.
And I have to do the manual crowling every 15 minutes!
 >:(


Pages added to sitemap: 1637
Pages scanned: 4146 (303,328.7 KB)
Pages left: 7940 (+ 2686 queued for the next depth level)
Time passed: 3:07:20
Time left: 5:58:45
Memory usage: 14,559.8 Kb
Resuming the last session (last updated: 2013-08-31 10:59:15)
Re: Sitemap.xml crawling very slow
« Reply #1 on: August 31, 2013, 01:41:57 PM »
Hello,

The crawling time itself depends on the website page generation time mainly, since it crawls the site similar to search engine bots.
For instance, if it it takes 1 second to retrieve every page, then 1000 pages will be crawled in about 16 minutes.
Re: Sitemap.xml crawling very slow
« Reply #2 on: August 31, 2013, 02:08:34 PM »
Thanks for your reply...
What does this mean? (2686 + queued for the next depth level)
By now has almost 5000 in queue!
And Is always increasing.
Regards...
Re: Sitemap.xml crawling very slow
« Reply #3 on: August 31, 2013, 02:23:42 PM »
Now I can not crawling and appears every time this information!
Can you help me please ...

Already in progress. Current process state is displayed:

Crawling already in progress.
Last log access time: 2013-08-31 13:13:08
Click here to interrupt it.
-------------------------------------------------------------------------


Pages added to sitemap: 2138
Pages scanned: 5650 (400,052.0 KB)
Pages left: 6436 (+ 5253 queued for the next depth level)
Time passed: 4:16:00
Time left: 4:51:36
Memory usage: 16,491.8 Kb
Re: Sitemap.xml crawling very slow
« Reply #4 on: September 01, 2013, 12:52:22 PM »
I have just finished one depth and now I have another to complete!
I'm in the 4th depht, when it ends?
I'm still waiting for a reply!
Regards...
Re: Sitemap.xml crawling very slow
« Reply #5 on: September 04, 2013, 09:12:31 PM »
Hello,

sitemap is created when crawling is completely finished (i.e. all pages found on your site are visited by generator bot).
Re: Sitemap.xml crawling very slow
« Reply #6 on: May 19, 2017, 02:58:56 AM »
I am getting 130000 pages in 23 minutes. Used to having 130000 pages in days.
Hint was in Narrow Indexed Pages Set. Try Exclusion preset. I am now using this software without a hassle.