Crawler running for 20+ hours
« on: February 01, 2017, 04:51:15 PM »
Dear Team, I am crawling my site which has 1000's of pages. The crawler has been running for around 20+ hours. Wanted to check if there is any setting I can change to make it faster or if this is the expected time.
Re: Crawler running for 20+ hours
« Reply #1 on: February 02, 2017, 06:24:22 AM »

The crawling time itself depends on the website page generation time mainly, since it crawls the site similar to search engine bots.
For instance, if it it takes 1 second to retrieve every page, then 1000 pages will be crawled in about 16 minutes.

i.e., the aster website pages are loading, the faster would be the crawling.