Crawler running for 20+ hours
« on: February 01, 2017, 04:51:15 PM »
Dear Team, I am crawling my site which has 1000's of pages. The crawler has been running for around 20+ hours. Wanted to check if there is any setting I can change to make it faster or if this is the expected time.
Re: Crawler running for 20+ hours
« Reply #1 on: February 02, 2017, 06:24:22 AM »

The crawling time itself depends on the website page generation time mainly, since it crawls the site similar to search engine bots.
For instance, if it it takes 1 second to retrieve every page, then 1000 pages will be crawled in about 16 minutes.

i.e., the aster website pages are loading, the faster would be the crawling.
Oleg Ignatiuk
Send me a Private Message

For maximum exposure and traffic for your web site check out our additional SEO Services.