crawler problem
« on: August 24, 2014, 09:45:59 AM »
Crawler is very slow

pages added in sitemap 2800 in 1 hour

can you help

Re: crawler problem
« Reply #1 on: August 24, 2014, 08:08:12 PM »

The crawling time itself depends on the website page generation time mainly, since it crawls the site similar to search engine bots.
For instance, if it it takes 1 second to retrieve every page, then 1000 pages will be crawled in about 16 minutes.