Is getting slowly after a while
« on: September 01, 2015, 03:50:53 PM »
the Generator works fine. But after a while it is getting slowly. It is scrawling one page each 2 seconds.
It would take a while with 5000 pages :-((

If I close the browser and continue with it, it is working fast for the next appr. 30 seconds
Re: Is getting slowly after a while
« Reply #1 on: September 01, 2015, 06:11:09 PM »
Hello,

The crawling time itself depends on the website page generation time mainly, since it crawls the site similar to search engine bots.
For instance, if it it takes 1 second to retrieve every page, then 1000 pages will be crawled in about 16 minutes.
Re: Is getting slowly after a while
« Reply #2 on: September 02, 2015, 08:27:12 AM »
I did some finetuning. Now it is running in 2 minutes for 7000 pages.


thx