Crawling screen never refreshed
« on: February 14, 2007, 10:49:07 AM »
When I start the crawler, the screen seems to frozen after only few minutes... Does it mean the script is stopped? How can I know if the script is in run or not?
Re: Crawling screen never refreshed
« Reply #1 on: February 15, 2007, 02:34:28 PM »
Hello,

it looks like your server configuration doesn't allow to run the script long enough to create full sitemap. Please try to increase memory_limit and max_execution_time settings in php configuration at your host (php.ini file) or contact hosting support regarding this.
Re: Crawling screen never refreshed
« Reply #2 on: February 26, 2007, 06:00:44 AM »
I am having the same problem, what should these variables be set to? I have a site with almost 50 000 pages
Re: Crawling screen never refreshed
« Reply #3 on: February 26, 2007, 02:25:44 PM »
Hello,

max_execution_time depends on the page generation time of your site mainly, since it crawls the site similar to search engine bots.
For instance, if it it takes 1 second to retrieve every page on your site, then 1000 pages will be crawled in about 16 minutes (and max_execution_time should be at least 1000).
Re: Crawling screen never refreshed
« Reply #4 on: May 15, 2009, 01:45:35 PM »
Hi,

I tried to put the memory at 500MB, crawl to max level of 3 (8000 pages)  and put a delay of 1s between every request
after 1 or 2 mn it freeze

Links depth: 2
Current page: french-alps/isere
Pages added to sitemap: 278
Pages scanned: 320 (10,179.3 KB)
Pages left: 1405 (+ 513 queued for the next depth level)
Time passed: 1:15
Time left: 5:30
Memory usage: 1,498.2 Kb

I don't think the problem is coming from the memory, can you help me please?

Thanks

Mat
Re: Crawling screen never refreshed
« Reply #5 on: May 15, 2009, 03:24:29 PM »
There should be no issues with memory limit with this number of links. Please PM me your generator URL/login so that I can check it.