Problem with run crawling
« on: October 02, 2014, 11:26:40 AM »
Hi
Excuseme but i don't speak english
I have a web site from 950000 url
I have buy this software and installa folder generator with permession (view tutorial)

I went to create the page url (Crawling --> RUN) but after 20 it stops scanning

I contacted my hosting (aruba.it) and the php.ini file is configured in this way
memory_limit 128 MB
max_execution_time 120

Can i help me?  ::)
Re: Problem with run crawling
« Reply #2 on: October 02, 2014, 11:55:34 AM »
I reply your email
In this page there are url with:
Category --> Provincia --> City

for example: [ External links are visible to forum administrators only ]

In italy there are 110 province and 8000 city and in the web site there are 70-80 category

In total there are 950000 url
Re: Problem with run crawling
« Reply #3 on: October 02, 2014, 12:33:14 PM »
Hello,

you need to separate that huge page on multiple pages, for instance one province listed per page.
Re: Problem with run crawling
« Reply #4 on: October 02, 2014, 01:58:53 PM »
Ok I'm editing the pages'll let you know

Thank you
Re: Problem with run crawling
« Reply #5 on: October 02, 2014, 02:36:36 PM »
I have this error
 Links depth: 2
Current page: ...../......html
Pages added to sitemap: 2514
Pages scanned: 2515 (852,397.1 KB)
Pages left: 7941 (+ 176429 queued for the next depth level)
Time passed: 0:12:38
Time left: 0:39:53
Memory usage: 126,527.8 Kb
Fatal error: Allowed memory size of 268435456 bytes exhausted (tried to allocate 26669050 bytes) in /web/htdocs/www.namesite.com/home/generator/pages/class.utils.inc.php on line 102

????
Re: Problem with run crawling
« Reply #6 on: October 03, 2014, 01:55:38 PM »
I don't have sa sitemap

from 6 days i don't a sitemap

This software are erro for create 960.000 url in my site

I don't understand.
Can i help me?
Re: Problem with run crawling
« Reply #7 on: October 04, 2014, 12:43:17 PM »
Hello,

it looks like your server configuration doesn't allow to run the script long enough to create full sitemap. Please try to increase memory_limit and max_execution_time settings in php configuration at your host (php.ini file) or contact hosting support regarding this.
 
Re: Problem with run crawling
« Reply #8 on: October 04, 2014, 03:04:10 PM »
I have to buy a hosting I want to know what should be the minimum value  memory_limit  and max_execution_time to be able to run your software

Thamk you