Crawling Stops
« on: April 30, 2007, 12:42:33 AM »
I have version v2.4, 2006-10-11 and can not find where to upgrade, I have a problem when I run the crawler it stops part way through crawling my forums and I keep having to put in Do not parse and exclude URL's.
I run for about 10 minutes before all stops, is there a way to stop the crawl failing and also where can I upgrade my version

Cheers

Re: Crawling Stops
« Reply #1 on: April 30, 2007, 03:51:24 AM »
Hello,

you should increase max_execution_time and memory_limit settings in your php configuration for that.

The newest version of Sitemap Generator can be obtained using the same download link sent to you in activation email.
Re: Crawling Stops
« Reply #2 on: April 30, 2007, 07:30:54 PM »
can you resend the link as I no longer have the mail that was sent, if you need to validate me I used the mail address the I use here

thanks

Martyn
Re: Crawling Stops
« Reply #3 on: April 30, 2007, 07:53:39 PM »
set all the defaults to unlimited and it still stops, any ideas
Re: Crawling Stops
« Reply #4 on: May 01, 2007, 10:45:26 PM »
Hello,

I've resent you the download link.

Please note that you should increase setting values in *php* configuration (php.ini), NOT in Sitemap Generator configuration.
Re: Crawling Stops
« Reply #5 on: May 06, 2007, 06:26:53 PM »
Thanks for the link, when you say change php.ini do you mean on the server?

Also I have a new issue. can you help out please

Links depth: 6
Current page: portal/forum/showthread.php?goto=lastpost&t=6467
Pages added to sitemap: 7131
Pages scanned: 8580 (203,417.1 Kb)
Pages left: 9359 (+ 10182 queued for the next depth level)
Time passed: 59:15
Time left: 64:38
Memory usage: 15,560.7 Kb
Resuming the last session (last updated: 2007-05-03 23:15:12)
Fatal error: Allowed memory size of 134217728 bytes exhausted (tried to allocate 6291374 bytes) in D:\sitename\wwwroot\generator\pages\class.grab.inc.php(2) : eval()'d code on line 298
Re: Crawling Stops
« Reply #6 on: May 07, 2007, 02:26:51 PM »
Hello,

yes, I mean php configuration on your server (php.ini file).
Quote
Also I have a new issue. can you help out please
This is related to the same php config issue.
Re: Crawling Stops
« Reply #7 on: May 08, 2007, 09:02:43 PM »
its not my server i'm on a hosted server, how can I get this sorted

Thanks
Re: Crawling Stops
« Reply #8 on: May 09, 2007, 02:33:05 AM »
Please ask your host if they allow custom php config parameters definition.