Competitor Analysis Tool: Analyze rival websites and gain valuable SEO and PPC insights
« previous next »
Pages: 1

info1304

*
  • *
  • 13
Cronjob on a file to launch crawler, hundred of crawler processes get started
« on: March 29, 2011, 01:07:14 PM »
I've been trying to find a way to start the crawler per cronjob but so far nothing has worked.

My webhoster advised me to create a separate php file which the cronjob should activate, which then will activate the crawler. Unfortunately this launches endless new processes of crawlers so that the server crashes.

The php file has the following entry:
Code: [Select]
<?phpsystem("/usr/local/php-5.3/bin/php /home/www/doc/123456/domain.com/www/sitemap/runcrawl.php");?>


My webhoster has asked if in the configurations of the crawler it can be set that it only launches once and not over and over again...

Does anyone have an idea how to solve this ?
Logged

XML-Sitemaps Support

*
  • *****
  • 11792
Re: Cronjob on a file to launch crawler, hundred of crawler processes get started
« Reply #1 on: March 29, 2011, 09:43:29 PM »
Hello,

if the script you quoted is started once, then generator will be started one time too. You should make sure that cron task is setup correctly (possibly you have set "*" for "minutes" setting, which means running every minute)
Logged
Pages: 1
« previous next »