Cron jobs and crawling problem
« on: July 13, 2009, 04:28:17 PM »
In a perfect world, I'd have xml-sitemaps automatically crawl and update my sitemap.xml automatically every week, without my having to do anything. I think this is a "cron job"... can this be done? I looked at documentation for cron jobs but I don't understand how to use this feature... it gives me a command line - how do I use the command line?

Currently EVERYtime I manually run the crawl, if times-out after approx 1,100 pages crawled (it just stops crawling but the screen doesn't actually tell me that it has stopped). I've tried waiting up to 12 hours but it doesn't resume. So... I figured out that whenever it stops, I can just click 'crawling' tab again, and then 'resume' last session. Ok, so that's the "work-around" for the problem... but if I set up a cron, will that time-out also?

Re: Cron jobs and crawling problem
« Reply #1 on: July 13, 2009, 09:37:53 PM »

scheduled  task (cron job) is usually setup in hosting control panel. Exact steps would depend on control panel software, but normally there is a special section called "cron jobs", where you will enter that command line and specify how often to run the task.