restarting from command line
« on: November 15, 2008, 03:41:05 PM »
When running runcrawl.php from command line, if the program stops for some reason, is there a way to restart from command line and save the urls that are already in crawl_dump.log? -- like the restart option on the  "crawling" screen??

My crawl_dump.log is currently 260Mb and has 1.5 million lines.

I have already used the exclude config to keep it as small as I can.

Re: restarting from command line
« Reply #1 on: November 15, 2008, 07:28:24 PM »

when you execute sitemap generator in command line with runcrawl.php script, it automatically resumes generation in case if crawl_dump.log is found.