Generator starts and stops, have 500k pages to crawl
« on: December 27, 2009, 08:09:06 PM »
I have a windows server.  I have the generator to pause 60 seconds after each 1000 pages.  It crawls for a while then stops.  I see other people have had same issue, but then you pm them for help.  Can you help me with this?
Re: Generator starts and stops, have 500k pages to crawl
« Reply #1 on: December 27, 2009, 10:36:23 PM »
Hello,

having 60 seconds delay is too long, a few seconds is sufficient.
Oleg Ignatiuk
www.xml-sitemaps.com
Send me a Private Message

For maximum exposure and traffic for your web site check out our additional SEO Services.