ok thanks for that - couple more quick question on this "mega crawl" ;-):
1./ all the 1,000,000+ pages have a common directory name in the utrl like:
[external links are visible to admins only]
if i add "directory" to the "do not parse URL" field, will it skip anything with "directory" in the URL?
2./ I let it crawl 100,000 pages so far and then i decided to pause it and adjust the max time to 10 seconds so it would create a sitemap for what its done so far, so i clicked on crawl and resume session but now all that shows in the crawl page is:
Links depth: -
Current page: -
Pages added to sitemap: -
Pages scanned: - (- Kb)
Pages left: - (+ - queued for the next depth level)
Time passed: -
Time left: -
Memory usage: -
and nothing more and no sitemaps - any ideas?