I ran into a few small issues that I would appreciate clarification on.
First - I had the system crawl my entire site, it was over 40,000 articles so it took a long time - ended up changing the depth because I had left it at unlimited. (But, it had reached 60,000+ which was odd).
Anyway - after I changed the depth to 100, now there's only 6519 pages crawled... (strange!)
This is an enormous article directory and has been live since 1998 - so obviously I'd like the articles to be showing properly.
Also - I told it to EXCLUDE the /cgi-bin/ directory, but it still provided many urls linked in it! THAT is a no no to me. So how do I get it to exclude completely the cgi-bin folders? I have over 120 categories that I want to be included... and not much else. Do I really have to manually add all 120+? I'd rather post the few "exclude" urls - it's faster, but obviously something isn't right.