I have over 100,000 articles in my directory, but it only crawls the index page, and gets 66 URL's, which is about the same number of actual catergories. It does not drill down anywhere from the main page. I know they can be indexed as Google has done over 10,000.
All pages are dynamically produced, so a URL of an article may look like:
No ROBOT code in txt or META
I've been piddling about with the software for around 2 hours and have lost the will to live
Any help appreciated.