Excellent little script. Everything works fine but I've got the problem that only the links on my [external links are visible to admins only] are indexed.
I have a directory [external links are visible to admins only] which has more subdirectories with maybe 3000 html pages and only like 60 get indexed from there, I need my main page to get listed as well as the /pages/ directory and it's subdirectories full of html pages ! !
What do I do to get this to work like that ?