I tried disabling canonical urls. No luck.
I don't know if this helps...
If I change the Starting URL to "[external links are visible to admins only]" (a page thats linked from the home page)
The crawler produces this error:
There was an error while retrieving the URL specified: [external links are visible to admins only]
HTTP headers follow:
Tue, 29 Jun 2010 17:07:48 GMT
The requested URL /leisure was not found on this server.
Additionally, a 404 Not Found error was encountered while trying to use an ErrorDocument to handle the request.
Thanks for your help.