When I attempt a crawl it fails with this message:
An error occured
There was an error while retrieving the URL specified: [external links are visible to admins only]
HTTP headers follow:
I have of course checked that there is an index.html on my domain, and this has correct HTTP response etc.