I get this error
"There was an error while retrieving the URL specified: [external links are visible to admins only]
HTTP/1.1 403 Forbidden
date: Mon, 14 Jul 2014 20:19:12 GMT
When I try and crawl, I just upgraded my site server but the domain name is the same. I PM'd admin but I wondered if anyone else had got this and found a cure. Previously admin solved it by "empying my server IP cache in configuration" but I can't find this option
EDIT: I also tried using the IP address rather than domain name but that gives a different error which I've had before. The "Header error"
There was an error while retrieving the URL specified: /
Sorry, another edit! When I use the Home page of my site instead of the index page in configuration it works perfectly. So it must be the cache issue for the index page?