Another piece of info
this comes up in the log file once the crawling is done, which btw takes hours???
<title>502 Bad Gateway</title>
<body bgcolor=#ffffff link=#CC0D2F vlink=#CC0D2F alink=#CC0D2F>
An error occurred while communicating with another application or
an upstream server.