400 Bad Request
« on: March 10, 2015, 04:12:31 PM »

I have about 2000 links but now it only gets 270 links in my site map. I check the log and realise that there is a lot of bad request. How can I resolve this?

Re: 400 Bad Request
« Reply #1 on: March 10, 2015, 04:29:43 PM »

it's possible that the server couldn't handle the number of requests generator crawler is sending. You can try to use "Make delay" setting to slow down crawling and avoid overloading server.