I updated the default robots.txt file with the attached one for my Magento site but now when I run the generator it completes but with zero entry files.
Can anyone spot what is causing this error?
Re: new robots.txt file makes the generator produce zero size files.
« Reply #1 on: April 18, 2015, 02:04:39 PM »
Hello,

replied to your email: you need to remove spaces form URL part of the Disallow directives in robots.txt