• Welcome to Sitemap Generator Forum.
 

robots.txt file

Started by dennis@dhl, August 09, 2017, 03:29:45 PM

Previous topic - Next topic

dennis@dhl

I have just installed version 7.2 on my subdomain. When I  run the generator, it runs for about 2 seconds and then finishes and no urls in the .xml file.
I have looked around your forum and it looks like the issue is with my robots.txt file which disallows any crawling with this
User-agent: *
Disallow: /
Is there a way to stop bots crawling my site but allow the generator to run?

Thanks


dennis@dhl

Thanks for that, I didn't know about that setting.

I found 'xs_robotstxt' => '1' in the config file so I have tried changing it to '0' and '' but still the same issue

Thanks


dennis@dhl

I have sent you a private message as requested

dennis@dhl

Have you had a chance ye to look at this issue ?