XML Sitemaps Generator

Author Topic: robots.txt file  (Read 240 times)

dennis@dhl

  • Registered Customer
  • Approved member
  • *
  • Posts: 9
robots.txt file
« on: August 09, 2017, 03:29:45 PM »
I have just installed version 7.2 on my subdomain. When I  run the generator, it runs for about 2 seconds and then finishes and no urls in the .xml file.
I have looked around your forum and it looks like the issue is with my robots.txt file which disallows any crawling with this
User-agent: *
Disallow: /
Is there a way to stop bots crawling my site but allow the generator to run?

Thanks

XML-Sitemaps Support

  • Administrator
  • Hero Member
  • *****
  • Posts: 10849
Re: robots.txt file
« Reply #1 on: August 09, 2017, 07:43:07 PM »
Hello,

you can disable "Support robots.txt" setting.
Oleg Ignatiuk
www.xml-sitemaps.com
Send me a Private Message

For maximum exposure and traffic for your web site check out our additional SEO Services.

dennis@dhl

  • Registered Customer
  • Approved member
  • *
  • Posts: 9
Re: robots.txt file
« Reply #2 on: August 10, 2017, 09:56:34 AM »
Thanks for that, I didn't know about that setting.

I found 'xs_robotstxt' => '1' in the config file so I have tried changing it to '0' and '' but still the same issue

Thanks

XML-Sitemaps Support

  • Administrator
  • Hero Member
  • *****
  • Posts: 10849
Re: robots.txt file
« Reply #3 on: August 10, 2017, 12:10:56 PM »
Hello,

please let me know your generator URL/login in private message to check this.
Oleg Ignatiuk
www.xml-sitemaps.com
Send me a Private Message

For maximum exposure and traffic for your web site check out our additional SEO Services.

dennis@dhl

  • Registered Customer
  • Approved member
  • *
  • Posts: 9
Re: robots.txt file
« Reply #4 on: August 10, 2017, 12:33:33 PM »
I have sent you a private message as requested

dennis@dhl

  • Registered Customer
  • Approved member
  • *
  • Posts: 9
Re: robots.txt file
« Reply #5 on: August 15, 2017, 11:35:05 AM »
Have you had a chance ye to look at this issue ?

XML-Sitemaps Support

  • Administrator
  • Hero Member
  • *****
  • Posts: 10849
Re: robots.txt file
« Reply #6 on: August 15, 2017, 10:27:59 PM »
Hello,

a reply was sent to you the same day, please check the inbox.
Oleg Ignatiuk
www.xml-sitemaps.com
Send me a Private Message

For maximum exposure and traffic for your web site check out our additional SEO Services.

 

SMF 2.0.14 | SMF © 2017, Simple Machines
XHTML RSS WAP2