• Welcome to Sitemap Generator Forum.
 

Import robots.txt into sitemap generator, to exclude pages from being crawled?

Started by mcnam007, August 10, 2018, 08:14:30 PM

Previous topic - Next topic

mcnam007

My robots.txt file is designed to exclude a variety of types of pages from being crawled by various bots.  Is there a way to import my robots.txt into the xmlgenerator, so that it excludes the same types of page?  Specifically, I'd like to exclude those with the following words in the URLs:
/category/
/feed/
/pages/
and any pages with "embed" in the URL.

Thanks!

XML-Sitemaps Support

Hello,

sitemap generator should detect robots.txt directives automatically, please let me know your generator URL/login private message to check this.

mcnam007


info2500

I have the same issue... my robots.txt is not detected properly.
All the disallowed routes are being parsed.
How to properly setup the robots.txt for the sitemap generator.
Where does it have to be available from?


dek2003

Hello.

I have the same problem - robots.php directives are ignored when generating sitemap