• Welcome to Sitemap Generator Forum.
 

Crawling pages that require authentication

Started by soulesschild, May 09, 2012, 12:22:01 AM

Previous topic - Next topic

soulesschild

Is there a way to set the crawler up in a way that it can use a user name/password to crawl through my site that has specific authenticated pages that require login?

Thanks

XML-Sitemaps Support

Generator bot crawls pages similar to search engine bots, so it can't access pages if they are password protected.