Crawling pages that require authentication
« on: May 09, 2012, 12:22:01 AM »
Is there a way to set the crawler up in a way that it can use a user name/password to crawl through my site that has specific authenticated pages that require login?

Thanks
Re: Crawling pages that require authentication
« Reply #1 on: May 10, 2012, 08:44:16 PM »
Generator bot crawls pages similar to search engine bots, so it can't access pages if they are password protected.