XML Sitemaps Generator

Author Topic: Crawling pages that require authentication  (Read 8851 times)

soulesschild

  • Approved member
  • *
  • Posts: 1
Crawling pages that require authentication
« on: May 08, 2012, 11:22:01 PM »
Is there a way to set the crawler up in a way that it can use a user name/password to crawl through my site that has specific authenticated pages that require login?

Thanks

XML-Sitemaps Support

  • Administrator
  • Hero Member
  • *****
  • Posts: 10622
Re: Crawling pages that require authentication
« Reply #1 on: May 10, 2012, 07:44:16 PM »
Generator bot crawls pages similar to search engine bots, so it can't access pages if they are password protected.
Oleg Ignatiuk
www.xml-sitemaps.com
Send me a Private Message

For maximum exposure and traffic for your web site check out our additional SEO Services.

 

SMF 2.0.12 | SMF © 2014, Simple Machines
XHTML RSS WAP2