Blocking specific pages/directories from getting crawled

Sitemap Generator Forum

December 05, 2008, 11:10:08 AM
Welcome, Guest. Please login or register.
Did you miss your activation email?

Home

Help

Sitemap software 2.9 released - Email notifications, html sitemap customizing and more

8403 Posts in 2075 Topics by Members
Latest Member: ovidiu.suciu

Sitemap Generator Forum > XML Sitemaps Discussions Category > Unlimited PHP Sitemap Generator > Blocking specific pages/directories from getting crawled

Pages: [1]

« previous next »

Author

Topic: Blocking specific pages/directories from getting crawled (Read 4443 times)

yacosta

Registered Customer
Newbie

Posts: 1

Blocking specific pages/directories from getting crawled

« on: July 25, 2007, 05:55:42 PM »

Hi is there a way to block specific pages/directories from being crawled?


	Logged

admin

Administrator
Hero Member

Posts: 3530

Re: Blocking specific pages/directories from getting crawled

« Reply #1 on: July 26, 2007, 12:24:58 AM »

Hello,

yes, you can block them using robots.txt or just define them in "Exclude pages" option in Sitemap generator configuration.


	Logged

Oleg Ignatiuk
http://www.xml-sitemaps.com

Pages: [1]

« previous next »