Exclusion preset for IPB 3.x.x
« on: February 28, 2010, 01:12:12 AM »
Hello, Im a new happy customer of XML Sitemaps, just got this yesterday and created my first sitemap today.
Everything went just fine but I have a couple of questions.

after the scan I reviewed the broken/404 links and most of them are not really "broken" I believe the robots.txt file is blocking them, because I can open them when I click on them.

I ran XML Sitemap on my IPB site, and I wanted to know if there are exclusions presets for IPB, I searched the forums but didnt see any.

if not, this is my robots.txt, can I create them from the robot.txt so they get excluded and dont show up as broken links?
Thank You for the great product!

Code: [Select]
User-agent: *

Disallow: /admin/

Disallow: /public/js/

Disallow: /public/style_captcha/

Disallow: /public/style_css/

Disallow: /index.php?app=core&module=task

Disallow: /index.php?app=core&module=usercp&tab=forums&area=forumsubs

Disallow: /index.php?app=core&module=usercp&tab=forums&area=watch&watch=topic

Disallow: /index.php?app=forums&module=extras&section=forward

Disallow: /index.php?app=members&module=messaging

Disallow: /index.php?app=members&module=chat

Disallow: /index.php?app=members&module=search

Disallow: /index.php?app=members&module=search&do=active

Disallow: /index.php?&unlockUserAgent=1

Disallow: /*app=core&module=usercp

Disallow: /*app=core&module=usercp

Disallow: /*app=members&module=messaging

Disallow: /*&p=

Disallow: /*&pid=

Disallow: /*&hl=

Disallow: /*&start=

Disallow: /*page__*$

Disallow: /*view__*$

Disallow: /*&view=*$

Disallow: /calendar/

Sitemap: http:// mysite/sitemap_index.xml.gz
Re: Exclusion preset for IPB 3.x.x
« Reply #1 on: March 01, 2010, 01:15:21 PM »
Hello,

generator reads the robots.txt file, so as long as you have them disallows in robots file, no need to define them in Exclusions list.