XML Sitemaps and Vbulletin

[mudanoman]

Hi,

Sounds like a great piece of software,  only question I have is how well it works for vbulletin and what files would I need to specifiy to exclude.

Thanks,

Ivan

[XML-Sitemaps Support]

Hello Ivan,

it should work well with vBulletin, the files to exclude depend on what do you keep included in sitemap, but the basic list is:

attachment.php
calendar.php
cron.php
editpost.php
image.php
member.php
memberlist.php
misc.php
newattachment.php
newreply.php
newthread.php
online.php
private.php
profile.php
register.php
search.php
usercp.php

[mudanoman]

Thanks,

Just to make sure I just throw the above into "Exclude URLs:" box, right?  Just trying to figure out how to limit crawling as my 500 post vbb has 4,000 plus pages on queue. 

TIA,

Ivan

[XML-Sitemaps Support]

Hello Ivan,

you should add this to both "Exclude URLs" and "Do not parse" options.

[Danny Weiss]

I run vBulletin and found that my scan time is greatly reduced by excluding URLs with just the following three or four files:
- calendar.php
- memberlist.php
- search.php
and possibly
- forumdisplay.php

Calendar.php made the biggest difference, followed by memberlist.php.

-Danny.

[XML-Sitemaps Support]

Thank you for information, Danny. I believe that calendar.php creates many pages (for every day/week/month etc) and it's good to exclude them indeed.
However, I would not suggest to remove forumdisplay.php pages as they are important and it's better to get them included in sitemap/indexed in search engines.

[Dave A]

Apart from putting in the contents of my robots.txt file, I added & as not to be parsed and URL's to exclude. This stripped a result of approx 5000 URLs down to about 1500 - which at a guess shoulld about cover everything at least once at this point, and is about 50% higher than Google's current indexed pages for my site.

I'm counting on 2 factors in doing this:
Google is still going to crawl additional links from the pages found in the sitemap.xml , and
the showthread?p= is going to cover subsequent pages of multipage threads.

I take the view that Google will only index as much as they deem fit - the goal of the sitemap is to make sure they index the right stuff.

Hopefully I'm not too far off the mark.