Crawling Xenforo forum
« on: April 19, 2013, 09:38:22 AM »
Hi,

I recently moved from phpBB to Xenforo.  I had the phpBB exclusions set and my sitemap took less than 30 mins to complete.

Since the Xenforo migration the crawl is taking at least 4 hours to run.  Do you have any exclusions or will I need to try and figure them myself?

Also, is there a way of cancelling a crawl as the sitemap seems to fail more often than not now - probably due to the server resources when it's running.

Thanks.
Re: Crawling Xenforo forum
« Reply #1 on: April 19, 2013, 01:08:41 PM »
I think I have found a better way now, so this isn't really an issue. But for anyone else searching this site for Xenforo might find it useful.

There is a sitemap module which can be installed in to Xenforo which is configured to index just the forum. 

Reconfigured this generator to index everything except the forum (mainly the WordPress site).

Now have two much smaller and more manageable sitemaps in my robots.txt.  Both read nicely by both Google and Bing.

Sitemap: [ External links are visible to forum administrators only ]
Sitemap: [ External links are visible to forum administrators only ]

The Xenforo map takes less than 30 secs, and the sitemap generator takes around 6 minutes.

Cheers, Gary