phpBB Locked forums not getting crawled

[robyrulz1]

Hi there!!

First of all  thanx for creating such a nice piece of code..

i have an issue with this script, i've bought this script mainly to crawl and create a sitemap of  my phpBB3 topics and posts.
I have some forums (that's how it's called in phpBB world) in phpBB which are locked and cannot be viewed by any guest... actually these forums can be viewed, but not read without entering u/n & password...

for example:
[external links are visible to admins only]

phpBB has a built in spiders/bots management sytem that will let them browse this forum internally, by doing some cross checking of agent signatures..

but how will google bot come to know about my pages when they are not included in the sitemap???

is there any way out for this??
how can i allow Standalone Sitemap generator script to access topics within locked forums???

[XML-Sitemaps Support]

Hello,

you can set this option in config.inc.php file to let generator crawl with User-agent set as Google bot:

Code: [Select]

'xs_crawl_ident' => 'Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)'
However, please note that hiding pages for regular visitors and showing them to SE bots might be considered by search engines as cloaking: http://en.wikipedia.org/wiki/Cloaking

[robyrulz1]

Thanx for the reply..

i read the post on Cloaking on Wikipedia.. is it harmful for my organization??
I've seen numerous sites indexed in google(that probably means google bot has access to their pages), but when a legitimate user (like u and me) visits the same page, it asks for a username and a password???
Is this a violation of SearchEngines Rules&Regulations...??

And what if i want to add more Bots?? what's the syntax??

And one thing more>
I've submitted a sitemap with 1146 URLS, out of which not even a single URL is indexed by google yet?
I saw these statistics in Google Webmaster Tools Dashboard

Shall i wait for my URL's to get indexed and appear in Google's search listings, or is there anything else required???

[XML-Sitemaps Support]

Hello,

1. That would work to some extent (until search engine will find out that), but potentially this would do more harm than good.

2. You should only set it to one bot type and sitemap generator will use it to crawl your site, as a result it will be allowed to access your private subforums.

3. It takes some time to get indexed, but the issue might be even related to point #1 above. I'd recommend to open the forums to avoid possible issue.

[robyrulz1]

Hello,

Just checked the statistics on Google Webmasters Tools, 119 URL's got indexed out of 1150..
this probably means that indexing is going on.. Thanx once again!!!

I'm so happy!!!

[XML-Sitemaps Support]

Yes, looks good!

[shop34]

Hello,

you can set this option in config.inc.php file to let generator crawl with User-agent set as Google bot:

Code: [external links are visible to admins only]

'xs_crawl_ident' => 'Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)'
However, please note that hiding pages for regular visitors and showing them to SE bots might be considered by search engines as cloaking: [external links are visible to admins only]

Hello!

I am new to the forum and I apologize first for my bad English!

I have the same problem described in this thread, but I can not find the file config.inc.php to make the change.
The version I installed is the: Standalone Sitemap Generator (PHP) v4.0, 2010-05-20

The address of my website is: [external links are visible to admins only]
while the address of the Forum: [external links are visible to admins only]

Thanks in advance who will help me!!

[XML-Sitemaps Support]

Hello,

please let me know your generator URL/login in private message to check this.

[shop34]

Hello,

please let me know your generator URL/login in private message to check this.

Private Message sent. I'm hoping for good news

[XML-Sitemaps Support]

In Sitemap Generator v4.x please try to add this setting in generator/data/generator.conf file:

Code: [Select]

<option name="xs_crawl_ident">Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)</option>

[shop34]

Thanks for the quick response.
I tried to update the file by adding the recommended setting.

Unfortunately phpBB3 continues to see the crawler as "Guest" and not as "Bots" 
In fact, in whois crawler is indicated:
Mozilla/5.0 (compatible; XML Sitemaps Generator; http://www.xml-sitemaps.com)
Gecko XML-Sitemaps/1.0 

The sitemap is still no hidden forums guests: some ideas on how to solve?
Thanks for your interest.

[XML-Sitemaps Support]

I've updated the code in my post above, please try it.

[shop34]

Thanks it works perfectly! 

Great job, congratulations!