Crawler not accessing all directories
« on: July 07, 2011, 03:24:22 AM »
Can you please review http://www.ozpwc.com (wordpress site) and see that there is a vbulletin forum in the directory 'forum'.

I have the crawler set to unlimited on pretty much all settings but for some reason I can never get it to retrieve the forum directory in its results.

The 'classifieds' directory is accessed ok and spits out lots of links for that directory but I'm trying to understand why I cannot even get the directory www.ozpwc.com/forum to be recognised and mapped?

You can view the current xml map at:

http://www.ozpwc.com/sitemap.xml

Would appreciate your assistance.

with thanks

Mike
Re: Crawler not accessing all directories
« Reply #1 on: July 07, 2011, 05:45:55 PM »
Hello,

you have /forum folder disallowed in your robots.txt:
Disallow: /forum/
Oleg Ignatiuk
www.xml-sitemaps.com
Send me a Private Message

For maximum exposure and traffic for your web site check out our additional SEO Services.
Re: Crawler not accessing all directories
« Reply #2 on: July 07, 2011, 10:52:51 PM »
bugga....sorry to waste your time!
Re: Crawler not accessing all directories
« Reply #3 on: July 09, 2011, 10:28:46 AM »
No problem, glad to help!
Oleg Ignatiuk
www.xml-sitemaps.com
Send me a Private Message

For maximum exposure and traffic for your web site check out our additional SEO Services.