XML Sitemaps Generator

Author Topic: Crawler not accessing all directories  (Read 7715 times)

paypal848

  • Registered Customer
  • Approved member
  • *
  • Posts: 3
Crawler not accessing all directories
« on: July 07, 2011, 02:24:22 AM »
Can you please review [external links are visible to admins only] (wordpress site) and see that there is a vbulletin forum in the directory 'forum'.

I have the crawler set to unlimited on pretty much all settings but for some reason I can never get it to retrieve the forum directory in its results.

The 'classifieds' directory is accessed ok and spits out lots of links for that directory but I'm trying to understand why I cannot even get the directory [external links are visible to admins only] to be recognised and mapped?

You can view the current xml map at:

[external links are visible to admins only]

Would appreciate your assistance.

with thanks

Mike

XML-Sitemaps Support

  • Administrator
  • Hero Member
  • *****
  • Posts: 10622
Re: Crawler not accessing all directories
« Reply #1 on: July 07, 2011, 04:45:55 PM »
Hello,

you have /forum folder disallowed in your robots.txt:
Disallow: /forum/
Oleg Ignatiuk
www.xml-sitemaps.com
Send me a Private Message

For maximum exposure and traffic for your web site check out our additional SEO Services.

paypal848

  • Registered Customer
  • Approved member
  • *
  • Posts: 3
Re: Crawler not accessing all directories
« Reply #2 on: July 07, 2011, 09:52:51 PM »
bugga....sorry to waste your time!

XML-Sitemaps Support

  • Administrator
  • Hero Member
  • *****
  • Posts: 10622
Re: Crawler not accessing all directories
« Reply #3 on: July 09, 2011, 09:28:46 AM »
No problem, glad to help!
Oleg Ignatiuk
www.xml-sitemaps.com
Send me a Private Message

For maximum exposure and traffic for your web site check out our additional SEO Services.

 

SMF 2.0.12 | SMF © 2014, Simple Machines
XHTML RSS WAP2