XML Sitemaps Generator

Author Topic: Directory crawling problem  (Read 17375 times)

Galaxi Designs

  • Registered Customer
  • Approved member
  • *
  • Posts: 3
Directory crawling problem
« on: December 05, 2007, 11:22:17 PM »
Hello. For some reason my unlimited sitemap generator crawls my forum and store sub directories, but not my articles and blog sub directories. They are linked on the home page just like the rest of my directories, but they just will not show up in my sitemap. Also, my articles and blog sub directories have an htaccess file in them, that re-writes URL's to be more search engine friendly. Could this be why?

Any help would be appreciated, thank you.

Sincerely,
Michael Cottier

XML-Sitemaps Support

  • Administrator
  • Hero Member
  • *****
  • Posts: 10622
Re: Directory crawling problem
« Reply #1 on: December 06, 2007, 12:55:06 AM »
Hello,

please make sure that you don't have them excluded in robots.txt or via <meta name="robots"> tag in html sorce code.
Oleg Ignatiuk
www.xml-sitemaps.com
Send me a Private Message

For maximum exposure and traffic for your web site check out our additional SEO Services.

Galaxi Designs

  • Registered Customer
  • Approved member
  • *
  • Posts: 3
Re: Directory crawling problem
« Reply #2 on: December 06, 2007, 01:23:51 AM »
Well on all of my pages I do not have any meta name="robots" tag, and in my robots file I have this:

User-agent: *
Disallow: /cgi-bin/

Do you know what I should do or change to fix it?

XML-Sitemaps Support

  • Administrator
  • Hero Member
  • *****
  • Posts: 10622
Re: Directory crawling problem
« Reply #3 on: December 07, 2007, 01:13:52 AM »
Please let me know your generator URL and an example URL that is not included in sitemap via private message.
Oleg Ignatiuk
www.xml-sitemaps.com
Send me a Private Message

For maximum exposure and traffic for your web site check out our additional SEO Services.

 

SMF 2.0.12 | SMF © 2014, Simple Machines
XHTML RSS WAP2