Directory crawling problem
« on: December 05, 2007, 11:22:17 PM »
Hello. For some reason my unlimited sitemap generator crawls my forum and store sub directories, but not my articles and blog sub directories. They are linked on the home page just like the rest of my directories, but they just will not show up in my sitemap. Also, my articles and blog sub directories have an htaccess file in them, that re-writes URL's to be more search engine friendly. Could this be why?

Any help would be appreciated, thank you.

Sincerely,
Michael Cottier
Re: Directory crawling problem
« Reply #1 on: December 06, 2007, 12:55:06 AM »
Hello,

please make sure that you don't have them excluded in robots.txt or via <meta name="robots"> tag in html sorce code.
Re: Directory crawling problem
« Reply #2 on: December 06, 2007, 01:23:51 AM »
Well on all of my pages I do not have any meta name="robots" tag, and in my robots file I have this:

User-agent: *
Disallow: /cgi-bin/

Do you know what I should do or change to fix it?
Re: Directory crawling problem
« Reply #3 on: December 07, 2007, 01:13:52 AM »
Please let me know your generator URL and an example URL that is not included in sitemap via private message.