Is this Sitemap Generator cannot crawl URL without filename?
« on: March 20, 2012, 03:59:19 PM »
We tried to crawl our site and seems its ok with the URL with filename like [ External links are visible to forum administrators only ]

However our site does .htaccess rewrite and thus some of the URL would be just the directory name like

[ External links are visible to forum administrators only ]
[ External links are visible to forum administrators only ]

which we found those could not be crawled.

May I know if there is any setting to fix this behavior?
Re: Is this Sitemap Generator cannot crawl URL without filename?
« Reply #1 on: March 21, 2012, 02:39:25 AM »
Hi, anyone can help here?

My site is [ External links are visible to forum administrators only ] and all the URLs end with .html are ok but those just the directory name cannot be crawled as shown below:

[ External links are visible to forum administrators only ]
[ External links are visible to forum administrators only ]
[ External links are visible to forum administrators only ]
etc.

Please help and thanks in advance.