Is this Sitemap Generator cannot crawl URL without filename?
« on: March 20, 2012, 03:59:19 PM »
We tried to crawl our site and seems its ok with the URL with filename like http://www.domain.com/customer-service.html

However our site does .htaccess rewrite and thus some of the URL would be just the directory name like

http://www.domain.com/customer-service/
http://www.domain.com/faq/

which we found those could not be crawled.

May I know if there is any setting to fix this behavior?
Re: Is this Sitemap Generator cannot crawl URL without filename?
« Reply #1 on: March 21, 2012, 02:39:25 AM »
Hi, anyone can help here?

My site is http://www.mobimega.com and all the URLs end with .html are ok but those just the directory name cannot be crawled as shown below:

http://www.mobimega.com/about-us
http://www.mobimega.com/contacts
http://www.mobimega.com/customer-service
etc.

Please help and thanks in advance.
Oleg Ignatiuk
www.xml-sitemaps.com
Send me a Private Message

For maximum exposure and traffic for your web site check out our additional SEO Services.