URLs with no file extensions are not crawled
« on: October 09, 2012, 08:59:39 AM »
Hi,

I'm evaluating your product for my client, and when I tried to crawl his site with your Free Online Sitemap Generator, the tool seems to pick up only the links with file extensions.

So using this page as an example: [ External links are visible to forum administrators only ]

The tool only picked out 3 links with .php extensions when the entire page is full of links. Am I using the tool incorrectly or is it not meant to handle so many links?

Most of the other links on the site are url-rewritten on a IIS 7 server - I'm not sure if it makes any difference to the results.

Thank you.
Re: URLs with no file extensions are not crawled
« Reply #1 on: October 10, 2012, 12:26:02 AM »
Hello,

if you specify starting URL as http://www.domain.com/customer_service/sitemap.php generator will include only URLs from the same folder level and below (ie.e. starting with "http://www.domain.com/customer_service/..").

Try to use just http://www.domain.com instead.