XML Sitemaps Generator

Author Topic: URLs with no file extensions are not crawled  (Read 19772 times)


  • Approved member
  • *
  • Posts: 1
URLs with no file extensions are not crawled
« on: October 09, 2012, 07:59:39 AM »

I'm evaluating your product for my client, and when I tried to crawl his site with your Free Online Sitemap Generator, the tool seems to pick up only the links with file extensions.

So using this page as an example: [external links are visible to admins only]

The tool only picked out 3 links with .php extensions when the entire page is full of links. Am I using the tool incorrectly or is it not meant to handle so many links?

Most of the other links on the site are url-rewritten on a IIS 7 server - I'm not sure if it makes any difference to the results.

Thank you.

XML-Sitemaps Support

  • Administrator
  • Hero Member
  • *****
  • Posts: 10599
Re: URLs with no file extensions are not crawled
« Reply #1 on: October 09, 2012, 11:26:02 PM »

if you specify starting URL as http://www.domain.com/customer_service/sitemap.php generator will include only URLs from the same folder level and below (ie.e. starting with "http://www.domain.com/customer_service/..").

Try to use just http://www.domain.com instead.
Oleg Ignatiuk
Send me a Private Message

For maximum exposure and traffic for your web site check out our additional SEO Services.


SMF 2.0.12 | SMF © 2014, Simple Machines