XML Sitemaps Generator

Author Topic: crawling problem  (Read 19044 times)

casper

  • Approved member
  • *
  • Posts: 1
crawling problem
« on: January 03, 2008, 03:41:10 AM »
Hi,

I  had the Standalone XML Sitemap Generator and would like to ask how to fix this problem because every time i run the crawling it generate in the sitemap the 2 different link of that same folder one with index.html and the other one is without index.html, for example this url :
[external links are visible to admins only] and the other one is  [external links are visible to admins only]

 ??? which is wrong is should only generate only 1 link not 2

THANKS,
CASPER
 


XML-Sitemaps Support

  • Administrator
  • Hero Member
  • *****
  • Posts: 10623
Re: crawling problem
« Reply #1 on: January 04, 2008, 12:55:39 AM »
Hello,

you should modify your site to only include one of the links, either "directory/" or "directory/index.html". If you use both of them, Sitemap Generator will find them too.
Oleg Ignatiuk
www.xml-sitemaps.com
Send me a Private Message

For maximum exposure and traffic for your web site check out our additional SEO Services.

info445

  • Registered Customer
  • Approved member
  • *
  • Posts: 1
Re: crawling problem
« Reply #2 on: February 28, 2008, 02:00:04 PM »
hello,

is there another way? I would hate to have to search my site to find all the occurences.

So is there a way to tell the script to ignore the link without index?

Like accepting only url with .php .html and .htm?

thank you in advance

nicolas

XML-Sitemaps Support

  • Administrator
  • Hero Member
  • *****
  • Posts: 10623
Re: crawling problem
« Reply #3 on: February 28, 2008, 11:44:00 PM »
That would be the only reliable way to avoid indexing of both URLs, since search engines will be able to find both links anyway, even if you will include only one in sitemap.
Oleg Ignatiuk
www.xml-sitemaps.com
Send me a Private Message

For maximum exposure and traffic for your web site check out our additional SEO Services.

 

SMF 2.0.12 | SMF © 2014, Simple Machines
XHTML RSS WAP2