Exclude a folder
« on: September 14, 2006, 03:05:52 AM »
In this topic, we were advised that an entire folder can be excluded:

https://www.xml-sitemaps.com/forum/index.php/topic,200.html

...such as:

[ External links are visible to forum administrators only ]

...but it appears that the crawling still includes the folder, thus taking a long time to crawl the folder anyway.

Is this right? An excluded folder is crawled but not included in the sitemap?

Don Strack
Re: Exclude a folder
« Reply #1 on: September 14, 2006, 03:27:58 AM »
Hello Don,

you should include it in both "Do not parse URLs" and "Exclude URLs" options for this.
Re: Exclude a folder
« Reply #2 on: September 14, 2006, 04:13:56 AM »
Thanks. But I must be doing it wrong. The folder is still being crawled. And takes at least 10 times as long because it is crawling this folder.

I have included the following in both the "Do Not Parse URL" field, and in the "Exclude URL" field:

[ External links are visible to forum administrators only ]

Apparently, a wildcard does not work either.

Please help.

Don Strack
Re: Exclude a folder
« Reply #3 on: September 14, 2006, 11:51:20 PM »
It's working now, so to close this thread, here is what I learned.

I was adding the entire "http:// mydomain.com/folder/" address, instead of just the folder, like this: "folder/".

I did a search on "exclude", rather than "folder" and found a link to the documentation.

Now I have some issues with the markup of the html sitemap, but that's for another thread.

Don Strack