XML Sitemaps Generator

Author Topic: Blocking sections from crawling not working.... and a few other things  (Read 8214 times)

sales850

  • Registered Customer
  • Approved member
  • *
  • Posts: 1
I ran into a few small issues that I would appreciate clarification on.

First - I had the system crawl my entire site, it was over 40,000 articles so it took a long time - ended up changing the depth because I had left it at unlimited. (But, it had reached 60,000+ which was odd).

Anyway - after I changed the depth to 100, now there's only 6519 pages crawled... (strange!)

This is an enormous article directory and has been live since 1998 - so obviously I'd like the articles to be showing properly.

Also - I told it to EXCLUDE the /cgi-bin/ directory, but it still provided many urls linked in it! THAT is a no no to me. So how do I get it to exclude completely the cgi-bin folders? I have over 120 categories that I want to be included... and not much else. Do I really have to manually add all 120+? I'd rather post the few "exclude" urls - it's faster, but obviously something isn't right.


Amber

XML-Sitemaps Support

  • Administrator
  • Hero Member
  • *****
  • Posts: 10624
Hello,

1. you can reverse the depth level to 150 in this case.

2. how did you specify it in Exclude URLs setting and what is an example URL that is still added in sitemap?
Oleg Ignatiuk
www.xml-sitemaps.com
Send me a Private Message

For maximum exposure and traffic for your web site check out our additional SEO Services.

 

SMF 2.0.12 | SMF © 2014, Simple Machines
XHTML RSS WAP2