XML Sitemaps Generator

Author Topic: How to restrict the sitemap to pull only specific urls.  (Read 5818 times)

jenny5

  • Registered Customer
  • Approved member
  • *
  • Posts: 2
How to restrict the sitemap to pull only specific urls.
« on: February 04, 2014, 10:42:24 PM »
I want to restrict the crawl to specific pages like mydomain.com/zip/ and then I want it to pull up any page in the /zip/directory and crawl each page.. meaing if the page is full of urls leading to an mls page I want to crawl all the urls on every page that is in the /zip/ directory.. does this make any sense?
Or if I crawl the pages and get the urls.. can a huge text file of urls be fed to the sitemap generator to format and place theim all in a sitemap..

XML-Sitemaps Support

  • Administrator
  • Hero Member
  • *****
  • Posts: 10624
Re: How to restrict the sitemap to pull only specific urls.
« Reply #1 on: February 05, 2014, 07:07:23 AM »
Hello,

you can use www.domain.com/zip/ as Starting URL to crawl/index only URLs including zip/ part.
Oleg Ignatiuk
www.xml-sitemaps.com
Send me a Private Message

For maximum exposure and traffic for your web site check out our additional SEO Services.

 

SMF 2.0.12 | SMF © 2014, Simple Machines
XHTML RSS WAP2