How to restrict the sitemap to pull only specific urls.
« on: February 04, 2014, 10:42:24 PM »
I want to restrict the crawl to specific pages like mydomain.com/zip/ and then I want it to pull up any page in the /zip/directory and crawl each page.. meaing if the page is full of urls leading to an mls page I want to crawl all the urls on every page that is in the /zip/ directory.. does this make any sense?
Or if I crawl the pages and get the urls.. can a huge text file of urls be fed to the sitemap generator to format and place theim all in a sitemap..
Re: How to restrict the sitemap to pull only specific urls.
« Reply #1 on: February 05, 2014, 07:07:23 AM »
Hello,

you can use [ External links are visible to logged in users only ] as Starting URL to crawl/index only URLs including zip/ part.