XML Sitemaps Generator

Author Topic: Data crawling issue  (Read 31302 times)


  • Registered Customer
  • Approved member
  • *
  • Posts: 1
Data crawling issue
« on: January 15, 2013, 03:55:47 AM »
I've installed the sitemap generator on my domain [external links are visible to admins only], however when crawling the data, i got 3000+ pages to crawl and somehow the pages crawled seems redundant. How do i removed the redundancy in sitemap and how do i configure the sitemap generator to only crawl the important pages and ignore the unnecessary pages?

The weird redundant link i saw in my sitemap looks like this...

       <loc>[external links are visible to admins only]</loc>

Thanks in advance.

XML-Sitemaps Support

  • Administrator
  • Hero Member
  • *****
  • Posts: 10599
Re: Data crawling issue
« Reply #1 on: January 15, 2013, 07:11:01 PM »
please try to add this  in "Exclude URLs" setting:
Oleg Ignatiuk
Send me a Private Message

For maximum exposure and traffic for your web site check out our additional SEO Services.


SMF 2.0.12 | SMF © 2014, Simple Machines