• Welcome to Sitemap Generator Forum.
 

Data crawling issue

Started by conn_man7, January 15, 2013, 03:55:47 AM

Previous topic - Next topic

conn_man7

I've installed the sitemap generator on my domain [ External links are visible to forum administrators only ], however when crawling the data, i got 3000+ pages to crawl and somehow the pages crawled seems redundant. How do i removed the redundancy in sitemap and how do i configure the sitemap generator to only crawl the important pages and ignore the unnecessary pages?

The weird redundant link i saw in my sitemap looks like this...

<url>
       <loc>[ External links are visible to forum administrators only ]</loc>

Thanks in advance.