Problem with the crawler
« on: July 19, 2008, 09:13:52 PM »
hi... i got a problem with the sitemap generator.

when i click on "crwaling" to generate my sitemap the genrator tries to crawl the following url:
Links depth: 184
Current page: nudeblog/category/nudeblog/category/nudeblog/category/nudeblog/category/nudeblog/category/nudeblog/category/nudeblog/category/nudeblog/category/nudeblog/category/nudeblog/category/nudeblog/category/nudeblog/category/nudeblog/category/nudeblog/category/nudeblog/category/nudeblog/category/nudeblog/category/nudeblog/category/nudeblog/category/nudeblog/category/nudeblog/category/nudeblog/category/nudeblog/category/nudeblog/category/nudeblog/category/nudeblog/category/nudeblog/category/nudeblog/category/nudeblog/category/nudeblog/category/nudeblog/category/nudeblog/category/nudeblog/category/nudeblog/category/nudeblog/category/nudeblog/category/nudeblog/category/nudeblog/category/nudeblog/category/nudeblog/category/nudeblog/category/nudeblog/category/nudeblog/category/nudeblog/category/nudeblog/category/nudeblog/category/nudeblog/category/nudeblog/category/nudeblog/category/nudeblog/category/nudeblog/category/nudeblog/category/nudeblog/category/nudeblog/category/nudeblog/category/nudeblog/category/nudeblog/category/nudeblog/category/nudeblog/category/nudeblog/category/nudeblog/category/nudeblog/category/nudeblog/category/nudeblog/category/nudeblog/category/nudeblog/category/nudeblog/category/nudeblog/category/nudeblog/category/nudeblog/category/nudeblog/category/nudeblog/category/nudeblog/category/nudeblog/category/nudeblog/category/nudeblog/category/nudeblog/category/nudeblog/category/nudeblog/category/nudeblog/category/nudeblog/category/nudeblog/category/nudeblog/category/nudeblog/category/nudeblog/category/nudeblog/category/nudeblog/category/nudeblog/category/nudeblog/category/nudeblog/category/nudeblog/category/galleries/artnudes.php
Pages added to sitemap: 710

So what´s wrong there?
this site: nudeblog/category/galleries/artnudes.php
does not excist in my urls.

Michael

Re: Problem with the crawler
« Reply #1 on: July 21, 2008, 01:27:47 AM »
Looks like there is an endless links loop on the site, I'd recommend to add this into "Exclude URLs" options:
Code: [Select]
nudeblog/category/nudeblogso only the main URL will be indexed and all nested are excluded.