Crawl and Directory problems
« on: July 09, 2009, 07:32:26 PM »

I'm having some problems running the crawl. It seems to be scanning/crawling a lot more pages than I actually have on the site and making up directories. I copied on from the progress screen:

Current page: newemiplasticscom/eoat/newemiplasticscom/eoat/conveyor/forms.htm

eoat and conveyor are both subdirectories under newemiplasticscom. The paths should be newemiplasticscom/eoat or newemiplasticscom/conveyor. Maybe I have the paths on the Configuration page wrong. How can I tell what they are supposed to be?

« Last Edit: July 09, 2009, 08:24:22 PM by xml-sitemaps »
Re: Crawl and Directory problems
« Reply #1 on: July 10, 2009, 07:46:26 PM »

try to add this in "Exclude URLs" setting:
Code: [Select]