sitemap crawler doesn't crawl the entire site
« on: November 12, 2005, 12:17:20 AM »
Only one directory is found.

I have set the starting URL to my domain's root e.g. http://www.somesite.com/

The crawler is completing with no errors as if all it could find was that one directory.

I ran the online version and everything works as expected, e.g., crawled the entire site.

Thanks for your help,

- Vaughn

Working now!
Its working now that I removed the www in the URL.
modified the URL to http://somesite.com/

Thanks,

- Vaughn
« Last Edit: November 12, 2005, 12:20:25 AM by vaughn »
Re: sitemap crawler doesn't crawl the entire site
« Reply #1 on: November 12, 2005, 02:30:07 AM »
Hi Vaughn,

I'm glad you got it working :)
Oleg Ignatiuk
www.xml-sitemaps.com
Send me a Private Message

For maximum exposure and traffic for your web site check out our additional SEO Services.