Problem crawling wordpress
« on: April 28, 2009, 04:26:33 PM »
Hi there, Until recently I have had no problems with the sitemap crawler accessing and indexing my wordpress blog. Now however, it seems to get stuck in a loop and crawls directories which don't exist - ie, weblog/null/null/null/null/null/null etc.

The site is [ External links are visible to forum administrators only ]

Any ideas? Thanks in advance
« Last Edit: April 28, 2009, 04:56:04 PM by ed.ockelton »
Re: Problem crawling wordpress
« Reply #1 on: April 28, 2009, 06:02:32 PM »

please try to add this in "Exclude URLs" option :
Code: [Select]