crawler wont scan SMF forum directory?
« on: February 01, 2011, 02:11:29 PM »

Our generator crawls the main website fine and produces XML and ROR sitemaps without problem, however I have noticed it only covers 48 links

I'm not sure why but it seems to miss out our forum installation in folder /smf

When I run a manual 'free' scan from the website to it scans the folder fine and produces an output file too (albeit limited to 500 url's)

I've checked all permissions and everything seems fine (755), there is no exclusion of this folder in the config.

Please help


Re: crawler wont scan SMF forum directory?
« Reply #1 on: February 01, 2011, 10:09:18 PM »

please let me know your generator URL/login in private message to check this.