crawling halts after some time
« on: December 19, 2008, 05:28:44 AM »
I need your help for 2 points
1) The crawling halts after some time and thereafter nothing happens if I interrupt and resume crawling.

2) I want to exclude profile page URLs, I have 7k+ users and the sitemap is including their profiles which I want to avoid.

Please advise.
Site : http:.//

Thanks in advance
Re: crawling halts after some time
« Reply #1 on: December 19, 2008, 08:11:15 PM »

1. please try to increase memory_limit setting in PHP configuration on your server.

2. you should add your profiles URL substring in "Exclude URLs" option like:
Code: [Select]