I have several pages in my site that displays rows from big tables by pager,
when pressing Next button its goes to next 10 pages in the pager,
I tryed to run the crawling five times at list, and every time when he reach to level 4/5/6 he seem to be stuck.
so I try to add those URLs to the exclude URL. and Its really helped.
so now for the first time he finished crawling.
but I really want him to be able to do it..
so I can make him not to be stuck?
(Or maybe to crawl just the first and second depth in this URL?)