How can I get a sense of progress in terms of how many depths and actual URLs are left?
I got to zero for URL scanned count for depth 3, only to see that there is a depth 4 with 86,000 URLs more!
My URL scanned count is now 300,000 pages and counting.
This also mistifies me as I know our site only has 300k pages or less.
In the middle of execution I excluded the pages that were not supposed to be indexed, so are those pages stilll going to be included?
1. How can I get a sense of progress in terms of how many depths and actual URLs are left?
2. How can I investigate this, when URLs scanned is now well more than the URLs I know are site has.
3. How can I know wc URL is currently being crawled?
4. What is a depth?
Thanks in advance.