I looked through the forum but the replies are a bit too cryptic for me to understand a solution to fix my missing links problem. We have about 200k urls in a mysql table but the crawler is not catching all of them. The way the site is set up:
Index page with 100 or so categories >
Under categories various subcategories with a list of titles showing >
Click on a title to bring up the individual record
Starting at the Index page level, the crawler catches about 5k of the 200k individual records. I dumped the table and randomly selected records the crawler is not finding and they resolve correctly to the page.
I changed the various fine tuning configuration settings on the crawler including depth and timeouts etc but still it will not catch all the records. Now I'm wondering if it is my site and consequently search engines may not be crawling all the records either? Yikes. Or is there something that may be causing the crawler just to pick those 5k because of an attribute set on configuration?
For so much information on a site, my Google rankings are terrible. I'm getting no traffic hardly - not even enough to finish out a top 20 traffic report. Did I mention yikes?
Don't know how much a sitemap will help the traffic but hoping it will. Thanks in advance for possible solutions as to why it's not getting all the URLs. Also, I am not sure how to hide the URLs in question in this forum?