Only part of site indexed
« on: February 22, 2008, 08:31:51 PM »
Hello,

When I try the demo on two of my sites, only parts of the site are indexed (a very small part).

The sites are:
[ External links are visible to forum administrators only ]
[ External links are visible to forum administrators only ]

both sites start out with two pages. One links to subdomains and the other subfolders.  Each subfolder has many pages.

Will I have the same problem with the script?
Re: Only part of site indexed
« Reply #1 on: February 23, 2008, 12:19:56 AM »
Hello,

according to the xml sitemap protocol, it's not allowed to include URLs from other (sub)domains in sitemap, so the crawler skips them.
Re: Only part of site indexed
« Reply #2 on: February 23, 2008, 06:01:56 PM »
ok, so if it won't index, say [ External links are visible to forum administrators only ], why won't it index [ External links are visible to forum administrators only ] which is under [ External links are visible to forum administrators only ]  Is it because the www is missing?
Re: Only part of site indexed
« Reply #3 on: February 24, 2008, 08:52:20 PM »
Yes, that's right - "www" prefix means another domain name in this case.
Re: Only part of site indexed
« Reply #4 on: February 25, 2008, 03:53:57 PM »
I still don't understand why the folders at [ External links are visible to forum administrators only ] are not being indexed - only 17 pages are indexed but each folder has many articles.  Here are the links:

            <table width="100%"  border="0" align="center" cellpadding="2" cellspacing="0">
              <tr valign="top" class="style2">
                <td class="style3"><a href="[ External links are visible to forum administrators only ]">Anxiety</a></td>
                <td class="style3"><a href="[ External links are visible to forum administrators only ]">Stress</a></td>
                <td class="style3"><a href="[ External links are visible to forum administrators only ]">Stress and Anxiety Medications </a></td>
                <td class="style3"><a href="[ External links are visible to forum administrators only ]">Herbal Medications</a></td>
                <td class="style3"><a href="[ External links are visible to forum administrators only ]">Alternative Medicines and Therapies</a></td>
              </tr>
              <tr valign="top" class="style2">
                <td class="style3"><a href="[ External links are visible to forum administrators only ]">Social Anxiety </a></td>
                <td class="style3"><a href="[ External links are visible to forum administrators only ]">Panic Disorders</a></td>
                <td class="style3"><a href="[ External links are visible to forum administrators only ]">Depression</a></td>
                <td class="style3"><a href="[ External links are visible to forum administrators only ]">Panic Attacks </a></td>
                <td class="style3"><a href="[ External links are visible to forum administrators only ]">Psychologist</a></td>
              </tr>
              <tr valign="top" class="style2">
                <td class="style3"><a href="[ External links are visible to forum administrators only ]">Stress Management</a></td>
                <td class="style3"><a href="[ External links are visible to forum administrators only ]">Behavior Modifications </a></td>
                <td class="style3"><a href="[ External links are visible to forum administrators only ]">Anxiety Symptoms and Stress Signs</a></td>
                <td class="style3"><a href="[ External links are visible to forum administrators only ]">Mental Health</a></td>
                <td class="style3"><a href="[ External links are visible to forum administrators only ]">Stress and Anxiety Relief</a></td>
              </tr>
              <tr valign="top" class="style2">
                <td colspan="5" class="style3"><a href="[ External links are visible to forum administrators only ]">Stress & Anxiety Blog</a></td>
            </tr>
            </table>

Am I missing something in the way the links are set up?
Re: Only part of site indexed
« Reply #5 on: February 25, 2008, 11:40:08 PM »
Hello,

the "www. yourdomain .com/anxiety" link is redirected to "yourdomain .com/anxiety" for me.
Re: Only part of site indexed
« Reply #6 on: February 26, 2008, 10:29:27 PM »
I still have the same result if I start spidering at [ External links are visible to forum administrators only ] instead of [ External links are visible to forum administrators only ].
Re: Only part of site indexed
« Reply #7 on: February 27, 2008, 12:20:21 AM »
The links are pointing to www domain on your pages (although they are redirected to non-www then). Make sure that ALL links (both on-page and landing URLs after redirect) reside  on exactly the same domain (non-www or www domain).
Re: Only part of site indexed
« Reply #8 on: February 27, 2008, 04:01:58 PM »
Thank you!

laugh555

*
Re: Only part of site indexed
« Reply #9 on: April 22, 2008, 05:20:36 PM »
Posted by: AndrewTraub
I still have the same result if I start spidering at [ External links are visible to forum administrators only ] instead of [ External links are visible to forum administrators only ].

Thank you
Re: Only part of site indexed
« Reply #10 on: April 22, 2008, 10:23:43 PM »
You *should* use www domain as Starting URL since your links are pointing to it.