Only part of site indexed

[AndrewTraub]

Hello,

When I try the demo on two of my sites, only parts of the site are indexed (a very small part).

The sites are:
[external links are visible to admins only]
[external links are visible to admins only]

both sites start out with two pages. One links to subdomains and the other subfolders.  Each subfolder has many pages.

Will I have the same problem with the script?

[admin]

Hello,

according to the xml sitemap protocol, it's not allowed to include URLs from other (sub)domains in sitemap, so the crawler skips them.

[AndrewTraub]

ok, so if it won't index, say [external links are visible to admins only], why won't it index [external links are visible to admins only] which is under [external links are visible to admins only]  Is it because the www is missing?

[admin]

Yes, that's right - "www" prefix means another domain name in this case.

[AndrewTraub]

I still don't understand why the folders at [external links are visible to admins only] are not being indexed - only 17 pages are indexed but each folder has many articles.  Here are the links:

            <table width="100%"  border="0" align="center" cellpadding="2" cellspacing="0">
              <tr valign="top" class="style2">
                <td class="style3"><a href="[external links are visible to admins only]">Anxiety</a></td>
                <td class="style3"><a href="[external links are visible to admins only]">Stress</a></td>
                <td class="style3"><a href="[external links are visible to admins only]">Stress and Anxiety Medications </a></td>
                <td class="style3"><a href="[external links are visible to admins only]">Herbal Medications</a></td>
                <td class="style3"><a href="[external links are visible to admins only]">Alternative Medicines and Therapies</a></td>
              </tr>
              <tr valign="top" class="style2">
                <td class="style3"><a href="[external links are visible to admins only]">Social Anxiety </a></td>
                <td class="style3"><a href="[external links are visible to admins only]">Panic Disorders</a></td>
                <td class="style3"><a href="[external links are visible to admins only]">Depression</a></td>
                <td class="style3"><a href="[external links are visible to admins only]">Panic Attacks </a></td>
                <td class="style3"><a href="[external links are visible to admins only]">Psychologist</a></td>
              </tr>
              <tr valign="top" class="style2">
                <td class="style3"><a href="[external links are visible to admins only]">Stress Management</a></td>
                <td class="style3"><a href="[external links are visible to admins only]">Behavior Modifications </a></td>
                <td class="style3"><a href="[external links are visible to admins only]">Anxiety Symptoms and Stress Signs</a></td>
                <td class="style3"><a href="[external links are visible to admins only]">Mental Health</a></td>
                <td class="style3"><a href="[external links are visible to admins only]">Stress and Anxiety Relief</a></td>
              </tr>
              <tr valign="top" class="style2">
                <td colspan="5" class="style3"><a href="[external links are visible to admins only]">Stress & Anxiety Blog</a></td>
            </tr>
            </table>

Am I missing something in the way the links are set up?

[admin]

Hello,

the "www. yourdomain .com/anxiety" link is redirected to "yourdomain .com/anxiety" for me.

[AndrewTraub]

I still have the same result if I start spidering at [external links are visible to admins only] instead of [external links are visible to admins only].

[admin]

The links are pointing to www domain on your pages (although they are redirected to non-www then). Make sure that ALL links (both on-page and landing URLs after redirect) reside  on exactly the same domain (non-www or www domain).

[AndrewTraub]

Thank you!

[laugh555]

Posted by: AndrewTraub
I still have the same result if I start spidering at [external links are visible to admins only] instead of [external links are visible to admins only].

Thank you

[admin]

You *should* use www domain as Starting URL since your links are pointing to it.