XML Sitemaps Generator

Author Topic: Not crawling whole site  (Read 19846 times)

helyn

  • Registered Customer
  • Approved member
  • *
  • Posts: 2
Not crawling whole site
« on: July 03, 2007, 07:49:55 PM »
I have the stand alone which you just installed...it is not doing my whole site. THe settings look right but only does 3943 pages I have or urls...I have over 35,000 .....I need whole site indexed. When it crawled it looked like it was getting all but the at end same number as before.

rajagopalan

  • Registered Customer
  • Approved member
  • *
  • Posts: 5
Re: Not crawling whole site
« Reply #1 on: July 05, 2007, 03:41:14 PM »
I have the same problem.

I have installed the standalone registered version.

I have giving the starting URL as [external links are visible to admins only]. It is doing all pages for this but it is not including my sub-domain at [external links are visible to admins only].

When I try to give the starting URL as [external links are visible to admins only] and click the button for crawling, it does not continue and comes back to the configuration page.

Please advice how to crawl the entire site including sub-domains.

Thanks
Regards
Raj

helyn

  • Registered Customer
  • Approved member
  • *
  • Posts: 2
Re: Not crawling whole site
« Reply #2 on: July 05, 2007, 08:26:17 PM »
I would like to add to my post with it did do more of site but still not close....it does about 15,000 pages...I noticed it says link level 2....this needs to be increased as my site has many folders with many galleries...so I think this may be the problem. Where do I change settings for this?
Thank you

XML-Sitemaps Support

  • Administrator
  • Hero Member
  • *****
  • Posts: 10624
Re: Not crawling whole site
« Reply #3 on: July 05, 2007, 09:48:35 PM »
Hello,

please let me know (in private message) an example URL that is not included in sitemap and how it can be reached from homepage.

Raj,
Quote
I have installed the standalone registered version.

I have giving the starting URL as  It is doing all pages for this but it is not including my sub-domain at
According to XML sitemap protocol, it is not allowed to include URLs from different domains or subdomains in the same sitemap, so you will have to create a separate sitemap for every domain.
Oleg Ignatiuk
www.xml-sitemaps.com
Send me a Private Message

For maximum exposure and traffic for your web site check out our additional SEO Services.

rajagopalan

  • Registered Customer
  • Approved member
  • *
  • Posts: 5
Re: Not crawling whole site
« Reply #4 on: July 06, 2007, 01:53:37 AM »
Hi Oleg,

If I cannot include my sub-domains as part of my main domain for site-mapping, does it mean that I have to install XML-Stiemap software again separately in the sub-domain folder?

Please advice.

Thanks
Regards
Raj


XML-Sitemaps Support

  • Administrator
  • Hero Member
  • *****
  • Posts: 10624
Re: Not crawling whole site
« Reply #5 on: July 06, 2007, 07:17:13 PM »
Hello,

yes, you should install separate instances of Sitemap Generator per (sub)domain.
Oleg Ignatiuk
www.xml-sitemaps.com
Send me a Private Message

For maximum exposure and traffic for your web site check out our additional SEO Services.

 

SMF 2.0.12 | SMF © 2014, Simple Machines
XHTML RSS WAP2