XML Sitemaps Generator

Author Topic: I can't get the crawler to crawl my entire website  (Read 23101 times)

matthew

  • Registered Customer
  • Approved member
  • *
  • Posts: 6
I can't get the crawler to crawl my entire website
« on: December 31, 2006, 04:06:55 PM »
I do not know how many pages I have in the entire site, but when I do a search for a certain category number it isn't showing so I know it is not reaching my entire site.

Anyways to fix this issue?

I have had it on 0 for unlimited still nothing.

matthew

  • Registered Customer
  • Approved member
  • *
  • Posts: 6
Re: I can't get the crawler to crawl my entire website
« Reply #1 on: December 31, 2006, 08:11:25 PM »
Update;

What I did was create a sitemap that had every single category for the site, and since then it was able to fully see the site and report it to the xml.

Thanks

XML-Sitemaps Support

  • Administrator
  • Hero Member
  • *****
  • Posts: 10624
Re: I can't get the crawler to crawl my entire website
« Reply #2 on: January 02, 2007, 09:11:34 PM »
Great!

I confirm that you should have links to all you category pages so that sitemap generator crawler is able to find your pages starting from homepage.
Oleg Ignatiuk
www.xml-sitemaps.com
Send me a Private Message

For maximum exposure and traffic for your web site check out our additional SEO Services.

 

SMF 2.0.12 | SMF © 2014, Simple Machines
XHTML RSS WAP2