XML Sitemaps Generator

Author Topic: crawling problmes  (Read 9648 times)

kajlan

  • Registered Customer
  • Approved member
  • *
  • Posts: 3
crawling problmes
« on: November 17, 2009, 10:07:12 PM »
Hi

my sitemape only index the main page

why

everything is correct i double check it


i got no errors also

[external links are visible to admins only]



kajlan

  • Registered Customer
  • Approved member
  • *
  • Posts: 3
Re: crawling problmes
« Reply #1 on: November 18, 2009, 02:11:48 AM »
Solved


problem was from


No-follow

is there a way to leave no-follow and still be able to crawle the hole site


it,s a webdirectory 

XML-Sitemaps Support

  • Administrator
  • Hero Member
  • *****
  • Posts: 10624
Re: crawling problmes
« Reply #2 on: November 18, 2009, 10:15:29 PM »
Hello,

basically, the "nofollow" attribute is designed especially to exclude pages from indexing, so it's always taken into consideration by sitemap generator.
Oleg Ignatiuk
www.xml-sitemaps.com
Send me a Private Message

For maximum exposure and traffic for your web site check out our additional SEO Services.

kajlan

  • Registered Customer
  • Approved member
  • *
  • Posts: 3
Re: crawling problmes
« Reply #3 on: November 19, 2009, 01:19:17 AM »
the no-follow is tag in the header

<META NAME="ROBOTS" CONTENT="INDEX, NOFOLLOW">

it,s saying index the page but don,t follow external links

follow internal page


but in the script it,s  indexing only the main page

is there an answer for this >>>?


XML-Sitemaps Support

  • Administrator
  • Hero Member
  • *****
  • Posts: 10624
Re: crawling problmes
« Reply #4 on: November 19, 2009, 09:18:43 PM »
Hello,

nofollow meta tag tells search engines not to follow ANY links on that page, not only external links: http://en.wikipedia.org/wiki/Robots_meta_tag#The_robots_attribute
Oleg Ignatiuk
www.xml-sitemaps.com
Send me a Private Message

For maximum exposure and traffic for your web site check out our additional SEO Services.

 

SMF 2.0.12 | SMF © 2014, Simple Machines
XHTML RSS WAP2