crawling problmes
« on: November 17, 2009, 10:07:12 PM »
Hi

my sitemape only index the main page

why

everything is correct i double check it


i got no errors also

[ External links are visible to forum administrators only ]


Re: crawling problmes
« Reply #1 on: November 18, 2009, 02:11:48 AM »
Solved


problem was from


No-follow

is there a way to leave no-follow and still be able to crawle the hole site


it,s a webdirectory 
Re: crawling problmes
« Reply #2 on: November 18, 2009, 10:15:29 PM »
Hello,

basically, the "nofollow" attribute is designed especially to exclude pages from indexing, so it's always taken into consideration by sitemap generator.
Re: crawling problmes
« Reply #3 on: November 19, 2009, 01:19:17 AM »
the no-follow is tag in the header

<META NAME="ROBOTS" CONTENT="INDEX, NOFOLLOW">

it,s saying index the page but don,t follow external links

follow internal page


but in the script it,s  indexing only the main page

is there an answer for this >>>?

Re: crawling problmes
« Reply #4 on: November 19, 2009, 09:18:43 PM »
Hello,

nofollow meta tag tells search engines not to follow ANY links on that page, not only external links: http://en.wikipedia.org/wiki/Robots_meta_tag#The_robots_attribute