Generator only indexing index.html and a few other link
« on: July 17, 2007, 03:47:59 PM »
Everytime I try to generate It only generates the index.html and 5 other links on the page. But its ignoring every other folder and even other pages in the root directory? any suggestions?
Re: Generator only indexing index.html and a few other link
« Reply #1 on: July 17, 2007, 09:09:57 PM »
Hello,

make sure that all links point to the same domain  - even example.com and www.example.com are assumed to be different domains and cannot be included in the same sitemap.
Re: Generator only indexing index.html and a few other link
« Reply #2 on: July 18, 2007, 01:12:51 AM »
All links are pointing to same domain Im just not sure why it doesnt scan the entire site
Does this not can all folders for html files??
Re: Generator only indexing index.html and a few other link
« Reply #3 on: July 18, 2007, 10:23:45 AM »
Try putting your homepage URL into our robots simulator to see what links are found on your page.

https://www.xml-sitemaps.com/se-bot-simulator.html

Bear in mind that whatever the results are on this is likely to be the same for search engine robots. If you have links that aren't picked up by this then the search engine robots are not going to find them either.

Possible causes are links embeded in Flash or Javascript menus.

The following is taken from Google's Webmaster Guidelines

"Use a text browser such as Lynx to examine your site, because most search engine spiders see your site much as Lynx would. If fancy features such as JavaScript, cookies, session IDs, frames, DHTML, or Flash keep you from seeing all of your site in a text browser, then search engine spiders may have trouble crawling your site."
Philip Nicosia