Pages are not added to site.

[arielk]

Hi,

I'm having some problems crawling my site (www.artmine.com)
There are some links that are not added to the sitemap although they are finding out by the crawl process.
Also when I run the Free Online Sitemap Generator those links are added to sitemap correctly.
All those links located in ArtistPage/....
This is example of links that was found by the crawler but was not added to the sitemap:

Links depth: 2
Current page: ArtistPage/Dr._Rich_Ragle.aspx
Pages added to sitemap: 102
Pages scanned: 400 (3,574.4 KB)
Pages left: 544 (+ 548 queued for the next depth level)
Time passed: 1:51
Time left: 2:31
Memory usage: 1,156.3 Kb

[XML-Sitemaps Support]

Hello,

try to set xs_crawl_ident option in generator/config.inc.php file like:

Code Select Expand

'xs_crawl_ident' => 'Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9) Gecko/2008051206 Firefox/3.0',