Pages are not added to site.
« on: May 19, 2008, 09:05:22 AM »

I'm having some problems crawling my site ([ External links are visible to forum administrators only ])
There are some links that are not added to the sitemap although they are finding out by the crawl process.
Also when I run the Free Online Sitemap Generator those links are added to sitemap correctly.
All those links located in ArtistPage/.
This is example of links that was found by the crawler but was not added to the sitemap:

Links depth: 2
Current page: ArtistPage/Dr._Rich_Ragle.aspx
Pages added to sitemap: 102
Pages scanned: 400 (3,574.4 KB)
Pages left: 544 (+ 548 queued for the next depth level)
Time passed: 1:51
Time left: 2:31
Memory usage: 1,156.3 Kb
Re: Pages are not added to site.
« Reply #1 on: May 19, 2008, 11:31:57 PM »

try to set xs_crawl_ident option in generator/ file like:
Code: [Select]
'xs_crawl_ident' => 'Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9) Gecko/2008051206 Firefox/3.0',