Inquiry about a site with 100,000+ links to crawl..

[Jaiz]

Hi I have had GSiteCrawler running for a few days now and it still hasn't finished crawling my site. I have a coppermine gallery which has thousands among thousands of links which is making it impossible for me to create full sitemaps like I want. My site have anywhere between 100,000-500,000 urls to crawl. Would the xml standalone sitemap creator be able to do this within a few hours or at least 24 hours? Or is it hopeless? Also, does the xml standalone sitemap creator have an option that would automatically not include all the urls that are not found and/or time out?

[admin]

Hello,

the total generation time mainly depends on how fast is your server is (how fast pages are accessed via http). Here are a few examples: http://www.xml-sitemaps.com/forum/index.php/topic,95.0.html

The URLs that are not found are not inlcuded in sitemap. Also, there is a special option in sitemap generator configuration that allows you to exclude certain pages from crawling.