Bandwidth usage?
« on: November 20, 2009, 10:56:32 AM »
Hello, on my site I crawl in the changelog it reports using 94.39MB

Which is alot, but my website is 2000pages... but just to check, is the crawler downloading images?

Obviously they are not in the sitemap as...
jpg is in the "Exclude from sitemap extensions:"

however it is not in the "Do not parse extensions:" will putting it there make the server completely ignore it? or make it list in the sitemap.

Or I could add /images/ into the "Exclude URLs:" section.

Could you please enlighten me if I can reduce bandwidth this way.
Rob
Re: Bandwidth usage?
« Reply #1 on: November 24, 2009, 09:33:05 PM »
Helllo,

images are not downloaded by sitemap generator, they are excluded by "exclude extensions" setting.
94Mb for 2,000 pages results in around 47KB per one html page, which is pretty much normal.