I dug into the sitemap last night and found that the image info was making it huge. I unchecked Include images information in the config and the sitemap shrunk.
Would be nice if I could disregard banner ads and keep the image info from the content. Don't know if that's possible though.