generator eating up bandwidth
« on: June 07, 2006, 10:13:12 AM »
HI,

I installed the generator last night.  I installed it in mydomain.com/generator and left all the default paths.  Problem is crawling my site is using alot of bandwidth, i though as the generator is on the server i wouldn't.  My host uses cpanel and although the bandwidth isn't a problem at the moment if i run the generator every day like i was planning i would become one.

So have i done something wrong when setting this up or is this the way it's suppose to be?

TIA
Re: generator eating up bandwidth
« Reply #1 on: June 08, 2006, 10:18:54 PM »
Hello,

the bandwidth calculation depends on the hosting platform and you may want to contact your hosting support regarding this. Sitemap generator only creates network connections to the domain you specified in the configuration.
On the other hand, it's not a must to generate sitemap daily. A weekly (or even monthly) generation should work fine as well.
Re: generator eating up bandwidth
« Reply #2 on: June 09, 2006, 12:54:13 PM »
ok no problem thanks for your reply.  Love the script by the way, i am recommending it to friends alreay!
Re: generator eating up bandwidth
« Reply #4 on: September 09, 2007, 11:05:40 AM »
I am having the same problem. Just installed the script and get this in my log:

Quote
           No Date/Time           Total pages Proc.time Bandwidth New URLs Removed URLs Broken links
             1 2007-09-09 02:29 1,768           33.74s        49.47         1000           0                        64

50 mb for less than 2000 pages??
Re: generator eating up bandwidth
« Reply #5 on: September 10, 2007, 09:34:34 PM »
Hello,

yes, why not - it's only 25Kb per page, which is normal for the html page.
Re: generator eating up bandwidth
« Reply #6 on: November 05, 2007, 10:06:11 PM »
I'm very worried about using this prog as it has shot my band width through the roof. My monthly usage is about 18GB however is only day 5 of this month and its already over 14GB!!!! and I've only run the prog about 4 times so far, and those were tests as its still listing urls that I though I'd set it to exclude.

I have gone to my host about this programme and they said
Quote
When the sitemap generator goes through your site - generally it will download/view everything it fines - thus logging in your webstats as traffic (as the webserver served the request).

I'd be careful with those - they can cause load issues on the server that may cause your site to be suspended or even worse, cause the server to become unresponsive and crash.
The 'unresponsive' and 'crashing' is very worrying.

Is there any way to limit its effect on band width?
Re: generator eating up bandwidth
« Reply #7 on: November 07, 2007, 12:18:48 AM »
Hello,

what are the bandwidth stats included in sitemap details report? ("Pages size" value)
Sitemap Generator only downloads html documents, so it should not cause any issues normally.
Re: generator eating up bandwidth
« Reply #8 on: November 09, 2007, 11:40:22 PM »
Presumably its these you want
Quote
No    Date/Time    Total pages    Proc.time    Bandwidth    New URLs    Removed URLs    Broken links
26    2007-11-02 22:30    12,558    4,445.80s     300.71       1000                884    0
27    2007-11-03 00:55    12,532    4,233.30s     299.55        202                228    0
28    2007-11-06 02:39    13,502    45,562.00s    650.25           0                   0    1
Re: generator eating up bandwidth
« Reply #9 on: November 12, 2007, 12:39:23 AM »
Hello,

the bandwidth used by generator in a single session is 300Mb for you 12,500+ URLs. I suggest you to generate the sitemap less frequently to keep total usage lower (weekly or even monthly sitemap refreshing is enough in most cases).