Opened 12 years ago
Closed 10 years ago
#2253 closed Enhancement (duplicate)
Use less memory when uploading by pruning overly-long write buffers
Reported by: | Syam | Owned by: | charles |
---|---|---|---|
Priority: | Normal | Milestone: | Sometime |
Component: | libtransmission | Version: | 1.72 |
Severity: | Normal | Keywords: | |
Cc: |
Description
Looks like transmission's memory resident memory usage increases linearly with uploading speed. At around 5000KBps , RSS peaks upto 400MB. If this is expected, it should be at-least a tunable as ability to tune max memory usage is a minimum for operating in production environments.
The operating environment is RHEL 4 Update 6, with curl statically compiled. I would be glad to provide any additional information if needed.
Change History (19)
comment:1 Changed 12 years ago by charles
comment:2 Changed 12 years ago by Syam
Actually no, I am referring to resident memory size of transmission-daemon process not the memory used by kernel disk buffers/caches. I should have mentioned that rss size comes down quickly after the upload is done. I am measuring rss using > ps -o rss,cmd | grep transmission-daemon, if it helps.
comment:3 Changed 12 years ago by charles
hmmm. Time for me to start an upload session in valgrind, I think.
comment:4 Changed 12 years ago by charles
Could you try a nightly build of r8765 or higher?
comment:5 Changed 12 years ago by charles
The only sustained growth I saw over an upload series was the growth of the per-peer upload buffers. If you had a lot of peers you were seeding to, I can see where this might have eaten a lot of memory.
Here are the valgrind / massif logs of two runs. These didn't have many peers, and only ran for about a quarter-hour each, so they didn't reach the kinds of memory you described. However there's a clear pattern: r8764 uses more memory over time, but r8765 is reasonably flat.
MB 10.81^ # | # | # | @:::::@::# | ,,.....,..@@:::::@::# | ,.... ,.....@:@::: ...: @ ::::::@:: :@@:::::@::@@:::::@::# | . ..:@:::: @:::::@:@::::::::::: @ ::::::@:: :@@:::::@::@@:::::@::# | : ::::@:::: @:::::@:@::::::::::: @ ::::::@:: :@@:::::@::@@:::::@::# | : ..::::@:::: @:::::@:@::::::::::: @ ::::::@:: :@@:::::@::@@:::::@::# | : .:::::::@:::: @:::::@:@::::::::::: @ ::::::@:: :@@:::::@::@@:::::@::# | :.::::::::@:::: @:::::@:@::::::::::: @ ::::::@:: :@@:::::@::@@:::::@::# | ::::::::::@:::: @:::::@:@::::::::::: @ ::::::@:: :@@:::::@::@@:::::@::# | ::::::::::@:::: @:::::@:@::::::::::: @ ::::::@:: :@@:::::@::@@:::::@::# | ::::::::::@:::: @:::::@:@::::::::::: @ ::::::@:: :@@:::::@::@@:::::@::# |.::::::::::@:::: @:::::@:@::::::::::: @ ::::::@:: :@@:::::@::@@:::::@::# |:::::::::::@:::: @:::::@:@::::::::::: @ ::::::@:: :@@:::::@::@@:::::@::# |:::::::::::@:::: @:::::@:@::::::::::: @ ::::::@:: :@@:::::@::@@:::::@::# |:::::::::::@:::: @:::::@:@::::::::::: @ ::::::@:: :@@:::::@::@@:::::@::# |:::::::::::@:::: @:::::@:@::::::::::: @ ::::::@:: :@@:::::@::@@:::::@::# |:::::::::::@:::: @:::::@:@::::::::::: @ ::::::@:: :@@:::::@::@@:::::@::# 0 +----------------------------------------------------------------------->Gi 0 13.46
MB 7.706^ # | # | # | # . | # . . . . : . : | # : : . : . . . : , :, : : , : : . : | #: : ::, . @:. :: :, :::.,:. : : .:. : : .@.::@ .: : :: @ : @ : :@::: | #: : ::@ @: @:: :: :@ ::::@:: : : :::: : : :@:::@ :: : :: @ : @ : :@::: | #: : ::@ @: @:: :: :@ ::::@:: : : :::: : : :@:::@ :: : :: @ : @ : :@::: | #: : ::@ @: @:: :: :@ ::::@:: : : :::: : : :@:::@ :: : :: @ : @ : :@::: | #: : ::@ @: @:: :: :@ ::::@:: : : :::: : : :@:::@ :: : :: @ : @ : :@::: | #: : ::@ @: @:: :: :@ ::::@:: : : :::: : : :@:::@ :: : :: @ : @ : :@::: | #: : ::@ @: @:: :: :@ ::::@:: : : :::: : : :@:::@ :: : :: @ : @ : :@::: | #: : ::@ @: @:: :: :@ ::::@:: : : :::: : : :@:::@ :: : :: @ : @ : :@::: | #: : ::@ @: @:: :: :@ ::::@:: : : :::: : : :@:::@ :: : :: @ : @ : :@::: | #: : ::@ @: @:: :: :@ ::::@:: : : :::: : : :@:::@ :: : :: @ : @ : :@::: | #: : ::@ @: @:: :: :@ ::::@:: : : :::: : : :@:::@ :: : :: @ : @ : :@::: | #: : ::@ @: @:: :: :@ ::::@:: : : :::: : : :@:::@ :: : :: @ : @ : :@::: | #: : ::@ @: @:: :: :@ ::::@:: : : :::: : : :@:::@ :: : :: @ : @ : :@::: | #: : ::@ @: @:: :: :@ ::::@:: : : :::: : : :@:::@ :: : :: @ : @ : :@::: 0 +----------------------------------------------------------------------->Gi 0 14.21
comment:6 Changed 12 years ago by charles
- Component changed from Transmission to GTK+ Client
- Owner set to charles
comment:7 Changed 12 years ago by Syam
I have tried seeding using transmissioncli for about 20-50 peers at once and upload speeds upto 2000KBps+ for about 15 min. Here are the results with 1.70 and 1.72+ [ 8766 ] . Your fix seems to have worked with max RSS memory around 60MB , while 1.70 peaked about 440MB
1.70
Command: /usr/local/bin/transmissioncli ./1246411620.torrent -w /home/syam/ Massif arguments: (none) ms_print arguments: massif.out.23769.1.70
MB
306.3 #. . .. ....... ., .. . .....,
Number of snapshots: 59
Detailed snapshots: [1, 11, 17, 26, 31 (peak), 44, 57]
1.72
Command: /home/syam/transmissioncli ./1246411620.torrent -w /home/syam/ Massif arguments: (none) ms_print arguments: massif.out.2255
MB
22.96 #
| @ # | @ # | @ # : . | @ #. : : | @ #: :. :
- | , @. #:
- :
- | . @ . @: #:
- : | : @ : @: #: ::: . :: @
- | : @ : @
- . #::: : :::: : :: :::.. @: | : :::: : @: : :: : @::: :@:: #::: : : :::::: :::@ :: ::::::@: | : :::: : @: : :: : @::: :@:: #::: : : :::::: :::@ :: ::::::@: | : :::: :. @: : :: : @::: :@::. #::: : :. :::::: :::@ :: ::::::@: | : :::: :: @: : :: :. @::: :@::: #::: : :: :::::: :::@ :: ::::::@: | : :::: :: @: : :: :: @::: :@::: #::: : :: :::::: :::@ :: ::::::@: | : :::: :: @: : :: :: @::: :@::: #::: : :: :::::: :::@ :: ::::::@: | : :::: :: @: : :: :: .@::: :@::: #::: : :: :::::: :::@ :: ::::::@:
- |
- :::: ::. @: : :: :: :@::: :@::: .#::: : :: :::::: :::@ :::: ::::::@:
- |
- :::: ::: @: : :: :: :@::: :@::: :#::: : :: :::::: :::@ :::: ::::::@:
- |
- :::: ::: @: : :: :: :@::: :@::: :#::: : :: :::::: :::@ :::: ::::::@: 0 +----------------------------------------------------------------------->Gi 0 23.75
comment:8 Changed 12 years ago by charles
- Component changed from GTK+ Client to libtransmission
- Milestone changed from None Set to 1.73
- Resolution set to fixed
- Status changed from new to closed
Thanks for the quick confirmation :)
comment:9 Changed 12 years ago by charles
- Summary changed from Linear increase in Memory usage while uploading to use less memory when uploading
comment:10 Changed 12 years ago by charles
- Summary changed from use less memory when uploading to Use less memory when uploading by pruning overly-long write buffers
comment:11 Changed 12 years ago by charles
- Priority changed from Normal to High
comment:12 Changed 12 years ago by charles
- Severity changed from Major to Normal
comment:13 Changed 12 years ago by charles
- Milestone 1.73 deleted
- Priority changed from High to Normal
- Resolution fixed deleted
- Status changed from closed to reopened
According to irc user fl_, this fix causes high CPU use (100% for him) on some machines.
libevent 2.0's evbuffer rewrite should fix this automatically without the CPU load, so let's leave this open for a little while until we have a cleaner fix or until we jump to libevent2.
comment:14 Changed 12 years ago by Syam
I confirm too; with the memory usage fix, CPU is at 100% most of the time during data transfer. I should however comment that even without memory fix transmission CPU usage hits 100%.
Charles - Do you recommend any other workaround before eventual fix ?
comment:15 Changed 12 years ago by charles
Syam: if the CPU is still running at 100% then it's an issue unrelated to this ticket, so my recommendation would be to open a new ticket about it. If you're on OS X, use the Shark profiler as described on the Transmission wiki.
comment:16 Changed 12 years ago by charles
- Type changed from Bug to Enhancement
comment:17 Changed 12 years ago by charles
- Milestone set to Sometime
comment:18 Changed 11 years ago by charles
This ticket hasn't been updated in 14 months, but FWIW this is still a useful improvement. The status is same as before: this will go in after we jump to libevent2.
comment:19 Changed 10 years ago by charles
- Resolution set to duplicate
- Status changed from reopened to closed
Superceded by ticket #3836, "libevent2 support"
This sounds like a RHEL variation of <http://trac.transmissionbt.com/wiki/InactiveMemory>