Opened 12 years ago

Closed 10 years ago

#2253 closed Enhancement (duplicate)

Use less memory when uploading by pruning overly-long write buffers

Reported by: Syam Owned by: charles
Priority: Normal Milestone: Sometime
Component: libtransmission Version: 1.72
Severity: Normal Keywords:
Cc:

Description

Looks like transmission's memory resident memory usage increases linearly with uploading speed. At around 5000KBps , RSS peaks upto 400MB. If this is expected, it should be at-least a tunable as ability to tune max memory usage is a minimum for operating in production environments.

The operating environment is RHEL 4 Update 6, with curl statically compiled. I would be glad to provide any additional information if needed.

Change History (19)

comment:1 Changed 12 years ago by charles

This sounds like a RHEL variation of <http://trac.transmissionbt.com/wiki/InactiveMemory>

comment:2 Changed 12 years ago by Syam

Actually no, I am referring to resident memory size of transmission-daemon process not the memory used by kernel disk buffers/caches. I should have mentioned that rss size comes down quickly after the upload is done. I am measuring rss using > ps -o rss,cmd | grep transmission-daemon, if it helps.

comment:3 Changed 12 years ago by charles

hmmm. Time for me to start an upload session in valgrind, I think.

comment:4 Changed 12 years ago by charles

Could you try a nightly build of r8765 or higher?

comment:5 Changed 12 years ago by charles

The only sustained growth I saw over an upload series was the growth of the per-peer upload buffers. If you had a lot of peers you were seeding to, I can see where this might have eaten a lot of memory.

Here are the valgrind / massif logs of two runs. These didn't have many peers, and only ran for about a quarter-hour each, so they didn't reach the kinds of memory you described. However there's a clear pattern: r8764 uses more memory over time, but r8765 is reasonably flat.

r8764:


    MB
10.81^                                                                       #
     |                                                                       #
     |                                                                       #
     |                                                              @:::::@::#
     |                                                   ,,.....,..@@:::::@::#
     |           ,.... ,.....@:@:::    ...: @ ::::::@:: :@@:::::@::@@:::::@::#
     | .      ..:@:::: @:::::@:@::::::::::: @ ::::::@:: :@@:::::@::@@:::::@::#
     | :     ::::@:::: @:::::@:@::::::::::: @ ::::::@:: :@@:::::@::@@:::::@::#
     | :   ..::::@:::: @:::::@:@::::::::::: @ ::::::@:: :@@:::::@::@@:::::@::#
     | : .:::::::@:::: @:::::@:@::::::::::: @ ::::::@:: :@@:::::@::@@:::::@::#
     | :.::::::::@:::: @:::::@:@::::::::::: @ ::::::@:: :@@:::::@::@@:::::@::#
     | ::::::::::@:::: @:::::@:@::::::::::: @ ::::::@:: :@@:::::@::@@:::::@::#
     | ::::::::::@:::: @:::::@:@::::::::::: @ ::::::@:: :@@:::::@::@@:::::@::#
     | ::::::::::@:::: @:::::@:@::::::::::: @ ::::::@:: :@@:::::@::@@:::::@::#
     |.::::::::::@:::: @:::::@:@::::::::::: @ ::::::@:: :@@:::::@::@@:::::@::#
     |:::::::::::@:::: @:::::@:@::::::::::: @ ::::::@:: :@@:::::@::@@:::::@::#
     |:::::::::::@:::: @:::::@:@::::::::::: @ ::::::@:: :@@:::::@::@@:::::@::#
     |:::::::::::@:::: @:::::@:@::::::::::: @ ::::::@:: :@@:::::@::@@:::::@::#
     |:::::::::::@:::: @:::::@:@::::::::::: @ ::::::@:: :@@:::::@::@@:::::@::#
     |:::::::::::@:::: @:::::@:@::::::::::: @ ::::::@:: :@@:::::@::@@:::::@::#
   0 +----------------------------------------------------------------------->Gi
     0                                                                   13.46

r8765:

    MB
7.706^ #                                                                      
     | #                                                                      
     | #                                                                      
     | #                                                           .          
     | #  .         .                                 .   .        :   .     :
     | #  :         :   . :  . .     .     :       ,  :,  : :    , :   : .   :
     | #: : ::,  . @:. :: :, :::.,:. : :  .:. : : .@.::@ .: : :: @ : @ : :@:::
     | #: : ::@ @: @:: :: :@ ::::@:: : : :::: : : :@:::@ :: : :: @ : @ : :@:::
     | #: : ::@ @: @:: :: :@ ::::@:: : : :::: : : :@:::@ :: : :: @ : @ : :@:::
     | #: : ::@ @: @:: :: :@ ::::@:: : : :::: : : :@:::@ :: : :: @ : @ : :@:::
     | #: : ::@ @: @:: :: :@ ::::@:: : : :::: : : :@:::@ :: : :: @ : @ : :@:::
     | #: : ::@ @: @:: :: :@ ::::@:: : : :::: : : :@:::@ :: : :: @ : @ : :@:::
     | #: : ::@ @: @:: :: :@ ::::@:: : : :::: : : :@:::@ :: : :: @ : @ : :@:::
     | #: : ::@ @: @:: :: :@ ::::@:: : : :::: : : :@:::@ :: : :: @ : @ : :@:::
     | #: : ::@ @: @:: :: :@ ::::@:: : : :::: : : :@:::@ :: : :: @ : @ : :@:::
     | #: : ::@ @: @:: :: :@ ::::@:: : : :::: : : :@:::@ :: : :: @ : @ : :@:::
     | #: : ::@ @: @:: :: :@ ::::@:: : : :::: : : :@:::@ :: : :: @ : @ : :@:::
     | #: : ::@ @: @:: :: :@ ::::@:: : : :::: : : :@:::@ :: : :: @ : @ : :@:::
     | #: : ::@ @: @:: :: :@ ::::@:: : : :::: : : :@:::@ :: : :: @ : @ : :@:::
     | #: : ::@ @: @:: :: :@ ::::@:: : : :::: : : :@:::@ :: : :: @ : @ : :@:::
   0 +----------------------------------------------------------------------->Gi
     0                                                                   14.21

comment:6 Changed 12 years ago by charles

  • Component changed from Transmission to GTK+ Client
  • Owner set to charles

comment:7 Changed 12 years ago by Syam

I have tried seeding using transmissioncli for about 20-50 peers at once and upload speeds upto 2000KBps+ for about 15 min. Here are the results with 1.70 and 1.72+ [ 8766 ] . Your fix seems to have worked with max RSS memory around 60MB , while 1.70 peaked about 440MB

1.70


Command: /usr/local/bin/transmissioncli ./1246411620.torrent -w /home/syam/ Massif arguments: (none) ms_print arguments: massif.out.23769.1.70


MB

306.3 #. . .. ....... ., .. . .....,

| ..#: :
::::::: :@ :: : :::::@ | @::::#: : :: ::::::: :@ :: : :::::@ | @::::#: : :: ::::::: :@ :: : :::::@ | @::::#: : :: ::::::: :@ :: : :::::@ | @::::#: : :: ::::::: :@ :: : :::::@ | :::: : @::::#: : :: ::::::: :@ :: : :::::@ | :::: : @::::#: : :: ::::::: :@ :: : :::::@ | : :::: : @::::#: : :: ::::::: :@ :: : :::::@ | : :::: : @::::#: : :: ::::::: :@ :: : :::::@
| .
:::: : @::::#: : :: ::::::: :@ :: : :::::@
| : @:
:::: : @::::#: : :: ::::::: :@ :: : :::::@
| : @:
:::: : @::::#: : :: ::::::: :@ :: : :::::@
| . : @:
:::: : @::::#: : :: ::::::: :@ :: : :::::@
| .: : @:
:::: : @::::#: : :: ::::::: :@ :: : :::::@
|
: @: :: :::: : @::::#: : :: ::::::: :@ :: : :::::@ | :::: : @: :: :::: : @::::#: : :: ::::::: :@ :: : :::::@ | @ :::: : @: :: :::: : @::::#: : :: ::::::: :@ :: : :::::@ | .: @ :::: : @: :: :::: : @::::#: : :: ::::::: :@ :: : :::::@
|
@ :::: : @: :: :::: : @::::#: : :: ::::::: :@ :: : :::::@ 0 +----------------------------------------------------------------------->Gi 0 20.43

Number of snapshots: 59

Detailed snapshots: [1, 11, 17, 26, 31 (peak), 44, 57]

1.72


Command: /home/syam/transmissioncli ./1246411620.torrent -w /home/syam/ Massif arguments: (none) ms_print arguments: massif.out.2255


MB

22.96 #

| @ # | @ # | @ # : . | @ #. : : | @ #: :. :

| , @. #:
:
| . @ . @: #:
: | : @ : @: #: ::: . :: @
| : @ : @
. #::: : :::: : :: :::.. @: | : :::: : @: : :: : @::: :@:: #::: : : :::::: :::@ :: ::::::@: | : :::: : @: : :: : @::: :@:: #::: : : :::::: :::@ :: ::::::@: | : :::: :. @: : :: : @::: :@::. #::: : :. :::::: :::@ :: ::::::@: | : :::: :: @: : :: :. @::: :@::: #::: : :: :::::: :::@ :: ::::::@: | : :::: :: @: : :: :: @::: :@::: #::: : :: :::::: :::@ :: ::::::@: | : :::: :: @: : :: :: @::: :@::: #::: : :: :::::: :::@ :: ::::::@: | : :::: :: @: : :: :: .@::: :@::: #::: : :: :::::: :::@ :: ::::::@:
|
:::: ::. @: : :: :: :@::: :@::: .#::: : :: :::::: :::@ :::: ::::::@:
|
:::: ::: @: : :: :: :@::: :@::: :#::: : :: :::::: :::@ :::: ::::::@:
|
:::: ::: @: : :: :: :@::: :@::: :#::: : :: :::::: :::@ :::: ::::::@: 0 +----------------------------------------------------------------------->Gi 0 23.75

comment:8 Changed 12 years ago by charles

  • Component changed from GTK+ Client to libtransmission
  • Milestone changed from None Set to 1.73
  • Resolution set to fixed
  • Status changed from new to closed

Thanks for the quick confirmation :)

comment:9 Changed 12 years ago by charles

  • Summary changed from Linear increase in Memory usage while uploading to use less memory when uploading

comment:10 Changed 12 years ago by charles

  • Summary changed from use less memory when uploading to Use less memory when uploading by pruning overly-long write buffers

comment:11 Changed 12 years ago by charles

  • Priority changed from Normal to High

comment:12 Changed 12 years ago by charles

  • Severity changed from Major to Normal

comment:13 Changed 12 years ago by charles

  • Milestone 1.73 deleted
  • Priority changed from High to Normal
  • Resolution fixed deleted
  • Status changed from closed to reopened

According to irc user fl_, this fix causes high CPU use (100% for him) on some machines.

libevent 2.0's evbuffer rewrite should fix this automatically without the CPU load, so let's leave this open for a little while until we have a cleaner fix or until we jump to libevent2.

comment:14 Changed 12 years ago by Syam

I confirm too; with the memory usage fix, CPU is at 100% most of the time during data transfer. I should however comment that even without memory fix transmission CPU usage hits 100%.

Charles - Do you recommend any other workaround before eventual fix ?

comment:15 Changed 12 years ago by charles

Syam: if the CPU is still running at 100% then it's an issue unrelated to this ticket, so my recommendation would be to open a new ticket about it. If you're on OS X, use the Shark profiler as described on the Transmission wiki.

comment:16 Changed 12 years ago by charles

  • Type changed from Bug to Enhancement

comment:17 Changed 12 years ago by charles

  • Milestone set to Sometime

comment:18 Changed 11 years ago by charles

This ticket hasn't been updated in 14 months, but FWIW this is still a useful improvement. The status is same as before: this will go in after we jump to libevent2.

comment:19 Changed 10 years ago by charles

  • Resolution set to duplicate
  • Status changed from reopened to closed

Superceded by ticket #3836, "libevent2 support"

Note: See TracTickets for help on using tickets.