Opened 7 years ago

Closed 5 years ago

Last modified 5 years ago

#2338 closed Enhancement (fixed)

Add uTP support (BEP #29)

Reported by: someone Owned by: jordan
Priority: Normal Milestone: 2.30
Component: libtransmission Version: 1.73
Severity: Normal Keywords: bep patch-needed
Cc: jch@…, wisesage5001@…, User294, grzegorz.dubicki@…, mancubus220@…

Description

Here is the first draft of uTorrent's micro transport protocol (uTP), which uses UDP for peer-to-peer communication: http://svn.bittorrent.org/trac/browser/dotorg/trunk/html/beps/bep_0029.rst?rev=11158

uTP is designed to be a background network protocol that will yield to TCP traffic, making it much more network friendly. It also carries the side-benefit of being immune to TCP resets, which is often used by many ISPs for throttling.

Attachments (10)

congestion-control.patch (5.4 KB) - added by jch 6 years ago.
congestion-test.c (713 bytes) - added by jch 6 years ago.
0001-Allow-changing-the-congestion-control-algorithm.patch (6.3 KB) - added by jch 6 years ago.
Screen shot 2011-02-18 at 12.38.18 PM.png (67.0 KB) - added by x190 5 years ago.
cpu-01.diff (2.1 KB) - added by jordan 5 years ago.
cpu-02.diff (4.9 KB) - added by jordan 5 years ago.
cpu-03.diff (4.1 KB) - added by jordan 5 years ago.
cpu-04.diff (6.4 KB) - added by jordan 5 years ago.
daemon_utp_noutp_options.patch (1.6 KB) - added by er13 5 years ago.
low_resource_systems__smaller_buffers.patch (463 bytes) - added by er13 5 years ago.

Download all attachments as: .zip

Change History (135)

comment:1 Changed 7 years ago by charles

  • Keywords patch-needed added

comment:2 Changed 7 years ago by someone

Here is a nicely formatted version of the protocol (I think it's the same revision): http://www.bittorrent.org/beps/bep_0029.html

comment:3 Changed 7 years ago by charles

  • Summary changed from Implement uTorrent's UDP-based micro transport protocol (uTP) to Add uTP support

comment:4 Changed 7 years ago by charles

  • Summary changed from Add uTP support to Add uTP support (BEP #29)

comment:5 Changed 7 years ago by charles

  • Keywords bep added

comment:6 Changed 7 years ago by jch

Excellent! They've documented the protocol at last!

--Juliusz

comment:7 Changed 7 years ago by jch

After reading BEP 29, I'd like to say that the specification is incomplete. In particular, it doesn't say how we find out if a given peer supports µTP.

I'm also fairly fairly suspicious of the congestion control as described in the document; I suspect that the document is incomplete, and that the µTorrent people are doing more careful stuff. I strongly recommend that this specification should not be implemented unless and until the IETF declares it to be safe for use on the Internet.

--Juliusz

comment:8 Changed 7 years ago by jch

  • Cc jch@… added

comment:9 follow-up: Changed 7 years ago by lgerbarg

Having looked over the spec and what uTorrent does I actually think it is complete with respect to negotiation, just unclear. BEP 29 defines the uTP protocol. uTorrent just tries use it by connecting to a UDP port on the client, and if the handshake fails it falls back to TCP. I suppose technically there might be situations where clients could have distinct port ranges for UDP and TCP, but from all appearances they don't worry about that.

In other words, there is no explicit communication that a peer supports uTP, you find out by trying to use uTP against it.

comment:10 in reply to: ↑ 9 Changed 7 years ago by jch

Replying to lgerbarg:

there is no explicit communication that a peer supports uTP, you find out by trying to use uTP against it.

That made me worry, since it would imply a timeout for every connection to a non-µTP peer. Apparently, they try µTP and TCP in parallel.

http://forum.utorrent.com/viewtopic.php?id=63408

Since they're definitely not stupid people, they're probably doing something smarter than that, but I'm not in touch with any of the µTorrent developers. Charles, perhaps you could drop your pals a note?

--Juliusz

comment:12 follow-up: Changed 6 years ago by jch

The following patch (Linux-only) allows using TCP-LP, a less than best-effort sender-only modification of TCP, together with Transmission.

After applying this patch, do the following:

  • as root, type
      # modprobe tcp_lp
      # echo reno cubic lp > /proc/sys/net/ipv4/tcp_allowed_congestion_control
    
  • add the following line to your settings.json:
      "peer-congestion-algorithm" : "lp"
    

Please let me know the results.

--Juliusz

Changed 6 years ago by jch

comment:13 in reply to: ↑ 12 Changed 6 years ago by Ideal

  • Cc rotmer@… added

Replying to jch:

The following patch (Linux-only) allows using TCP-LP, a less than best-effort sender-only modification of TCP, together with Transmission.

Just tried this - almost i'd say nothing at all in difference compared to deluge doing the same downloads with the same connections amount.. Hydri says that tcp-lp requires support on the both ends, maybe because of this its this useless..

comment:14 follow-up: Changed 6 years ago by jch

Interesting, thanks for testing. What are your ping times like during the transfer?

[quote]Hydri says that tcp-lp requires support on the both endsquote

No, it doesn't. It only requires that the receiver implement TCP timestamps (RFC 1323). Almost everyone does.

Can you confirm that the code actually triggered? Did you have any warnings about being unable to set congestion control in the log?

--Juliusz

comment:15 in reply to: ↑ 14 Changed 6 years ago by Ideal

Replying to jch:

Interesting, thanks for testing. What are your ping times like during the transfer?

Something like this, i don't see any changes..

64 bytes from ds94.reliablehosting.com (216.131.94.54): icmp_seq=160 ttl=43 time=1979 ms

Hydri says that tcp-lp requires support on the both ends

No, it doesn't. It only requires that the receiver implement TCP timestamps (RFC 1323). Almost everyone does.

Well, maybe.. But on practice anyway then not much difference at least here..

Can you confirm that the code actually triggered? Did you have any warnings about being unable to set congestion control in the log?

Yea, i put some prints like this:

            if(rc < 0) {
...
            } else {
                printf( "Set congestion control algorithm: %s\n",
                         strerror(errno));
            }

in both places from your patch and it printed these things:

...
Set congestion control algorithm: Operation now in progress
Set congestion control algorithm: Operation now in progress
Set congestion control algorithm: Operation now in progress
Set congestion control algorithm: Success
Set congestion control algorithm: Success
Set congestion control algorithm: Success
...
Set congestion control algorithm: Success
Set congestion control algorithm: Success
Set congestion control algorithm: Success
Set congestion control algorithm: Resource temporarily unavailable
Set congestion control algorithm: Resource temporarily unavailable
Set congestion control algorithm: Success
Set congestion control algorithm: Success
Set congestion control algorithm: Success
...

comment:16 follow-up: Changed 6 years ago by jch

Hmm... what kernel is that?

--Juliusz

comment:17 in reply to: ↑ 16 Changed 6 years ago by Ideal

Replying to jch:

Hmm... what kernel is that?

--Juliusz

plisk ~ # uname -a Linux plisk 2.6.32-rc6-git3 #3 SMP PREEMPT Sun Nov 8 20:20:04 EET 2009 x86_64 Intel(R) Core(TM)2 CPU 6400 @ 2.13GHz GenuineIntel? GNU/Linux

comment:18 Changed 6 years ago by ericperko

  • Cc wisesage5001@… added

comment:19 Changed 6 years ago by jch

Hmm... I'm completely stumped.

--Juliusz

Changed 6 years ago by jch

comment:20 follow-up: Changed 6 years ago by jch

Perhaps you can try working that out with congestion-test.c? Try

   ./a.out reno
   ./a.out cubic
   ./a.out lp

--Juliusz

comment:21 in reply to: ↑ 20 Changed 6 years ago by Ideal

Replying to jch:

Perhaps you can try working that out with congestion-test.c? Try

   ./a.out reno
   ./a.out cubic
   ./a.out lp

--Juliusz

I haven't enabled that cubic so it fails..

plisk@plisk ~/tmp/1 $ ./a.out reno Yep, setsockopt(TCP_CONGESTION, reno) works fine. plisk@plisk ~/tmp/1 $ ./a.out lp Yep, setsockopt(TCP_CONGESTION, lp) works fine. plisk@plisk ~/tmp/1 $ ./a.out cubic setsockopt(TCP_CONGESTION): No such file or directory

comment:22 Changed 6 years ago by charles

jch: is there any progress on this? Should / could this ticket be milestoned for 1.80?

comment:23 Changed 6 years ago by jch

Should / could this ticket be milestoned for 1.80?

Do you mean µTP, or the patch to tweak the congestion control algorithm?

If the latter, then It Works For Me (TM). The failure report above was on a hacked non-standard unreleased kernel, and doesn't make sense to me -- it indicates that setsockopt returns -1 but doesn't set errno.

Furthermore, my patch doesn't do anything unless the user manually edits settings.json. Hence, it should be safe to deploy.

Unless you can see a reason why it could break anything, I suggest committing.

--Juliusz

comment:24 Changed 6 years ago by Ideal

  • Cc rotmer@… removed

comment:25 Changed 6 years ago by User294

  • Cc User294 added

As for me, any attempts to do something with TCP flow control looks like a fruitless effort. Look, you can do whatever you want on local machine with it's scheduling, BUT that's where one problem occurs: what if there is router on the way? You can schedule packets on fast local 100MBps link to router as you want them to. But then packets hit router(s) which have to forward them to ISP. Router usually lacks 100MBps link to Internet and hence have to buffer all your data if they arrive fast and send them as actual connection rate permits (and it's much slower than 100MBps).

So: 1) Your scheduling efforts for TCP are useless in many configurations. Routers would use their own algos to (re)schedule your (buffered) data packets queued in their queues. 2) As the result, usually you have too much stuff in output buffers of router and awfully bad ping times since there is ton of TCP connections who can't accurately measure state of link and transmit direction getting jammed on DSL and similar slow-send links.

There is not much you can do with TCP about this issue. On other hand UDP-based protocol can provide low-level link status data (like round-trip times and packets loss rate) quite easily so it's possible to dynamically readjust speeds according to actual state of things (i.e. reduce speed when link becomes busy and round-trip times are getting high, more packets getting lost, etc). Also in theory it's possible to do STUN-like tricks to connect two peers behind NATs via UDP by playing certain small tricks (like STUN does).

comment:26 Changed 6 years ago by grzegorzdubicki

  • Cc grzegorz.dubicki@… added

What's the status of this enhancement?

Because I think that because:

  1. uTorrent v 2.0 with uTP support has been released about 3 weeks ago (http://blog.bittorrent.com/2010/02/03/%c2%b5torrent-v2-0-stable-release/),
  1. most uTorrent clients has already been autoupdated to this version (as I see in my torrents statistics)
  1. uTorrent is at least one of the most popular bt clients (according to http://forum.tribler.org/viewtopic.php?f=2&t=368, for example)

adding this feature may make a real difference now.

comment:27 Changed 6 years ago by charles

The status is that nothing's been done on it. This is going to be a major undertaking. Unless my timeline is wrong, uTorrent's had paid programmers working on this feature for over a year and they still don't have all the details hammered out, and it's still not fully documented.

Transmission is a much smaller and unfunded project. In order for us to adopt uTP we will need at least two things:

  • A fully written spec. According to Switeck, the uTorrent developers have made some major changes since the BEP was written.
  • Another programmer who's well-versed with the ins and outs of UDP programming

One alternative would be to wait for libtorrent-rasterbar to implement uTP and use that as our reference implementation. Even if uTP has undocumented changes, libtorrent-rasterbar has a leg up on the competition because its author works for uTorrent and has either written parts of uTP or can at least read its source code, which is something that the authors of other engines like rTorrent, KTorrent, Transmission, and Vuze can't do.

comment:28 Changed 6 years ago by jch

Folks,

Please read

http://forum.bittorrent.org/viewtopic.php?pid=750

and make sure that you've understood it.

--Juliusz

comment:29 Changed 6 years ago by charles

http://forum.bittorrent.org/viewtopic.php?pid=1014#p1014

"So in short, there is a standard, and µTorrent doesn't follow it, and I'm left reverse engineering the protocol. How nice."

comment:30 Changed 6 years ago by grzegorzdubicki

charles, jch: thank you for letting us know that you monitor this feature request's situation and for clearing up it's current status! :)

comment:31 Changed 6 years ago by jch

Charles, what about applying congestion-control.patch?

comment:32 Changed 6 years ago by ghazel

That header documentation/implementation mismatch was fixed: http://forum.bittorrent.org/viewtopic.php?pid=1038#p1038

comment:33 Changed 6 years ago by jch

Charles,

This version fixes the tr_strdup and missing tr_free. As to your other objections:

  • NULL and "" mean "do not change your system's default"; this is handled in tr_peerIoNew.
  • If peer_congestion_algorith in NULL or "", then the function setCongestionControl will never be called, hence no message.

As mentioned, this code changes nothing by default, and is safe to apply.

--Juliusz

comment:34 Changed 6 years ago by charles

the TCP_CONGESTION patch has been moved to ticket #3162

comment:35 Changed 6 years ago by charles

  • Milestone changed from None Set to 2.10

I haven't reviewed libutp yet, so this is just a tentative milestoning.

comment:36 Changed 6 years ago by jch

Milestone changed from None Set to 2.10

Charles, I very strongly object.

A number of serious objections to µTP have been raised:

You will notice the complete lack of replies that make sense to any of these objections.

So let me make this clear:

  • µTP (the framing mechanism) is broken beyond repair;
  • LEDBAT (the delay-based mechanism) might or might not be a good idea; at any rate, it belongs in TCP, not in a broken, proprietary framing mechanism.

I would like to very strongly recommend that Transmission should not implement µTP until the above points are adressed in a satisfactory manner by the µTP proponents.

--jch

comment:37 Changed 6 years ago by ghazel

jch:

I have trouble seeing how any of these are objections serious enough to prevent use. Also I believe each has been addressed before, but I'll reply to them anyway.

  1. µTP (the framing mechanism) is broken beyond repair

I'm not sure what you mean by "broken beyond repair" since it actually works and can be improved upon. I'm going to guess, though, that you mean there is no MTU discovery. Our solution is a to guess at MTU based on address type, and use the most conservative packet sizes for each scenario. Since we just released the source to uTP, you can see the MTU calculation we do in utp_utils.cpp Nothing, however, is stopping more explicit MTU discovery from being added to the implementations should there be a case this does not handle well.

2a. LEDBAT (the delay-based mechanism) might or might not be a good idea;

In our measurements at massive scale and the extensive study of the congestion controller at Internet2, the LEDBAT delay-based congestion controller does perform well. A key property is that in the worst case it reacts with the same behaviors as TCP on loss and timeout, so it will not "break the internet" should the delay be unrelated to congestion.

2b. at any rate, it belongs in TCP, not in a broken, proprietary framing mechanism.

It would be great if this were implemented as an option in TCP. LEDBAT is the first step towards this goal, not to mention each OS which has to implement it, and get it right. TCP-based BitTorrent? has a problem today, and a UDP-based transport solves it. I believe it is even stronger proof that the OSes should support it if a wide-scale application like BitTorrent? has adopted it.

3?. uTP uses small packets (sometimes, if misconfigured)

This complaint by basically one forum member is a non-issue for adopting uTP elsewhere. The dynamic packet size code is uTorrent specific feature, and is not required by or related to implementing uTP. Packets of the full size (up to the MTU) may be used at all times.

In any case, uTP is open source now. You may test it yourself for issues, and we can all contribute to making BitTorrent? clients interoperate without smashing the internet with congestion.

http://github.com/bittorrent/libutp

Last edited 6 years ago by ghazel (previous) (diff)

comment:38 Changed 6 years ago by jch

Dear Ghazel,

As I have extensively explained on the BitTorrent?.org forums, using an idiosyncratic, ad-hoc, mis-designed framing scheme causes a number of serious issues which can be avoided simply by layering LEDBAT over TCP. (Note that I am speaking about µTP here, and deliberately avoiding discussing LEDBAT, which I am not competent to judge.)

Since I have already spent a non-negligible amount of my copious free time writing up my criticism, I refuse to repeat it here. I suggest we move this discusion back where it belongs, namely to

http://forum.bittorrent.org/viewtopic.php?id=131

Regards,

--Juliusz

comment:39 Changed 6 years ago by grzegorzdubicki

Maybe this changes the situation somehow:

http://arstechnica.com/open-source/news/2010/05/bittorrent-open-sources-new-protocol-implementation.ars

EDIT: Oops, I have just noticed this was already mentioned above...

Last edited 6 years ago by grzegorzdubicki (previous) (diff)

comment:40 Changed 6 years ago by User294

There is at least one major issue with uTP exists at the moment: when network is congested, uTP resorts to very small packets. The problem is: what if congestion occurs not in user's router buffer but anywhere else, i.e. say, ISP network runs on full capacity, some router reached it's limits, or some remote channels were saturated? Then, uTP dumbly reduces packet size. The "only" problem is that this behavior makes things even worse and leads to some kind of collapse. This increases total PPS (packets per seconds) rate in channels, making routers jobs even harder. Even some decent ISP routers may fail to cope with such "gift" and user's modems and other routers are often just running out of CPU power when trying to process such PPS. Also there is more headers to send and still same amount of data to transfer. Hence more stuff to transfer in total. More headers, less useful data. Surely, overloaded channels really need to be overloaded even better. So, while uTP solves particular problem and nice idea, actual implementation creates new troubles: it actually helps networks to collapse under heavy loads when there is any "global" congestion occurs. So, some ISPs even have started filtering out uTP packets to reduce network loads. As for me, it have to be reimplemented without resorting to small packets.

comment:41 Changed 6 years ago by ghazel

The small packet issue was not a problem with uTP at all, thus libutp was not affected. It was a problem with uTorrent's rate limiter, and it has been fixed for 2.0.3.

comment:42 Changed 6 years ago by rafi

ghazel wrote: uTP uses small packets ... This complaint by basically one forum member ... ...and it has been fixed for 2.0.3

I also believe that implementation issues should be set apart from the uTP core protocol/lib, and not hold up other clients from implementing it. Still, uTorrent is kind of feasibility demo for it, and I'm still eager to see a test with 2.0x demonstrating how effective it is in congestion control.

I'm glad to see positive corrective action on behalf of uTorrent devs (though delayed by ~6 months since being reported ... ). It surely looks much improved in 2.03 beta, and reduce both payload's overhead and it's related PPS. Though, connections related PPS can still be much improved.

Last edited 6 years ago by rafi (previous) (diff)

comment:43 Changed 6 years ago by charles

  • Milestone changed from 2.10 to Sometime

This is still on the TODO list but it's a large task and the rest of the checklist for 2.10 is almost done. Punting this to post-2.10

comment:44 Changed 5 years ago by georg123

How is uPT going then... are we going to see it any time soon?

comment:45 Changed 5 years ago by charles

No progress to report atm.

comment:46 Changed 5 years ago by jch

You can see what I've been up to by doing

  git clone git://git.wifi.pps.jussieu.fr/transmission.git
  cd transmission
  git checkout utp

[Edit: changed the name of the server above. Please update your remotes.]

It doesn't do µTP yet, but I've cleanly split UDP handling from the DHT, so that libutp should be easy to plug in. Charles, I'll tell you more about my plans by private mail.

--jch

Last edited 5 years ago by jch (previous) (diff)

comment:47 Changed 5 years ago by jch

I've put my private clone of libutp on

https://github.com/jech/libutp

Just do

  git clone git://github.com/jech/libutp.git

--jch

comment:48 Changed 5 years ago by jch

The utp branch has been rebased, the old utp branch is now called utp-20110109 and won't be updated any more.

--jch

comment:49 follow-up: Changed 5 years ago by jch

This evening, shortly before midnight, Transmission made its first download over µTP.

http://www.pps.jussieu.fr/~jch/software/bittorrent/utp.png

"T" stands for µTP.

comment:50 Changed 5 years ago by jch

At Jordan's request, this has moved to GitHub?: https://github.com/jech/transmission

  git clone -b utp git://github.com/jech/transmission.git

--jch

comment:51 Changed 5 years ago by jch

Charles,

These are the easy tasks:

  • import libutp from git://github.com/jech/libutp.git, integrate it into the build process;
  • define a userpref for UTP support; should have three values, disabled, incoming-only, fully enabled;
  • make tr_peerIoNewOutgoing take a tr_bool utp argument; note that tr_peerIoNew already groks µTP;
  • make peer-mgr try µTP, fall back to TCP after a timeout. The strategy should be different for different kind of atoms -- PEX carries a flag to say if µTP is supported;
  • do the right thing, whatever that is, in peerIoReconnect.

These are the more tricky ones:

  • implement read flow control -- we currently read as much data as the peer will give us;
  • implement write flow control -- I'm pretty sure we're doing it all wrong right now;
  • work out how computing overhead works -- see utp_on_overhead, which currently does nothing.

--jch

Last edited 5 years ago by jch (previous) (diff)

comment:52 Changed 5 years ago by jordan

import libutp from git://github.com/jech/libutp.git, integrate it into the build process;

Done. https://github.com/jech/transmission/pull/1

comment:53 follow-up: Changed 5 years ago by jordan

jch, what is the use case for incoming only? and, is it a use case that many users will encounter? at first glance it seems like most users would be better off with an enabled/disabled toggle.

comment:54 in reply to: ↑ 49 Changed 5 years ago by rafi

Replying to jch:

This evening, shortly before midnight, Transmission made its first download over µTP.

http://www.pps.jussieu.fr/~jch/software/bittorrent/utp.png

"T" stands for µTP.

Congrats... about time... ;) now do upload... my uTorrent needs you... :P

comment:55 Changed 5 years ago by jordan

  • define a userpref for UTP support; should have three values, disabled, incoming-only, fully enabled;

Preliminary userpref for enable/disable added to https://github.com/jordanl/transmission. This can be widened to a tristate as necessary.

comment:56 Changed 5 years ago by jch

Pulled, although I don't necessarily agree with "make a note" -- I'm not planning to do any packet size variation in Transmission (we've discussed this in detail with Arvid, and he's not really sure it's a good thing either).

Please make sure you pull from me before you continue working.

--jch

comment:57 Changed 5 years ago by jch

Okay, I've done read pacing and overhead computation. Still not so sure about write pacing.

--jch

comment:58 follow-up: Changed 5 years ago by grzegorzdubicki

Do you think µTP support will be ready in 2.20? :)

comment:59 in reply to: ↑ 58 Changed 5 years ago by jch

Replying to grzegorzdubicki:

Do you think µTP support will be ready in 2.20? :)

Certainly not. We haven't even decided whether it'll ever go in Transmission.

My personal opinion is that the issues solved by µTP are better solved by implementing LEDBAT within the TCP stack; see #3162. See also https://forum.bittorrent.org/viewtopic.php?id=131 .

--jch

comment:60 Changed 5 years ago by jch

Outgoing connections implemented. Not using them anywhere yet.

I'm seeing wide instabilities in read bandwidth, which indicates that the bandwidth management code might need to be tweaked for µTP; it's more likely that since the bandwidth measurement performed with µTP is lower down the stack (there are no kernel buffers for µTP), it simply needs to be smoothed before being displayed to the user.

--jch

comment:61 Changed 5 years ago by jch

The instabilities have been fixed. Remains doing the write side properly, it currently deadlocks.

comment:62 in reply to: ↑ 53 ; follow-up: Changed 5 years ago by user

Replying to jordan:

jch, what is the use case for incoming only? and, is it a use case that many users will encounter?

Outgoing entails making lots of probably-going-to-fail uTP attempts. Accepting only incoming uTP, not so much. Weak networking will choke-and-die from heavy uTP packet rates, more from number of connections than speed. ...though some to treat each UDP packet as though it were a separate session/connection! uTorrent v1.8.x did incoming only by default to ease the appearance of uTP ...devs could test "against" uTorrent v1.8.x clients in the wild without almost the whole customer base running uTP. Unfortunately, that's probably not so much an option now...since uTorrent itself is "firing blind" with both TCP and uTP at ips gathered from the tracker and probably DHT.

comment:63 in reply to: ↑ 62 Changed 5 years ago by rafi

Replying to user:

Replying to jordan:

jch, what is the use case for incoming only? and, is it a use case that many users will encounter?

Outgoing entails making lots of probably-going-to-fail uTP attempts. Accepting only incoming uTP, not so much. Weak networking will choke-and-die from heavy uTP packet rates, more from number of connections than speed. ...though some to treat each UDP packet as though it were a separate session/connection! uTorrent v1.8.x did incoming only by default to ease the appearance of uTP ...devs could test "against" uTorrent v1.8.x clients in the wild without almost the whole customer base running uTP. Unfortunately, that's probably not so much an option now...since uTorrent itself is "firing blind" with both TCP and uTP at ips gathered from the tracker and probably DHT.

uTorrent (2.2x) can still control in/out direction for both TCP and uTP in bt.transp_disposition advanced settings. Practically it has not use now. Tho, in the past, it was useful for testing. Plus it makes it possible to be able to 'cooperate' with other uTP clients, while minimizing outgoing traffic, that can heart crappy routers...

comment:64 Changed 5 years ago by jch

I've just finished trying µTP and falling back to TCP.

So what remains is:

  • fix tr_peerIoReconnect to do µTP (it currently does TCP unconditionally);
  • fix or remove what Jordan calls "phase 2" of bandwidth computation;
  • testing.

--jch

Last edited 5 years ago by jch (previous) (diff)

comment:65 Changed 5 years ago by jordan

  • Milestone changed from Sometime to None Set

comment:66 Changed 5 years ago by jch

I've just merged the latest changes to libutp, and rebased to the current head.

--jch

comment:67 Changed 5 years ago by jordan

Thank you jch!

comment:68 Changed 5 years ago by jch

We've just merged the utp branch into the Transmission trunk. The main change is that it is now possible to decide at compile time whether utp is enabled or disabled (configure --enable-utp).

We need to decide how the default build behaves. My recommendation, for now would be

  • include µTP support by default (plain ./configure) if a C++ compiler can be found;

but

  • disable µTP by default at runtime (set utp-enabled to false).

This last point is the safe choice, and the µTP preference is sufficiently easy to find in the configuration, at least in the Gtk+ interface.

--jch

comment:69 Changed 5 years ago by rafi

Unless your uTP performance sucks - the proper choice would be to enable it. From my uTorrent experience - an intermediate choice (till you gain some experience in the "wild"...) might be to disable uTP output (initiation of uTP connections) but enable input (respond to uTP requests).

just my 2 cents...

comment:70 follow-ups: Changed 5 years ago by jordan

Alternatively, is there a known downside to building libutp unconditionally? I doubt there are many environments building Transmission that don't have a C++ compiler built in. Making that a precondition would allow us to get rid of some ugly #ifdefs in libtransmission.

On the user-configuration level, utp-enabled should be default to 'false', at least for now.

comment:71 in reply to: ↑ 70 ; follow-up: Changed 5 years ago by x190

Replying to jordan:

On the user-configuration level, utp-enabled should be default to 'false', at least for now.

Should be, but isn't on Mac Client. r11970

Whatever it's trying to do here isn't working.

2011-02-17 23:11:44 -0700 fdlimit.c:797 [Debug] Transmission: setrlimit( RLIMIT_NOFILE, 1024 )

2011-02-17 23:11:44 -0700 fdlimit.c:805 [Debug] Transmission: socket limit is 100

2011-02-17 23:11:44 -0700 tr-udp.c:53 [Error] UDP: Failed to set receive buffer: No buffer space available

2011-02-17 23:11:44 -0700 tr-udp.c:72 [Error] UDP: Failed to set receive buffer: requested 4194304, got 42080

2011-02-17 23:11:44 -0700 rpc-server.c:807 [Info] RPC Server: Adding address to whitelist: 127.0.0.1

This is where buffer size is normally set-up and does now succeed.

2011-02-17 23:14:08 -0700 Torrent.m:318 [Info] 1998 - : restarting via startTransfer

2011-02-17 23:14:17 -0700 fdlimit.c:645 [Debug] Transmission: SO_SNDBUF size is 131070

2011-02-17 23:14:17 -0700 fdlimit.c:647 [Debug] Transmission: SO_RCVBUF size is 262140

The following is some stuff I played around with:

Disabled/(wait a few mins)re-enabled uTP in prefs:

2011-02-17 23:41:22 -0700 tr-udp.c:53 [Error] UDP: Failed to set receive buffer: No buffer space available

2011-02-17 23:41:22 -0700 tr-udp.c:72 [Error] UDP: Failed to set receive buffer: requested 4194304, got 32768

Subsequently lost all connections/all speed. Pause/Resume? re-establishes connections.

Real Memory 30-50MB in 40 min (12MB more than a non-uTP client r11889 which appears to stabilize at 38 MB and then slowly drops to 32-34MB with the same load).

Connections per second 80-500 for 2 torrents, 75 peers vs. 1.5/second for non-uTP client.

Are our ISPs, routers, and console logs going to love this??? Thankfully, it's configurable.

Last edited 5 years ago by x190 (previous) (diff)

comment:72 Changed 5 years ago by jordan

2011-02-17 23:11:44 -0700 fdlimit.c:805 [Debug] Transmission: socket limit is 100

There is no fdlimit.c:805. Please report bugs (and I'm sure there are many ;) against unpatched versions of trunk. Thank you!

Last edited 5 years ago by jordan (previous) (diff)

comment:73 in reply to: ↑ 70 Changed 5 years ago by jch

Replying to jordan:

Alternatively, is there a known downside to building libutp unconditionally? I doubt there are many environments building Transmission that don't have a C++ compiler built in. Making that a precondition would allow us to get rid of some ugly #ifdefs in libtransmission.

I'm a little worried about embedded builds. I know that OpenWRT includes a C++ cross-compiler, but how many other embedded systems are either including Transmission or are considering doing so?

I'd much rather we did more work on cleaning up the ifdefs.

--jch

comment:74 in reply to: ↑ 71 ; follow-up: Changed 5 years ago by jch

Replying to x190:

2011-02-17 23:11:44 -0700 tr-udp.c:53 [Error] UDP: Failed to set receive buffer: No buffer space available

2011-02-17 23:11:44 -0700 tr-udp.c:72 [Error] UDP: Failed to set receive buffer: requested 4194304, got 42080

Please see https://trac.transmissionbt.com/changeset/11956 . Check notably the #ifdef linux bits. I'd be grateful if you could either find the equivalent sysctl on Mac OS, or find the largest values that work for you.

(The failure shouldn't harm much, though, it'll just reduce performance of µTP on busy systems.)

2011-02-17 23:14:17 -0700 fdlimit.c:645 [Debug] Transmission: SO_SNDBUF size is 131070

2011-02-17 23:14:17 -0700 fdlimit.c:647 [Debug] Transmission: SO_RCVBUF size is 262140

That's TCP sockets, not related to µTP.

Subsequently lost all connections/all speed. Pause/Resume? re-establishes connections.

That's not related to the above. I haven't seen that.

Connections per second 80-500 for 2 torrents, 75 peers vs. 1.5/second for non-uTP client.

How are you measuring that? We shouldn't be connecting over µTP any faster than we do over TCP.

--jch

comment:75 in reply to: ↑ 74 Changed 5 years ago by x190

Replying to jch:

Replying to x190:

2011-02-17 23:11:44 -0700 tr-udp.c:53 [Error] UDP: Failed to set receive buffer: No buffer space available

2011-02-17 23:11:44 -0700 tr-udp.c:72 [Error] UDP: Failed to set receive buffer: requested 4194304, got 42080

Please see https://trac.transmissionbt.com/changeset/11956 . Check notably the #ifdef linux bits. I'd be grateful if you could either find the equivalent sysctl on Mac OS, or find the largest values that work for you.

Maybe this one (net.inet.udp.recvspace: 42080) based on log above, but what amount would be prudent to use? Also, exactly what code do I edit?

Subsequently lost all connections/all speed. Pause/Resume? re-establishes connections.

That's not related to the above. I haven't seen that.

The problem starts when I re-establish use of uTP, not on disabling it. I suppose other clients get confused about my type of connection? Not sure if it would recover with time but a pause all/resume all fixes the problem.

Connections per second 80-500 for 2 torrents, 75 peers vs. 1.5/second for non-uTP client.

How are you measuring that? We shouldn't be connecting over µTP any faster than we do over TCP.

--jch

Please see attachment which shows periodic spikes to 6 every 10 seconds using r11889 and 80 connected peers, no DHT or LPD, and browser to post. Same conditions with r11970 produced a steady average of ~80 spiking to 200-300. Max of 500 was seen for several seconds at application start-up.

comment:76 Changed 5 years ago by jch

AFAIK, PeerGuardian? doesn't interpret µTP yet. The data is most likely spurious.

--jch

comment:77 Changed 5 years ago by jch

Oh, one thing comes to mind.

We're currently not handling the case of an outgoing connexion to a µTP peer that doesn't do encryption. I.e. if we're configured to attempt encrypted connexions, we do

(1) encrypted µTP; on failure

(2) encrypted TCP; on failure

(3) plaintext TCP.

(We're slightly smarter than that, actually, if (1) fails during the handshake, we skip to (3) directly.)

Now what that means is that we never attempt a plaintext µTP connexion -- if the peer fails encrypted µTP, we move to plaintext TCP straight away. That's good enough for me, but if anyone wants to try his hand at fixing that, the fun happens in handshake.c:gotError which interacts in marvelous ways with peer-io.c:tr_peerIoReconnect.

--jch

comment:78 Changed 5 years ago by jch

Another note -- we're not currently shutting down the µTP sockets at Transmission shutdown, which causes our peers to timeout.

--jch

comment:79 follow-up: Changed 5 years ago by jordan

jch, I'm not seeing that behavior. If I add a global UTPSocket counter to libtransmission, it keeps balancing out to 0 at the end. How should I trigger the behavior you're describing?

comment:80 in reply to: ↑ 79 Changed 5 years ago by jch

Replying to jordan:

If I add a global UTPSocket counter to libtransmission, it keeps balancing out to 0 at the end.

Ah, sorry -- I didn't realise we're properly destroying all the peer-ios. I was under the impression we were just trusting the OS to shut down the sockets (which obviously doesn't happen in the case of µTP sockets).

So all's right -- it looks like the only remaining issue is the throughput controller.

--jch

comment:81 Changed 5 years ago by jordan

I checked in some simple code for throughput control last night in r12063. It's still preliminary and will benefit from testing/tuning, but it's much closer to the desired speed levels than earlier builds.

comment:82 Changed 5 years ago by livings124

  • Milestone changed from None Set to 2.30

comment:83 Changed 5 years ago by jordan

r12103: (trunk libT) #2338: "uTP support" -- enable uTP by default.

Last edited 5 years ago by jordan (previous) (diff)

comment:84 Changed 5 years ago by jch

Jordan, could you please explain r12063?

--jch

comment:85 Changed 5 years ago by gunzip

transmission-daemon 2.22 (12088)
transmission-daemon 2.22+ (12117)

just a couple of observations comparing the above two builds (i have both installed in Linux Debian):

1) with utp running the CPU is about twice that for the same torrent using the 2.22 release 2) also, the upload rate with utp is not as steady, i.e., has a tendency to swing up and down more when observing in real time.

overall utp-enabled torrents are working fine, but with the above 2 caveats

comment:86 Changed 5 years ago by jch

Thanks for the report, gunzip. Both of these are known issues.

On what platform are you seeing the CPU increase? And are you measuring just user, or user+system?

(While the libutp does have a number of places where it's not as efficiently coded as it should be, I wouldn't expect the CPU load to as much as double.)

--jch

comment:87 Changed 5 years ago by jch

Oh, and one important thing that I'd like to be sure of: the CPU load goes back down to what it was in 2.20 when you disable µTP, right? That's really important if we want Transmission to remain useful on embedded hardware.

--jch

comment:88 follow-up: Changed 5 years ago by jordan

gunzip: there have been a lot of changes in trunk since the 2.2x branch was forked, so without more information it's hard to know whether the increased CPU load is a uTP issue or not. If you disable uTP and restart, does that have any effect on CPU load?

Also, would it be possible to pastebin a perf profile of where trunk is using your cycles?

comment:89 Changed 5 years ago by gunzip

those observations in comment:85 were based on an ancient Dell Pent-3 500MHz i386 box.

and yes, running the 2.22+ version with --no-utp does return the CPU levels down to that of the 2.22 release.

the CPU was monitored with "top -p <daemon PID>" and was by no means scientific, but looking at 3 torrents the usage with utp enabled is higher by somewhere in the range of 1.5 to 2 times.

comment:90 in reply to: ↑ 88 Changed 5 years ago by gunzip

Replying to jordan:

Also, would it be possible to pastebin a perf profile of where trunk is using your cycles?


i did run oprofile for 2.22+ with and without utp enabled, repeating on the same torrent:

http://pastebin.com/xF0zZTMH (with utp enabled)

http://pastebin.com/BmTMfauN (with utp disabled)

both outputs were made using

$ opreport -l ./transmission-2.22+r12121/daemon/transmission-daemon

and here's an update with a more recent build:

http://pastebin.com/SVswpcNy (r12137 with utp enabled)

Last edited 5 years ago by gunzip (previous) (diff)

Changed 5 years ago by jordan

comment:91 follow-up: Changed 5 years ago by jordan

gunzip, could you try applying cpu-01.diff and run another cpu profile with utp enabled?

comment:92 in reply to: ↑ 91 Changed 5 years ago by gunzip

Replying to jordan:

gunzip, could you try applying cpu-01.diff and run another cpu profile with utp enabled?


http://pastebin.com/w1aUY5mr

transmission-daemon 2.22+ (12168) with cpu-01.diff and utp-enabled

comment:93 Changed 5 years ago by jordan

Well that's a good improvement. The most expensive function has dropped down to #12 in the list.

The #1 and #2 functions are tr_isPeerIo() and tr_bandwidthClamp() now. Let's try inlining tr_isPeerIo() to see if that makes any difference. Also I see a way to avoid some number crunching in tr_bandwidthClamp().

gunzip, could you apply cpu-02.diff to r12168 or higher and report back? :)

Changed 5 years ago by jordan

comment:94 Changed 5 years ago by gunzip

r12168 with cpu-02.diff keeps crashing when i add a torrent:

transmission-daemon: bandwidth.c:313: tr_bandwidthSetPeer: Assertion `( peer == ((void *)0) ) || tr_isPeerIo( peer )' failed. Aborted

comment:95 Changed 5 years ago by rb07

Just for the record: It works fine under Windows, using Transmission-Qt of course, revision 12137.

Only one test, about 99.9% of the peers used uTP, no high CPU use at all, bandwidth was see-saw but I don't know if it would be any different not using uTP and in average was pretty good.

The important thing is: great work guys!

comment:96 Changed 5 years ago by jordan

gunzip: whoops! try cpu-03.diff :)

Changed 5 years ago by jordan

comment:97 Changed 5 years ago by jordan

rb07: thanks for the Windows update. I was wondering if any of these code changes in libtransmission had broken anything in mingw...

Changed 5 years ago by jordan

comment:98 Changed 5 years ago by jordan

even better, try cpu-04.diff. I found & fixed the inlining issue that was in cpu-02.diff...

comment:99 Changed 5 years ago by gunzip

OK, here is with cpu-04.diff and utp-enabled

http://pastebin.com/Bbs4Na7B

comment:100 follow-up: Changed 5 years ago by jordan

r12170 libtransmission/ (bandwidth.c bandwidth.h peer-io.c peer-io.h): (trunk libT) cpu load improvements based on profiling by gunzip

I'm wondering what the load / profiling looks like with r12170 with uTP enabled and disabled now. Are enabled and disabled closer together in wrt CPU load? What is the load like compared to a few days ago before we started testing CPU cycles?

comment:101 follow-up: Changed 5 years ago by jch

Hmm, not a big fan of inlining isPeerIO -- it bloats the code uselessly. Wouldn't it be better to simply remove some of the asserts?

--jch

comment:102 in reply to: ↑ 101 Changed 5 years ago by jordan

Replying to jch:

Hmm, not a big fan of inlining isPeerIO -- it bloats the code uselessly.

In the abstract I agree about inlining bloat.

The reason I let this one through is that r12170 didn't trigger gcc's warning that inlining function X will grow the code.

If there's a better test that should use instead of gcc's warning system, I would love to hear about it, because that's something I've wondered about too.

Wouldn't it be better to simply remove some of the asserts?

Yes, and I did remove some redundant ones in r12170. And, of course, assertions are compiled out of release builds anyway. IMO tr_isPeerIo() is not the most interesting line in that profile list.

I'm more surprised that bandwidthClamp() didn't drop more in cpu-04.diff. Of the libtransmission functions, bytesUsed() is the next highest on the list, and I don't see any way at all to tweak that.

Of the libutp functions, the most expensive function is UTPSocket::check_timeouts()... we could try increasing our periodic call to UTP_CheckTimeouts() from 50ms to some higher value. Is 50ms that much better than, say, 100ms? The comment in the code says alus "says 50ms works for them".

comment:103 Changed 5 years ago by ghazel

The behavior of running check_timeouts less often than every 50ms is Untested. I know at intervals of seconds it has Bugs. Could you provider more insight into which part of check_timeouts is taking all the time? It's a fairly simple function.

comment:104 in reply to: ↑ 100 Changed 5 years ago by gunzip

Replying to jordan:

r12170 libtransmission/ (bandwidth.c bandwidth.h peer-io.c peer-io.h): (trunk libT) cpu load improvements based on profiling by gunzip

I'm wondering what the load / profiling looks like with r12170 with uTP enabled and disabled now. Are enabled and disabled closer together in wrt CPU load? What is the load like compared to a few days ago before we started testing CPU cycles?

http://pastebin.com/tRQkrckx (report-r12170-utp-disabled)

http://pastebin.com/qfTfWEQs (report-r12170-utp-enabled)

since the total percent always adds to 100% in those profile reports, if you push something down something else has to go up?

in terms of the CPU load on my system there is no noticable difference now between utp enabled/disabled:

http://i55.tinypic.com/9jhvso.png

thats a 5-minute time period sampled every second using top command in batch mode, and it's the CPU of the daemon only. also, the tests were started shortly after the torrent was added and when I always see the highest CPU rates. when the torrent reaches seeding, the CPU drops considerably in both cases.

comment:105 Changed 5 years ago by jordan

UTPSocket::check_timeouts() is much lower in these new profiles, for whatever reason, even though we're calling it just as frequently as before. Maybe these profiles are "noisy" and we should look at trends rather than individual data points? It's possible that check_timeouts() isn't as bad off as it looked earlier.

In both new reports, bandwidthClamp() is still the most expensive function, and probably could be banged on some more.

tr_isPeerIo() is in second place in both reports, but since it'll be compiled out of the official builds, I'm not worried about that.

Anyway, since we've worked things down to the point where using uTP doesn't cost any more than not using uTP, that solves the issue for this uTP ticket :)

comment:106 Changed 5 years ago by jch

I'm not sure that holds across all arches.

Recall that µTP computes all timeouts in user-space; that's fine on systems that implement clock_gettime as a vsyscall, but I'd like to see a profile on a system that doesn't (e.g. Linux/MIPS, as in most cheap NAS systems).

-- jch

comment:107 Changed 5 years ago by blagishnessosity

  • Cc mancubus220@… added

comment:108 follow-up: Changed 5 years ago by calin.burloiu

Hello,

I am working at the European Project P2P-Next [www.p2p-next.org]. Within this project a new protocol similar with uTP is designed and developed, named swift. We want to make a comparison test between classical BitTorrent? transfer over TCP, uTP transfer and swift transfer. We are planning to test TCP and uTP transfer with a Transmission client.

Our experiments could also be useful to you by testing transmission uTP facility and discovering bugs. We are going to use a cluster from University Politehnica of Bucharest along with a virtualization solution that will provide the possibility of running a big swarm of transmission clients simultaneously.

I installed two transmission 2.30b2 clients on two machines that are part of University Politehnica of Bucharest. Each one has an external IP address and are part of the same network. One of them is seeding a 64MiB file and the other is trying to download it using its torrent file. Both transmission clients are configured to use uTP (utp-enabled property from settings.json is set to true).

The problem is that the leecher cannot download from the seeder. I also tried to add into the swarm a KTorrent Linux client, configured to use only uTP. It manages to download a small amount of the file, but after a while it stops and the peers from its peer list disappear. Transmission clients also manage to transfer some percents of the file while KTorrent runs, but they also stop after a while. I also tried to use uTorrent from Windows, but the seeder is soon marked as snobbed because it is not able to deliver more than a few bytes from the file. Most of the time the download speed is 0, but from time to time it increses to about 1KiB/s.

If I deactivate uTP from the seeder by setting utp-enabled to false in settings.json, the transfer works perfectly at maximum speed.

What is the problem?

I would also like to know if utp-enabled property sets transmission to exclusively use uTP or it also can use TCP if it figures out that the client does not support uTP?

comment:109 in reply to: ↑ 108 Changed 5 years ago by Vincent

Replying to calin.burloiu:

I installed two transmission 2.30b2 clients on two machines that are part of University Politehnica of Bucharest. Each one has an external IP address and are part of the same network. One of them is seeding a 64MiB file and the other is trying to download it using its torrent file. Both transmission clients are configured to use uTP (utp-enabled property from settings.json is set to true).

The problem is that the leecher cannot download from the seeder. I also tried to add into the swarm a KTorrent Linux client, configured to use only uTP. It manages to download a small amount of the file, but after a while it stops and the peers from its peer list disappear. Transmission clients also manage to transfer some percents of the file while KTorrent runs, but they also stop after a while. I also tried to use uTorrent from Windows, but the seeder is soon marked as snobbed because it is not able to deliver more than a few bytes from the file. Most of the time the download speed is 0, but from time to time it increses to about 1KiB/s.

If I deactivate uTP from the seeder by setting utp-enabled to false in settings.json, the transfer works perfectly at maximum speed.

What is the problem?

There is a bug in libtransmission/bandwidth.c in function phaseOne:

223	    n = peerCount;
224	    dbgmsg( "%d peers to go round-robin for %s", n, (dir==TR_UP?"upload":"download") );
225	    i = n ? tr_cryptoWeakRandInt( n ) : 0; /* pick a random starting point */
226	    while( n > 1 )

write to peer here

so if there is only one peer we will never write data to UTP socket (cause as I understand it, libtransmission doesn't write to utp socket at second phase).

Also, if there are even 2 or more utp peers, the upload speed will be low cause in while cycle it writes only 512 bytes per iteration:

228 const size_t increment = 512;1024;

Jordan, could you please fix this issue or explain if I'm wrong?

Last edited 5 years ago by livings124 (previous) (diff)

comment:110 follow-up: Changed 5 years ago by jordan

Vincent, thank you for reviewing the code and reporting that issue. line 226 looks like it should be while( n > 0 ). Is that what you're referring to?

Also, 1024 probably is a better value for the uTP peers in this loop.

I've made these changes in r12361.

comment:111 in reply to: ↑ 110 ; follow-up: Changed 5 years ago by Vincent

Replying to jordan:

Vincent, thank you for reviewing the code and reporting that issue. line 226 looks like it should be while( n > 0 ). Is that what you're referring to?

Also, 1024 probably is a better value for the uTP peers in this loop.

I've made these changes in r12361.

jordan, I think 1024 bytes is to small value. Cause these 1024 bytes are sent every BANDWIDTH_PERIOD_MSEC, which is 500 ms by default. As a result we get too low upload speed. In practice I get about 20 mbit/s UTP upload speed with value of 1024000 bytes. But I guess it's not best solution. I suppose you need optimize this chunk of code regarding UTP behaviour.

comment:112 Changed 5 years ago by jordan

Well, one, we're in freeze for 2.30 right now so unless the current code is broken for the average user, I'm hesitant to refactor a critical piece of code like this right now. 20 Mbit/s is not something that the average user has to worry about :)

Two, that 1024 value is used inside a loop, so right now we call UTP_Write() multiple times with small buffer sizes. After looking over UTP_Write(), it's not clear to me how that approach is much worse than what your proposal of using a larger buffer size in fewer calls.

The downside to the "big buffer" approach is that a single peer would get the lion's share of the client's upload bandwidth.

comment:113 Changed 5 years ago by calin.burloiu

I corrected the bug marked in r12361 and then I tested again. Now the transfer from one seeder to one leecher works and also with two seeders.

When using transmission with uTP disabled in a swarm with two peers the TCP transfer occurs at an average of 4MiB/s between seeder and leecher. But when I restart the test with uTP enabled the average speed is about 500KiB/s which is very small. The two machines that host each peer are in the same network and do not have bandwidth limitation.

I also made a test with two transmission seeders using uTP placed on the same network and two other clients from a different network. I first tested with KTorrent in Ubuntu and then uTorrent in Windows. KTorrent could not download the whole file from the seeders. After a while it stops and all the peers disappear from its list passing into state "stalled". It may be an issue of KTorrent, which is the first Linux client to implement uTP as I know. On the other hand uTorrent has the best transfer. It reaches a download speed of 1.3MiB/s from each seeder. However, this speed is still smaller than the TCP one witch was about 4MiB/s with just one seeder.

Could I do something to optimize transfer speed? Or transmission with uTP is not yet optimized on this part?

comment:114 Changed 5 years ago by jordan

calin.burloiu: Transmission's use of uTP is of course pretty new, so yes it's possible that there are ways to improve the way we use it. Have you tried the "big buffer" suggestion Vincent made in comment:111? I'd be happy to be proven wrong in exchange for faster peers... :)

Changed 5 years ago by er13

comment:115 Changed 5 years ago by er13

@devs: Could you please add utp & no-utp options to transmission-daemon (s. attached patch). Thanks!

comment:116 Changed 5 years ago by jordan

er13: added in 12377. Thanks for the patch. :)

comment:117 Changed 5 years ago by er13

Nor sure about the exact values to be used, but shouldn't the socket receive and send buffers be smaller in case of "low-resource systems" (i.e. transmission is configured with --enable-lightweight).

comment:118 Changed 5 years ago by jordan

No, I don't think so -- memory and hard drive constraints don't translate to bandwidth constraints.

Moreover, sending the data out faster should actually cause the send buffers to be smaller, rather than larger...

comment:119 Changed 5 years ago by jch

Please note that we use a single UDP socket for all our µTP traffic -- so you only pay the send and receive buffers once.

No major objection to reducing the send buffer -- it's probably not used much. The receive buffer, on the other hand, is all we have in order to buffer µTP packets that arrive while we're context-switched, and on slow machines it's essential.

I'd be glad to see benchmarks (and I'm sure so would ghazel).

-- jch

comment:120 in reply to: ↑ 111 Changed 5 years ago by jch

Replying to Vincent:

Replying to jordan:

jordan, I think 1024 bytes is to small value. Cause these 1024 bytes are sent every BANDWIDTH_PERIOD_MSEC, which is 500 ms by default. As a result we get too low upload speed.

Jordan, he's right. Sending such a small amount will cause Nagle to trigger.

I'd suggest 3000 bytes, which will send one full-size frame straight away, and leave enough buffered data for the next frame to go out in a timely manner.

-- jch

comment:121 Changed 5 years ago by jordan

Objective testing of this doesn't seem to be possible, but firsthand observation seems seems to bear out the rationale -- increasing the buffer size to 3000 seems to make uTP peers "stutter" less.

r12415.

comment:122 Changed 5 years ago by gunzip

i'm also seeing a noticeable improvement with r12415.

in past, especially in seeding state, transmission seemed to discriminate against those peers with utp (T flag) and i noticed a strong tendency to upload to non-utp peers in the swarm. this behavior was repeatable and persistent over many torrents so don't think it was by chance. i often disabled utp because it seemed to inhibit seeding performance.

now, after testing with 4 torrents, uploading appears much more "balanced", i.e., seeding is occurring with both utp and non-utp peers.

comment:123 Changed 5 years ago by jordan

  • Owner changed from charles to jordan
  • Status changed from new to assigned

I'm going to tentatively mark this ticket as fixed.... things seem to be working well.

If there's something I've overlooked that needs to be addressed, please reopen the ticket with info.

comment:124 follow-up: Changed 5 years ago by jordan

  • Resolution set to fixed
  • Status changed from assigned to closed

comment:125 in reply to: ↑ 124 Changed 5 years ago by rafi

Replying to jordan: My initial interoperability tests show that Transmission 2.31 seems to not transfer ANY data to/from uTorrent in uTP even if uTorrent is set to use only uTP.

I suggest to re-test against uTorrent.

http://img829.imageshack.us/img829/5056/36874660.png

Last edited 5 years ago by rafi (previous) (diff)
Note: See TracTickets for help on using tickets.