Opened 12 years ago
Closed 12 years ago
#4221 closed Bug (invalid)
Scrape and announce socket file descriptors are not closed for some trackers
Reported by: | x190 | Owned by: | jordan |
---|---|---|---|
Priority: | Normal | Milestone: | None Set |
Component: | libtransmission | Version: | 2.22+ |
Severity: | Normal | Keywords: | |
Cc: |
Description
Scrapes and announces to some trackers spawn a secondary server http connection which Transmission subsequently neglects to properly close.
Netstat shows the connection remaining in the CLOSE_WAIT state for the balance of the session. This unnecessarily ties up sockets and also data could still be sent.
tcp4 0 0 192.168.x.x.50642 rs19210.rapidspe.menan* CLOSE_WAIT tcp4 0 0 192.168.x.x.50641 1337x.org.http CLOSE_WAIT *may also be rs19210.rapidspe.http After a short time 1337x.org.http is gone but rapidspe remains for duration of session. tcp4 0 0 192.168.x.x.50642 rs19210.rapidspe.menan CLOSE_WAIT
Tested with genesis.1337x.org:1337/announce
http://www.sunmanagers.org/pipermail/summaries/2006-January/007068.html https://forum.transmissionbt.com/viewtopic.php?f=4&t=11642&p=53949#p53949
Change History (19)
comment:1 follow-up: ↓ 2 Changed 12 years ago by jordan
- Resolution set to invalid
- Status changed from new to closed
comment:2 in reply to: ↑ 1 ; follow-up: ↓ 3 Changed 12 years ago by x190
Replying to jordan:
The http scrape/announce sockets are managed by libcurl, not libtransmission. Please report this upstream to libcurl. Thanks!
A rather unsatisfying response. Were you able to replicate the bug? If so, why don't you report it, as you would be able to make a much more meaningful report than I could hope to?
According to everything I've read, it is still the application's responsibility to ensure that sockets are properly dismantled. Also, an aspect of this report ties in with your weeks long discussion with Astara in that the socket in question appears to be re-used or at least not replicated during the session.
comment:3 in reply to: ↑ 2 Changed 12 years ago by jordan
Replying to x190:
According to everything I've read, it is still the application's responsibility to ensure that sockets are properly dismantled.
Please cite the libcurl documentation that says that?
comment:4 Changed 12 years ago by x190
- Resolution invalid deleted
- Status changed from closed to reopened
comment:5 Changed 12 years ago by x190
Thanks for the curl links! Can we deal with the problem by calling curl_easy_cleanup() as suggested in your forum curl links? Please note that even if the tracker is removed from the list then the connection is still not closed.
I'm a lite torrenter but for the heavies this could translate into a lot of wasted sockets.
comment:6 follow-up: ↓ 7 Changed 12 years ago by jordan
I don't understand what you're requesting be done in libtransmission (as opposed to in libcurl).
It looks like you're suggesting that libtransmission call curl_easy_cleanup(), but it already does.
comment:7 in reply to: ↑ 6 Changed 12 years ago by x190
Replying to jordan:
I don't understand what you're requesting be done in libtransmission (as opposed to in libcurl).
It looks like you're suggesting that libtransmission call curl_easy_cleanup(), but it already does.
Gave this one my best effort: :)
• tracker redirects result in 2 connections when only one was initially requested • both these connections belong to the Transmission process (lsof IPv4) • appears that when curl_easy_cleanup() is called and gets passed the parameter e, which I'm guessing is a list of connection IP addresses to cleanup, that secondary connection IP address is missing. • can a bit of code be designed for that section of web.c around Line 380-400 that will ensure that when 'e' is passed to curl_easy_cleanup() that it includes all the IP addresses associated with sockets that were setup as result of a scrape or announce including the ones spawned by a redirect? • can a test be designed to check to see if curl_easy_cleanup() was fully successful and if not, call the appropriate close() function on the missed sockets (connections)?
comment:8 Changed 12 years ago by jordan
How do I do that with the libcurl API?
comment:9 follow-up: ↓ 10 Changed 12 years ago by jordan
An addendum to that last question: I'm not trying to be difficult. I just don't see how this is something that can be done in libtransmission...
...or even something that *should* be done in libtransmission. if libcurl is leaking sockets, then it should be reported to libcurl.
comment:10 in reply to: ↑ 9 Changed 12 years ago by x190
Replying to jordan:
if libcurl is leaking sockets, then it should be reported to libcurl.
Only you could do that effectively, as you could explain your code, it's functions and values.
comment:11 Changed 12 years ago by x190
Just found this:
http://curl.haxx.se/mail/tracker-2005-12/0024.html
This seems interesting as I'm pretty sure the persistent CLOSE_WAIT connection does get reused.
"Comment By: Daniel Stenberg (badger) Date: 2005-12-09 09:31
Message: Logged In: YES user_id=1110
"Explicit close when perform() is done is forced with CURLOPT_FORBID_REUSE. "
Try and patch that in the code somewhere. :)
comment:12 Changed 12 years ago by x190
http://php.net/manual/en/function.curl-setopt.php
"CURLOPT_FORBID_REUSE TRUE to force the connection to explicitly close when it has finished processing, and not be pooled for reuse. "
http://curl.haxx.se/mail/lib-2011-03/0315.html
"set CURLOPT_FORBID_REUSE for the connections that need to be closed as soon as possible after use. "
comment:13 Changed 12 years ago by jordan
Passing this upstream for another pair of eyes:
https://sourceforge.net/tracker/?func=detail&aid=3297118&group_id=976&atid=100976
comment:14 Changed 12 years ago by x190
Guess what... Curiosity got the best of me, so I kept at 'till I found how to set CURLOPT_FORBID_REUSE and it certainly works like a charm!
static CURL * createEasy( tr_session * s, struct tr_web_task * task ) { const tr_address * addr; bool is_default_value; CURL * e = curl_easy_init( ); const long verbose = getenv( "TR_CURL_VERBOSE" ) != NULL; char * cookie_filename = tr_buildPath( s->configDir, "cookies.txt", NULL ); task->timeout_secs = getTimeoutFromURL( task ); curl_easy_setopt( e, CURLOPT_AUTOREFERER, 1L ); curl_easy_setopt( e, CURLOPT_COOKIEFILE, cookie_filename ); curl_easy_setopt( e, CURLOPT_ENCODING, "gzip;q=1.0, deflate, identity" ); curl_easy_setopt( e, CURLOPT_FOLLOWLOCATION, 1L ); curl_easy_setopt( e, CURLOPT_FORBID_REUSE, true ); /*Line 167 web.c r12380*/ curl_easy_setopt( e, CURLOPT_MAXREDIRS, -1L ); curl_easy_setopt( e, CURLOPT_NOSIGNAL, 1L ); curl_easy_setopt( e, CURLOPT_PRIVATE, task );
Netstat listing is as clean as a whistle. :) While testing this, I uncovered a couple of items in announcer-udp.c that also result in a CLOSE_WAIT forever connection, which again is resolved using CURLOPT_FORBID_REUSE.
- udp announce results in "Connection failed". or
- udp announce results in retry interval being used.
In both of these cases, the connection is left in CLOSE_WAIT state until session end even if the announce later is successful.
Adding curl_easy_setopt( e, CURLOPT_FORBID_REUSE, true ); /*Line 167 web.c r12380*/ cleans out all CLOSE_WAIT states in both situations reported (tracker redirect and tracker retries).
Tested on r12380 and r11889(#include <stdbool.h>). SL 10.6.7 libcurl 7.19.7
comment:15 follow-up: ↓ 16 Changed 12 years ago by jordan
Meh. Yes CURLOPT_FORBID_REUSE has its advantages but it also prevents reusing connections when you're going to have a lot of scrapes + announces to a single tracker. It might be useful as a last resort but it seems like a pretty blunt instrument for this issue.
As for fixing announce-udp... that code doesn't use libcurl. Was your router so full of CLOSE_WAIT that no network connections of any kind were possible?
comment:16 in reply to: ↑ 15 Changed 12 years ago by x190
Replying to jordan:
Meh.
Wasn't sure how to take this but after some reflection it brought back good memories of Clyde the camel and Pepe Le Peeuw the goat (I-15, Glendale, South NV), to which you may reply, MEH. :)
Yes CURLOPT_FORBID_REUSE has its advantages but it also prevents reusing connections when you're going to have a lot of scrapes + announces to a single tracker. It might be useful as a last resort but it seems like a pretty blunt instrument for this issue.
As for fixing announce-udp... that code doesn't use libcurl. Was your router so full of CLOSE_WAIT that no network connections of any kind were possible?
It seems strange to me that libcurl doesn't care about cleaning these fds and sockets up. Their attitude seems to be that it doesn't matter and if you have a customer who complains then you would need to use CURLOPT_FORBID_REUSE. From what I can gather, CLOSE_WAITs can't be re-used anyway and are considered 'dead' sockets (see badger's reply to your ticket).
In my case, it's highly unlikely I will ever hit the wall because of this (and anyway, I now know the solution :) ) but I could see it perhaps tipping the scales for those who run many hundreds or even thousands of torrents.
Meh, right?
P.S. How do you do the burro bray? :)
comment:17 Changed 12 years ago by sharpshot_uk
"The CLOSE_WAIT sockets are probably the ones that libcurl has in its connection cache but that have been "closed" (a FIN was sent) by the server already but not yet by libcurl. They are not there "indefinitely" (and they really can't be) since the connection cache has a limited size so eventually the old connections should get closed. Also, for each newly created connection when searching if there's an existing one, libcurl is supposed to detect and close "dead" connections such as these. Possibly that detection is somehow not working on Mac OS X. Your code looks correct and quite frankly, I don't think you describe a bug but mere symptoms of how libcurl works."
By Badger at curl and libcurl on sourceforge in reply to Jordans ticket.
comment:18 Changed 12 years ago by x190
I am satisfied Transmission is doing all it can while using libcurl, short of using CURLOPT_FORBID_REUSE, which is considered overkill. This ticket may be closed as far as I'm concerned. Thanks for the clarification provided.
comment:19 Changed 12 years ago by jordan
- Resolution set to invalid
- Status changed from reopened to closed
Closing by x190's suggestion in comment:18
The http scrape/announce sockets are managed by libcurl, not libtransmission. Please report this upstream to libcurl. Thanks!
http://sourceforge.net/tracker/?group_id=976&atid=100976