Opened 10 years ago
Last modified 10 years ago
#5306 reopened Bug
Open File Limit - Unable to Change with CFLAGS - FD_SETSIZE
Reported by: | helloadam | Owned by: | jordan |
---|---|---|---|
Priority: | Normal | Milestone: | None Set |
Component: | libtransmission | Version: | 2.50+ |
Severity: | Normal | Keywords: | fd_setsize, libcurl, open file limit |
Cc: | cfpp2p@… |
Description
Looking around trac and IRC, ticket #4164 has come to my attention as a possible solution to transmission max open files limit. Currently we are experiencing the following at transmission start up:
Changed open file limit from 1000000 to 1024 (fdlimit.c:541)
then sometime later,
Couldn't save temporary file "/data/apps/transmission/2.50/conf/resume/Torrent.Name.Here.f462feaa9cc554f5.resume.tmp.MpkxKv": Too many open files (bencode.c:1744) Torrent.Name.Here Unable to save resume file: Too many open files (torrent.c:487)
Of course, this is a problem since files are not seeding. Taking the recommendations from ticket #4164 I did an out of source build and compiled curl/7.29 and transmission/2.50 with the following options, but still no luck.
Any ideas? Tips on how I can debug where this 1024 limit is coming from? I don't feel like doing strace but if I must, I must.
Thanks!
libevent/2.0.21 ./configure --prefix=/data/apps/libevent/2.0.21 make -j 4 make install curl/7.29 ./configure --prefix=/data/apps/curl/7.29 CPPFLAGS="-DFD_SETSIZE=16384" make -j 4 make install transmission/2.50 export PKG_CONFIG_PATH="/data/apps/libevent/2.0.21/lib/pkgconfig:/data/apps/curl/7.29/lib/pkgconfig:$PKG_CONFIG_PATH" export LD_LIBRARY_PATH="/data/apps/curl/7.29/lib:$LD_LIBRARY_PATH" export PATH="/data/apps/curl/7.29/bin:$PATH" ./configure --prefix=/data/apps/transmission/2.50 CFLAGS="-DFD_SETSIZE=16384" LIBCURL_CFLAGS="-DFD_SETSIZE=16384" make -j 4 make install
Change History (8)
comment:1 follow-up: ↓ 2 Changed 10 years ago by x190
comment:2 in reply to: ↑ 1 Changed 10 years ago by helloadam
Replying to x190:
"where this 1024 limit is coming from'
libtransmission/fdlimit.c: ensureSessionFdInfoExists()
That was done because supposedly libcurl has problems if you go over 1024. I think the code is flawed as that also includes 'open files', so you need to keep 'peer limit' to <800.
libcurl is flawed, or transmission? What I am trying to achieve is a larger max open file limit. We run ~4,000 torrents and having a limit of 1024 is not optimal.
I did modify the following peer limit options to under 800, restart transmission, but Changed open file limit from 1000000 to 1024 (fdlimit.c:541) still appears.
"peer-limit-global": 300, "peer-limit-per-torrent": 120,
Looking at #5161 should I continue to keep lowering my peer limit? Does not seem like a valid solution.
Thanks for the help!
comment:3 Changed 10 years ago by x190
Quoting myself from #5161:
"Total of peer sockets + FILE_CACHE_SIZE(32) + all other open files for this process must be <= FD_SETSIZE (hopefully 1024)"
It is intentional that fdlimit.c:541 goes to 1024 as that is regarded as the safe upper limit for libcurl.
Whether you run 40 or 400000 torrents should not matter as 'FILE_CACHE_SIZE(32)' should ensure that no more than 32 files are open at any one time.
So the math is: 1024 - 32 (FILE_CACHE_SIZE) - ~200 for process files - ~800 (peers global) = 0.
EDIT: "peer-limit-global": 300, should be fine. Are you still getting "Too many open files (torrent.c:487)" with this setting?
comment:4 Changed 10 years ago by jordan
- Resolution set to duplicate
- Status changed from new to closed
This cap at FD_SETSIZE comes from bug #4164
comment:5 Changed 10 years ago by cfpp2p
- Cc cfpp2p@… added
comment:6 Changed 10 years ago by jordan
See also https://trac.transmissionbt.com/ticket/5265#comment:3 for some thoughts on possibly replacing libcurl in future versions of Transmission
comment:7 Changed 10 years ago by x190
Here's a pertinent quote from the summary of #4164 that should be addressed now as it would eliminate these "Too many open files" errors.
"Would be better to validate FD_SETSIZE on startup and do not allow open-file-limit + peer-limit-global to be greater than FD_SETSIZE."
'peer-limit-global' is allowed to be the full 1024 leaving 0 fds for 'open-file-limit' including process files. Is this not why these tickets continue to be filed? Replacing libcurl with something that can handle more sockets will not solve the problem until the 'open-file-limit' issue is addressed.
This regression was introduced in r12514.
comment:8 Changed 10 years ago by x190
- Resolution duplicate deleted
- Status changed from closed to reopened
"where this 1024 limit is coming from'
libtransmission/fdlimit.c: ensureSessionFdInfoExists()
That was done because supposedly libcurl has problems if you go over 1024. I think the code is flawed as that also includes 'open files', so you need to keep 'peer limit' to <800.