Opened 8 years ago

Last modified 8 years ago

#5306 reopened Bug

Open File Limit - Unable to Change with CFLAGS - FD_SETSIZE

Reported by: helloadam Owned by: jordan
Priority: Normal Milestone: None Set
Component: libtransmission Version: 2.50+
Severity: Normal Keywords: fd_setsize, libcurl, open file limit
Cc: cfpp2p@…

Description

Looking around trac and IRC, ticket #4164 has come to my attention as a possible solution to transmission max open files limit. Currently we are experiencing the following at transmission start up:

Changed open file limit from 1000000 to 1024 (fdlimit.c:541)

then sometime later,

Couldn't save temporary file "/data/apps/transmission/2.50/conf/resume/Torrent.Name.Here.f462feaa9cc554f5.resume.tmp.MpkxKv": Too many open files (bencode.c:1744)
Torrent.Name.Here Unable to save resume file: Too many open files (torrent.c:487)

Of course, this is a problem since files are not seeding. Taking the recommendations from ticket #4164 I did an out of source build and compiled curl/7.29 and transmission/2.50 with the following options, but still no luck.

Any ideas? Tips on how I can debug where this 1024 limit is coming from? I don't feel like doing strace but if I must, I must.

Thanks!

libevent/2.0.21
./configure --prefix=/data/apps/libevent/2.0.21
make -j 4
make install

curl/7.29
./configure --prefix=/data/apps/curl/7.29 CPPFLAGS="-DFD_SETSIZE=16384"
make -j 4
make install

transmission/2.50
export PKG_CONFIG_PATH="/data/apps/libevent/2.0.21/lib/pkgconfig:/data/apps/curl/7.29/lib/pkgconfig:$PKG_CONFIG_PATH"
export LD_LIBRARY_PATH="/data/apps/curl/7.29/lib:$LD_LIBRARY_PATH"
export PATH="/data/apps/curl/7.29/bin:$PATH"
./configure --prefix=/data/apps/transmission/2.50 CFLAGS="-DFD_SETSIZE=16384" LIBCURL_CFLAGS="-DFD_SETSIZE=16384"
make -j 4
make install

Change History (8)

comment:1 follow-up: Changed 8 years ago by x190

"where this 1024 limit is coming from'

libtransmission/fdlimit.c: ensureSessionFdInfoExists()

That was done because supposedly libcurl has problems if you go over 1024. I think the code is flawed as that also includes 'open files', so you need to keep 'peer limit' to <800.

Last edited 8 years ago by x190 (previous) (diff)

comment:2 in reply to: ↑ 1 Changed 8 years ago by helloadam

Replying to x190:

"where this 1024 limit is coming from'

libtransmission/fdlimit.c: ensureSessionFdInfoExists()

That was done because supposedly libcurl has problems if you go over 1024. I think the code is flawed as that also includes 'open files', so you need to keep 'peer limit' to <800.

libcurl is flawed, or transmission? What I am trying to achieve is a larger max open file limit. We run ~4,000 torrents and having a limit of 1024 is not optimal.

I did modify the following peer limit options to under 800, restart transmission, but Changed open file limit from 1000000 to 1024 (fdlimit.c:541) still appears.

    "peer-limit-global": 300, 
    "peer-limit-per-torrent": 120,

Looking at #5161 should I continue to keep lowering my peer limit? Does not seem like a valid solution.

Thanks for the help!

comment:3 Changed 8 years ago by x190

Quoting myself from #5161:

"Total of peer sockets + FILE_CACHE_SIZE(32) + all other open files for this process must be <= FD_SETSIZE (hopefully 1024)"

It is intentional that fdlimit.c:541 goes to 1024 as that is regarded as the safe upper limit for libcurl.

Whether you run 40 or 400000 torrents should not matter as 'FILE_CACHE_SIZE(32)' should ensure that no more than 32 files are open at any one time.

So the math is: 1024 - 32 (FILE_CACHE_SIZE) - ~200 for process files - ~800 (peers global) = 0.

EDIT: "peer-limit-global": 300, should be fine. Are you still getting "Too many open files (torrent.c:487)" with this setting?

Last edited 8 years ago by x190 (previous) (diff)

comment:4 Changed 8 years ago by jordan

  • Resolution set to duplicate
  • Status changed from new to closed

This cap at FD_SETSIZE comes from bug #4164

comment:5 Changed 8 years ago by cfpp2p

  • Cc cfpp2p@… added

comment:6 Changed 8 years ago by jordan

See also https://trac.transmissionbt.com/ticket/5265#comment:3 for some thoughts on possibly replacing libcurl in future versions of Transmission

comment:7 Changed 8 years ago by x190

Here's a pertinent quote from the summary of #4164 that should be addressed now as it would eliminate these "Too many open files" errors.

"Would be better to validate FD_SETSIZE on startup and do not allow open-file-limit + peer-limit-global to be greater than FD_SETSIZE."

'peer-limit-global' is allowed to be the full 1024 leaving 0 fds for 'open-file-limit' including process files. Is this not why these tickets continue to be filed? Replacing libcurl with something that can handle more sockets will not solve the problem until the 'open-file-limit' issue is addressed.

This regression was introduced in r12514.

Last edited 8 years ago by x190 (previous) (diff)

comment:8 Changed 8 years ago by x190

  • Resolution duplicate deleted
  • Status changed from closed to reopened
Note: See TracTickets for help on using tickets.