Open File Limit - Unable to Change with CFLAGS - FD_SETSIZE

[helloadam]

Looking around trac and IRC, ticket #4164 has come to my attention as a possible solution to transmission max open files limit. Currently we are experiencing the following at transmission start up:

Changed open file limit from 1000000 to 1024 (fdlimit.c:541)

then sometime later,

Couldn't save temporary file "/data/apps/transmission/2.50/conf/resume/Torrent.Name.Here.f462feaa9cc554f5.resume.tmp.MpkxKv": Too many open files (bencode.c:1744)
Torrent.Name.Here Unable to save resume file: Too many open files (torrent.c:487)

Of course, this is a problem since files are not seeding. Taking the recommendations from ticket #4164 I did an out of source build and compiled curl/7.29 and transmission/2.50 with the following options, but still no luck.

Any ideas? Tips on how I can debug where this 1024 limit is coming from? I don't feel like doing strace but if I must, I must.

Thanks!

libevent/2.0.21
./configure --prefix=/data/apps/libevent/2.0.21
make -j 4
make install

curl/7.29
./configure --prefix=/data/apps/curl/7.29 CPPFLAGS="-DFD_SETSIZE=16384"
make -j 4
make install

transmission/2.50
export PKG_CONFIG_PATH="/data/apps/libevent/2.0.21/lib/pkgconfig:/data/apps/curl/7.29/lib/pkgconfig:$PKG_CONFIG_PATH"
export LD_LIBRARY_PATH="/data/apps/curl/7.29/lib:$LD_LIBRARY_PATH"
export PATH="/data/apps/curl/7.29/bin:$PATH"
./configure --prefix=/data/apps/transmission/2.50 CFLAGS="-DFD_SETSIZE=16384" LIBCURL_CFLAGS="-DFD_SETSIZE=16384"
make -j 4
make install

[x190]

"where this 1024 limit is coming from'

libtransmission/fdlimit.c: ensureSessionFdInfoExists()

That was done because supposedly libcurl has problems if you go over 1024. I think the code is flawed as that also includes 'open files', so you need to keep 'peer limit' to <800.

[helloadam]

Replying to x190:

"where this 1024 limit is coming from'

libtransmission/fdlimit.c: ensureSessionFdInfoExists()

That was done because supposedly libcurl has problems if you go over 1024. I think the code is flawed as that also includes 'open files', so you need to keep 'peer limit' to <800.

libcurl is flawed, or transmission? What I am trying to achieve is a larger max open file limit. We run ~4,000 torrents and having a limit of 1024 is not optimal.

I did modify the following peer limit options to under 800, restart transmission, but Changed open file limit from 1000000 to 1024 (fdlimit.c:541) still appears.

    "peer-limit-global": 300, 
    "peer-limit-per-torrent": 120,

Looking at #5161 should I continue to keep lowering my peer limit? Does not seem like a valid solution.

Thanks for the help!

[x190]

Quoting myself from #5161:

"Total of peer sockets + FILE_CACHE_SIZE(32) + all other open files for this process must be <= FD_SETSIZE (hopefully 1024)"

It is intentional that fdlimit.c:541 goes to 1024 as that is regarded as the safe upper limit for libcurl.

Whether you run 40 or 400000 torrents should not matter as 'FILE_CACHE_SIZE(32)' should ensure that no more than 32 files are open at any one time.

So the math is: 1024 - 32 (FILE_CACHE_SIZE) - ~200 for process files - ~800 (peers global) = 0.

EDIT: "peer-limit-global": 300, should be fine. Are you still getting "Too many open files (torrent.c:487)" with this setting?

[jordan]

This cap at FD_SETSIZE comes from bug #4164

[jordan]

See also https://trac.transmissionbt.com/ticket/5265#comment:3 for some thoughts on possibly replacing libcurl in future versions of Transmission

[x190]

Here's a pertinent quote from the summary of #4164 that should be addressed now as it would eliminate these "Too many open files" errors.

"Would be better to validate FD_SETSIZE on startup and do not allow open-file-limit + peer-limit-global to be greater than FD_SETSIZE."

'peer-limit-global' is allowed to be the full 1024 leaving 0 fds for 'open-file-limit' including process files. Is this not why these tickets continue to be filed? Replacing libcurl with something that can handle more sockets will not solve the problem until the 'open-file-limit' issue is addressed.

This regression was introduced in r12514.