Opened 12 years ago

Last modified 5 years ago

#1753 assigned Enhancement

worker disk IO thread to ease the load in the libtransmission/libevent thread

Reported by: turbo Owned by: charles
Priority: Normal Milestone: Sometime
Component: libtransmission Version: 1.42
Severity: Normal Keywords: heavy work locking move
Cc: leroy.bodacious@…, ajf88, LB06, tiennou7@…, bollywood, kip@…, keizie@…, gvdl@…, thomas_reardon@…, mannionpatrick@…, krisfulgham@…, roberth.sjonoy@…, decathected@…, antivirtel@…

Description

Some work is very heavy and currently the 'whole shebang' locks (hangs) for a unspecified period of time.

There need to be a 'worker thread', that does the 'heavy lifting'.

Change History (77)

comment:1 Changed 12 years ago by turbo

This is the 'solution' to permanently fix #554, #920, #1521, #651. This worker thread is mentioned in #554.

But dispatching work to TheWorker? must be done somehow. I've previously worked with Semaphores and the're a mess! It's very difficult to get stable.

Is there any other way to implement intercomunication between threads?

comment:2 Changed 12 years ago by charles

Yes, see how trevent.c does it

comment:3 Changed 12 years ago by charles

  • Summary changed from Worker thread to ease the load on main to worker disk IO thread to ease the load in the libtransmission/libevent thread
  • Type changed from Bug to Enhancement
  • Version changed from 1.42+ to 1.42

probably the worker thread would need to be given a fifo queue of tasks to perform, and then notify libtransmission when the work was done somehow.

All the code in libtransmission that assumes local data can be read and write immediately would need to be changed to a state machine that would wait for tasks to finish.

comment:4 Changed 12 years ago by j3r3miah

  • Cc leroy.bodacious@… added

comment:5 Changed 11 years ago by charles

tr_torrentSetLocation() should also be changed when we get nonblocking IO in libtransmission.

  • we'd need TR_STATUS_MOVE_WAIT and TR_STATUS_MOVE to tr_torrent_activity
  • GUIs would need to disable start, verify, and move when these statuses were true
  • deleting the torrent would need to abort the move
  • probably other special cases I haven't thought of yet :/

comment:6 Changed 11 years ago by charles

tr_torrentSetLocation() should also be changed when we get nonblocking IO in libtransmission.

  • we'd need TR_STATUS_MOVE_WAIT and TR_STATUS_MOVE to tr_torrent_activity
  • GUIs would need to disable start, verify, and move when these statuses were true
  • deleting the torrent would need to abort the move
  • probably other special cases I haven't thought of yet :/

comment:7 Changed 11 years ago by ajf88

  • Cc ajf88 added

comment:8 Changed 11 years ago by jch

In the short term, just prefetching the data whenever we receive a chunk request (posix_fadvise) should improve the situation quite a bit.

comment:9 Changed 11 years ago by charles

because incoming block requests are near-random, there doesn't seem to be a good way of knowing when prefetching would make things better, or worse.

If we have multiple requests queued up, we could probably look across all of them to figure out what blocks to prefetch...

comment:10 Changed 11 years ago by charles

Ticket #2205 has been marked as a duplicate of this ticket.

comment:11 Changed 11 years ago by charles

Ticket #2235 has been marked as a duplicate of this ticket.

comment:12 Changed 11 years ago by LB06

  • Cc LB06 added

comment:13 Changed 11 years ago by charles

Ticket #2289 has been marked as a duplicate of this ticket.

comment:14 Changed 11 years ago by charles

Ticket #554 has been marked as a duplicate of this ticket.

comment:15 Changed 11 years ago by charles

  • Milestone changed from None Set to Sometime
  • Status changed from new to assigned

comment:16 Changed 11 years ago by haravikk

Just wanted to add my voice to encourage this one along; I've currently got four big torrents (around 20gb), and 20 smaller ones (500mb to 1.5gb). All are active, with only around six are ever doing anything at one time, Transmission's GUI is almost completely unresponsive. It's still downloading quite happily, but there's no visible sign other than my disk activity light.

This seems especially bad when a torrent is checking existing data for whatever reason.

Regarding prefetching; I'd say there isn't much point as already mentioned due to the inherent randomness, it would only really benefit torrents where some blocks are in particularly high demand. The machines that would benefit most from using RAM properly to cache blocks are dedicated torrent servers, where you can usually configure a big chunk of RAM to speed up disk-access anyway.

So I think a thread is the best solution for most users.

comment:17 Changed 11 years ago by jb510

  • Keywords move added
  • Version 1.42 deleted

I was about to file an enhancement request of my own when I found these.... both #2712 and #554 have been sited as duplicates of #1753. I just want to be clear that I think REGARDLESS of the threading issues discussed here there should be some graphical feedback to the user as to the progress of the move. This doesn't seem to me directly related to the discussion here, but others seem to mark it as duplicate so I won't argue.

In other words regardless of whether the move to IO hangs transmission or slowly makes progress in the background there still needs to be some kind of feedback to the user demonstrating move progress.

comment:18 Changed 11 years ago by livings124

  • Version set to 1.42

comment:19 Changed 11 years ago by jch

I have a branch of Transmission that includes an extremely efficient thread pool. I'm waiting for Charles to branch 1.80 before submitting it.

comment:20 Changed 11 years ago by tiennou

  • Cc tiennou7@… added

I chimed on IRC earlier, but I can't stay around much longer. I have fixed the locking in the OS X GUI here : http://github.com/tiennou/transmission/tree/async-move

Feel free to comment/review, or tell me I should have waited for the thread pool changes to be pulled in ;-).

comment:21 Changed 11 years ago by bollywood

  • Cc bollywood added

great to see a couple of solutions for this problem. looking forward to seeing them implemented in an official build. thanks guys!

comment:22 Changed 10 years ago by charles

  • Milestone changed from Sometime to 2.10

comment:23 Changed 10 years ago by charles

  • Milestone changed from 2.10 to Sometime

Bah.

Bumping to post-2.10 because most of the rest of 2.10 is ready; this isn't, and it's a large item that would hold up the rest.

comment:24 Changed 10 years ago by charles

Ticket #3655 has been closed as a duplicate of this ticket.

comment:25 follow-up: Changed 10 years ago by jusid

Just a note to keep in mind: Preallocation should be done in a separate thread and progress should be shown to a user when a full preallocation (zero fill) is performed. Currently the full preallocation just blocks all operations for very long time (it can take 10-20 minutes for large files on a slow NMT device)...

comment:26 Changed 10 years ago by charles

Ticket #3533 has been closed as a duplicate of this ticket.

comment:27 in reply to: ↑ 25 Changed 10 years ago by x190

Replying to jusid:

Just a note to keep in mind: Preallocation should be done in a separate thread and progress should be shown to a user when a full preallocation (zero fill) is performed. Currently the full preallocation just blocks all operations for very long time (it can take 10-20 minutes for large files on a slow NMT device)...

Agreed. I think this might either eliminate or at least give the user a visual reason for some lengthy hangs on quit.

comment:28 Changed 10 years ago by Astara

Just a thought, but you might want to consider a 'pool of worker threads' that will auto-adjust up or down depending on the user's system needs.

Slower systems might need more threads to stay responsive.

OS's (linux, windows) use worker threads (aka soft-interrupts in some contexts) for things that take too long to do 'synchronously' and they are usually dynamically allocated as needed.

comment:29 Changed 10 years ago by kip

  • Cc kip@… added

comment:31 Changed 10 years ago by keizie

  • Cc keizie@… added

comment:32 Changed 9 years ago by jordan

  • Priority changed from High to Normal
  • Severity changed from Major to Normal

comment:33 Changed 9 years ago by x190

This issue as regards pause/quit of newly added large torrents (more than 1GB) is really quite major on a Mac (preallocation issue) and folks would really like to see something done about it. Please see Mark Lee's thread at Mac Update.

https://www.macupdate.com/app/mac/19378/transmission

comment:34 Changed 9 years ago by taligent

  • Severity changed from Normal to Major

This need to be raised in priority.

It is a MAJOR issue on OSX where stopping even a small torrent (400MB) causes the entire UI to just lock up for minutes. It is significantly more important than all of the other silly things (e.g. animations) that have been added to the OSX client.

comment:35 Changed 9 years ago by livings124

  • Severity changed from Major to Normal

comment:36 Changed 9 years ago by livings124

There are different developers for this and the UI, and the priority for this functionality has not changed.

comment:37 Changed 9 years ago by paulmillr

I'm really sick of the issue. From the first day I've used OSX / transmission, it follows me and all transmission users.

It's been three years and developers still don't care about it. Meh.

comment:38 Changed 9 years ago by livings124

paulmillr: This is an open source project. We gladly accept patches.

comment:39 Changed 9 years ago by taligent

livings124:

No offense. But that is ridiculous and you know it.

This is such a core change that it needs to be done by people with experience across the entire codebase. Unless of course you will "gladly accept" something I hack together in a couple of nights over a few beers ?

comment:40 Changed 9 years ago by livings124

Changing the severity of this ticket is like giving us marching orders. If you're not willing to contribute, then please accept that this will be done when it's done. It's clearly a ton of work and the developers have time constraints and different priorities.

comment:41 follow-up: Changed 9 years ago by taligent

livings124:

I've worked on open source projects and it was never intended to tell developers what to do. And I know there is rarely an appetite to make major changes. The issue is just pretty severe on OSX.

Now this enhancement is for a generic solution for handling any long running tasks.

Is it at all possible to make just the preallocation part threaded as an interim measure ?

comment:42 Changed 9 years ago by livings124

Jordan would know more of that, but it's likely a huge undertaking.

comment:43 in reply to: ↑ 41 Changed 9 years ago by x190

Replying to taligent:

livings124: Is it at all possible to make just the preallocation part threaded as an interim measure ?

See comment:19 and comment:28. Also, I recall Jordan mentioning that a worker thread is already being used for some specific task elsewhere in the code, so maybe this specific issue could be addressed without too much difficulty.

comment:44 Changed 8 years ago by collegeitdept

So will this VERY annoying, which some may say is CRITICAL (I as well) ever get fixed???

This bug was opened 3 YEARS AGO!!!!

I understand this is a major undertaken, but 3 YEARS????

comment:45 Changed 8 years ago by paulmillr

In this case I don't know any C/C++, so I cannot submit patches.

If I maintain an open-source project and someone submits this sort of bug, I fix it. For real. It doesn't seem to be boring routine work.

Is it only me who really cares about software he produces / open-sources?

comment:46 Changed 8 years ago by gvdl

  • Cc gvdl@… added

I'm looking into internal threading in libtransmission. It is going to be a big job and we need to get the first baby steps out there before we can take the big leap of full multi-threading the I/O systems. Right now I envisage that some simple queueing and setting up logging and torrent directory watching tasks #4907 will be the best way to start.

Last edited 8 years ago by gvdl (previous) (diff)

comment:47 Changed 8 years ago by reardon

  • Cc thomas_reardon@… added

comment:49 Changed 8 years ago by mannionp

  • Cc mannionpatrick@… added

FWIW, also seeing this issue with 10.8 and current build (knowing its not even slated to be fixed yet).

comment:50 Changed 8 years ago by jordan

See also #5168

comment:51 Changed 8 years ago by krs1

  • Cc krisfulgham@… added

This is a huge issue for me. Transmission is currently managing my Media Server and I have 3TB worth of data on external HDD. I recenty added another to the mix and when I move data files on more than 25+GB worth of data, transmission beach balls overnight.

At the least I would like a progress bar to know if the whole process froze, or the current status.

comment:52 Changed 8 years ago by x190

The behavior for pausing newly added torrents in the Mac Client has changed. You may now pause them without freezing the gui and the zeroing-out will be done in the background. However, if you try to re-start the torrent before that process is complete the old freeze will return. Same old delay on quit. Over all a small improvement. 2.76 +(14023) SL 10.6.8

Last edited 8 years ago by x190 (previous) (diff)

comment:53 Changed 8 years ago by collegeitdept

This issue will only get worse and worse as file size increases.

This really should (start at least) to get addressed.

comment:54 Changed 8 years ago by MechMK1

When working with torrents > 1 TB and/or > 5000 files, this becomes quite of a problem

comment:55 Changed 8 years ago by MechMK1

Please set at least a milestone and/or increase the severity of this bug. Transmission becomes practically unusable when working with big file sizes.

Is there any way to help getting this actually done?

comment:56 Changed 8 years ago by Astara

Is there any reason it has to be done with threading?

It seems, (if I understand the comments) that when moving a large file from one disk to another (assuming a 'rename' on the same disk isn't usually an issue on most OS's), that you need to do a file copy. Why not fork{child: system("cp -a curobject dest")} {parent:keep track of outstanding copying children; get sigchild-- read status; update status...blahblahblah)?

If it is hard to do it in the current threading system why limit yourself to threads? Or for that matter, if a system util is available to do it, why not fork off a copy and let it do the task, *for now*... until someone wants to duplicate the system functionality within libtransmission, and until someone wants to do it in a multithreaded version.

I.e. consider it a 'stopgap' measure to deal with the issue for now, and then include this issue in the overall forward design (you know, like when you fix code formatting and stuff ;-)).

Hey, just saying, sometimes not doing it yourself can save you headache in the short-term until you are ready to replace it?

Or am I missing some big picture (yeah, I know 'cp' isn't part of every platform, but perl can do an exact file copy on Windows and Mac, (but not Unix/Linux? -- for there, you'd need cp -a)...(doesn't that make lots of sense? ;-))...

I have a util that I need to have a file-duplicate operation in, until I get to writing my own, 'cp' will copy ACL's and extended file attrs if they are present (as well as the normal owner and date/time info)...

comment:57 Changed 8 years ago by MechMK1

Does that apply to writing whole files to disk (i.e. as part of the downloadinh process) or already pieces? Or is this solely about moving files arounf on disk (i.e. moving a torrent from /foo to /bar)?

comment:58 follow-up: Changed 7 years ago by thecoda

Threads and Semaphores are far too low level, it's no surprise that this still hasn't been tackled after 5 years!

You should be looking at an async library (there's one in Boost), or a message-passing or actor framework, or some form of "active objects". These are good places to start:

Admittedly... a lot of this is biased towards C++ (I see that the transmission codebase is mostly vanilla C), but there are definitely solutions in that lot with plain C interfaces available.

Some of this stuff is better explained at a high level in the Netty and Akka frameworks:

They're both for the JVM, but give give a good overview of the concepts involved.

comment:59 in reply to: ↑ 58 Changed 7 years ago by Astara

Replying to thecoda:

Threads and Semaphores are far too low level, it's no surprise that this still hasn't been tackled after 5 years!

You should be looking at an async library (there's one in Boost), or a message-passing or actor framework, or some form of "active objects". These are good places to start:

--- If it was being designed from scratch, maybe. But if it is going to fit in with the current event lib, it might be good to try for something more portable.

Does the eventlib handle events coming in from the POSIX aio calls?

Maybe the I/O could be done using aio_read/write which make calls back into libevent when done?

FWIW, this issue doesn't bother me, but I don't use an integrated GUI client.

I use the remote GUI talking to an instance of the daemon (I am running 2 on the same machine now, to allow for better priority control). It is rare for the remote GUI to hang due to IO issues in the daemon, though it could timeout.

Perhaps running the existing GUI's in their own processes would allow a more convenient design... I.e. the daemon wouldn't need to handle time-sensitive stuff like echo'ing keys and screen updating. Being rpc/transaction based, it could have 1 second latencies in responding to an rpc call, now and then and it likely wouldn't be so much of an issue. Vs. If it it has to update the screen and process keys for input, a 1-second hang is noticeable.

Just a thought -- not sure what the reasons were for designing multiple programs that don't re-use the same communications code (i.e. if the daemon is running, then starting the gtk interface might stomp on the daemon, or vice-versa as most vendors don't bother to ensure that each would run in its own space and don't overwrite another instance.

comment:60 follow-up: Changed 7 years ago by jusid

Transmission already uses threads. It preforms torrent verification in a dedicated thread. So it is not hard to add dedicated threads for RPC, file pre-allocation, file moves.

comment:61 in reply to: ↑ 60 Changed 7 years ago by Astara

Replying to jusid:

Transmission already uses threads. It preforms torrent verification in a dedicated thread. So it is not hard to add dedicated threads for RPC, file pre-allocation, file moves.


Isn't torrent verification done on a torrent-block-by-block basis? (i.e. not a file checksum or checksum for whole torrent)? And aren't some torrents 100's or 1000's of blocks long? The main (though *rare*) issue for me is not really even using 1 full cpu for checksumming (last I checked there were waits in the disk i/o). What would be much more awesome, would be being able to say how many concurrent verification threads I could have run at a time... I.e. on a 2-cpu (12-core) machine, even though it's only 1.6-2.4GHz (clocking down vs. turbo (which is rare)), it shines under parallel loads. Wrote a script to convert my CD's to flac and it takes about 10s/album, though, it can take over 30s if it is only 1 song on the entire album. But 99% of my albums take under 10s to go from wav->highest optimized flac.

I'd estimate at least up to 8x in speed boost if I could run verification in multiple threads (and no disk waits, obviously)...

But that said -- I'd much prefer work for finer grain priority control than torrent verification -- the latter not being something I do alot of.

(FWIW -- I set my verifications to be done in "all-at-once-mode"...)

comment:62 Changed 7 years ago by surias.0

#5609 closed as a duplicate of #1753.

5 years and running.. I guess I should consider myself lucky that this just started happening over the last several months...

comment:63 Changed 6 years ago by lazybones

This ticket has been open a very long time... I was checking in to see if there way any progress...

Torrents continue to get bigger and so does the time transmission is completely unresponsive..

incomplete-dir is becoming almost unusable to me, a 7-17GB torrent doing a file move between local and remote disk takes a significant amount of time. When you are running the daemon via the RPC you basically loose all contact for well over 1min not knowing if it is down or still working on moving the file.

If the RPC didn't go off line I would be a little happier, but in general I won't consider this fixed until the work doesn't impact the system as a whole. If I didn't care about seeding my torrents a forked shell script on complete would work better to move the files... However moving and renaming within the client is a major feature for seeding long term.

Heaven forbid you queue up several torrents via the RPC and they all finish near the same time.... the RPC goes up and down like a yo-yo.

comment:64 Changed 6 years ago by Astara

Comments you submit will be routed for moderation. Modify ↓ ======= The bug system is broken -- I've seen no other bug system that requires moderation in order to submit or update a bug (update one's own at least).

What problems are you seeing...??

I don't have the problems you are describing and I have about 296+ torrents in the seeding state of which 61 are active. When I move a torrent it is usually between directories and that happens instantly. The only lags I have are on torrent verification when multiple parts could have chksums (generic term) computed in parallel. xmission tops out at about 130-140MB/s with 100% of a cpu used (which is why I say using multiple cpu's to verify would help).

What type of operations can you have going in the background at the same time?

comment:65 Changed 6 years ago by lazybones

@ Astra

Transmission-daemon 2.84 Linux Ubuntu

  • I am using incomplete-dir which means during initial download the torrent saves to the incomplete-dir path and then at on completion moves to its final location.
  • the torrents are moving from incomplete-dir which is a LOCAL disk to a REMOTE disk.. I do this because the I/O during download is somewhat HIGH to do over the network.
  • Transfer speed between the LOCAL incomplete-dir and the remote share is between 96-100MB, however with a 11 GB torrent that still takes about 1min30s per torrent.
  • During the transfer transmissions API and web UI are completely unresponsive. When many torrents are queued up this happens repeatedly as they complete.

comment:66 Changed 5 years ago by roberth

  • Cc roberth.sjonoy@… added

comment:67 Changed 5 years ago by ictus

  • Cc decathected@… added

comment:68 Changed 5 years ago by thecoda

As @lazybones states, this is ALL about having the incomplete and download directories on different filesystems.

  • If both are local, then the file move is near-instantaneous. The design of the app seems to assume that this will always be the case.
  • If both are on the same remote mountpoint, the move is still very fast (but this is not an ideal configuration, due to the heavy I/O for downloads in progress)
  • If incomplete is local and complete is remote then the move operation is slow, and hangs the entire app. I use a remote client and any download over 1Gb is usually enough to cause the connection to time out.

However this is achieved (forked process, thead pool, background shell op, etc.), the important goal is that file moves don't cause the UI/API to hang.

comment:69 Changed 5 years ago by Astara

It seems like it should be 'semi-trivial' on mac & *nix, to fork off a copy of 'mv' in the background, and use the child-sig to know when it is done? On windows... I suppose similar would be cmd +copy?

If it was on linux, I'd likely use "nice -19 ionice -c3 mv <old> <new>. BTW, 96-100MB is about tops for a disk transfer unless you have SSD's or RAIDs. If you want *more control*, I'd use 'dd' for single file transfer and use and use "oflag=direct", to bypass the system cache -- it should go faster in most cases and won't bog down the system by eating cache (note -- I do have a RAID, and it is ALWAYS faster on a large file move to write directly to disk. The larger the buffer you give 'dd', the better (up to ~256MB).

Yeah, windows "cmd /c move [from] [to]"... I don't know how to set windows ioprio from the cmd line though.

comment:70 follow-ups: Changed 5 years ago by dannyzb

Has anyone addressed this? How do I go around this issue?

I have a system set up that finishes a torrent on average every 20 seconds, and having the system frozen half the time is becoming unusable.

On average, there are 100 1GB torrents downloading simultaneously on a 1GBit connection with Raid5. Everything possible tuneable on the system has been tuned, the only thing left slow is transmission freezing when torrents download, as mentioned in this issue

If you do not intend to fix this issue anytime soon (And it doesn't look like you are) What can I do to the system so it stops freezing?

Some random times 5-6 torrents will finish simultaneously and freeze the system for over a minuts.

comment:71 in reply to: ↑ 70 Changed 5 years ago by roberth

Replying to dannyzb:

Has anyone addressed this? How do I go around this issue?

I have a system set up that finishes a torrent on average every 20 seconds, and having the system frozen half the time is becoming unusable.

On average, there are 100 1GB torrents downloading simultaneously on a 1GBit connection with Raid5. Everything possible tuneable on the system has been tuned, the only thing left slow is transmission freezing when torrents download, as mentioned in this issue

If you do not intend to fix this issue anytime soon (And it doesn't look like you are) What can I do to the system so it stops freezing?

Some random times 5-6 torrents will finish simultaneously and freeze the system for over a minuts.

Run it in daemon mode instead, that helped me avoid complete system freeze.

comment:72 Changed 5 years ago by dannyzb

It is running as a Daemon on a headless server. The issue is - not being able to track progress more than once per minute is pretty terrible.

I've actually got to the point where I wrote code to run with rTorrent/deluge and they both turned out to be far inferior in speeds to transmission(surprisingly slower actually!)

Even though transmission is considerably faster, this issue needs a solution. Torrents are getting more and more massive and HDDs are not catching up.

comment:73 in reply to: ↑ 70 Changed 5 years ago by mike.dld

Replying to dannyzb:

What can I do to the system so it stops freezing?

Don't use incomplete directory, so that Transmission doesn't move any data on completion...

comment:74 follow-up: Changed 5 years ago by dannyzb

the incomplete directory isn't used and nothing is moved. The only system loads left are writes, checksums and any tricks Transmission plays with buffering, unless I've missed something. The server has enterprise-grade HDDs but it still peaks and stalls (not surprisingly with 65MB/sec download rate.. It's ok if Transmission can't keep up... but how do I stop it stalling?)

comment:75 Changed 5 years ago by mike.dld

Closed #6029 as duplicate of this ticket.

comment:76 in reply to: ↑ 74 Changed 5 years ago by Astara

Replying to dannyzb:

the incomplete directory isn't used and nothing is moved. The only system loads left are writes, checksums ...

---

Unless you have special HW, transmission will only calculate

checksums at a max of ~ 200MB/s. If you have 'Enterprise grade HD's, that doesn't say what file system, or their speed. Enterprise Hd's can run from 5400RPM - up to 15000RPM, but again, max out at less than 200MB/s on a single HD.

If you want speed go with a RAID10. You might do with RAID0, if you use SSD's and keep regular backups, but for speed, you need an array of HD's... RAID5 and 6 will both slow down writes considerably even if done on a controller card. You also haven't mentioned your OS, CPU or memory. If you are tight on memory, have slow CPU's or are running a MacOS, all those can affect speed.

I don't see any reason why xmission would need to recalc checksums at the end. I don't see that happening on my machine. But for checksums... the best thing there is divide and conquer. I keep my audio CD collection in FLAC and on-line. Used to take 3-5 minutes depending on the CD on a single CPU -- now, with highest optimization, running encode streams on each cpu, a single CD takes about 30-40 seconds. I had it down under a minute just using a shell script and w/perl and semaphores got it down to usually under 30 seconds. It's trivial. No special threads -- just spawn/fork off a separate process for each file or if it needs to be done by 'parts', divide the parts by #cpus.

The only thing that is slow on xmission is the single-threaded verify, which doesn't happen that often (it does help to recompile the source with optimization turned on and with the wait/sleep states commented out (that were put in for the Mac, because it has a slow disk subsystem).

Another thing the code prevents is using the OS to regulate memory, cpu IO priorities/usage and letting it run with 10K file descriptors (default on Windows -- but easily set on linux). Again, apparently Macs are rarely configured as servers -- so they don't have the higher limits, so transmission runs a higher overhead to open & close files on demand to keep # of open files low -- but each file open is costly and closing them throws away buffers resulting in more memory fragmentation and slower performance. It really does well given all the things it has in it that decrease efficiency, but it is rare to see xmission used on a seedbox -- usually rtorrent or such -- because, I guess it was optimized for servers -- which xmission doesn't have as an option -- it tries to be a one-size fits all and does a darn good job for what it does -- just irks me that usually 'one-size-fits' all usually works to fit the low end of the market more than the upper end.

The server has enterprise-grade HDDs but it still peaks and stalls (not surprisingly with 65MB/sec download rate.. It's ok if Transmission can't keep up... but how do I stop it stalling?)

I can build a linux kernel running over 100 processes at the same time (system load goes over 100) -- but it doesn't slow down everything else... I can watch full screen video's or whatever, so I don't get what you mean by when you say "stalls out".... Just because 1 process stalls, it shouldn't take out the OS. What do you mean by "stalls out"...what stalls out? You need to define what you mean -- and also figure out what is causing the slowdown. If you don't have RAID, your disks won't keep up with hardly any load -- a single HD just can't handle today's loads.

comment:77 Changed 5 years ago by antivirtel

  • Cc antivirtel@… added
  • Version 1.42 deleted

I just can confirm, that it is still a problem with the latest version. I hope someone start fixing this issue, since it is a very basic architectural problem.

comment:78 Changed 5 years ago by Robby

  • Version set to 1.42
Note: See TracTickets for help on using tickets.