High CPU use in refillPulse()

[qqlapraline]

Platform: Syabas (NMT)

Description: The version compiled for Syabas platform is significantly slower than the previous version (1.42 build 7496). It consumes a lot of CPU (not memory).

[livings124]

Please run a Shark trace: ​http://trac.transmissionbt.com/wiki/Shark

[qqlapraline]

Unfortunately, as stated, it's running on a NMT box: I won't be able to go much further a "top" or "ps" on this platform.

[Ger Teunis]

That binary is created by me. Some more information regarding the build: Build flags: -O2 -frename-registers -march=4kc And binaries are stripped after that.

I do not have a profiler on the MIPS media box (NMT).

[qqlapraline]

What kind of information do you need exactly ? May be GDB can be helpful ?

[charles]

Sounds good to me.

I don't know what kind of profiling tools NMT has, if any. Probably the most low-level profiling is one you could do by hand in gdb -- start it running inside gdb, and when you see it reaching a high CPU use, hit ctrl-c and get a backtrace. Then continue running and take another sample a few seconds later if the CPU use is still high.

You need to use a NON-STRIPPED executable when doing this. Stripping it tears out most of what makes the gdb backtraces useful.

Ger, do you have any other suggestions on better ways to do profiling on NMT?

[Ger Teunis]

I currently do not see the high cpu load, some do some don't. But I am willing to provide qqlapraline a non-stripped build so he can use gdb to do a manual profiling session?

[qqlapraline]

Ger, I'll be waiting for your non-stripped build. Just post a URL here.

[qqlapraline]

By the way, if you find a MIPS version of GDB first, it will save me some time...

[qqlapraline]

I think I got one here: ​http://www.macraigor.com/full_gnu.htm#For%20Linux

I'll try it and see what happens..

[qqlapraline]

Well, I've downloaded it, send it to my HDX by ftp and here is the result of GDB launch: ./mips-elf-gdbtui: line 1: syntax error: "(" unexpected

[charles]

qqlapraline: sounds like maybe you should run Ger's version :)

The session should look like this:

$ gdb transmission (gdb) handle SIGPIPE nostop noprint nopass (gdb) r then run transmission for a little while until the cpu load is high, and press ctrl-c (gdb) thread apply all bt gdb will spit out the first backtrace that you'll provide (gdb) continue repeat as necessary

[charles]

$ gdb transmission
(gdb) handle SIGPIPE nostop noprint nopass
(gdb) r
// then run transmission for a little while until the cpu load is high, and press ctrl-c
(gdb) thread apply all bt
// gdb will spit out the first backtrace that you'll provide
(gdb) continue
// repeat as necessary

[qqlapraline]

Well, I got the GDB error by launching it only...

[repattila]

Shark sample: ​http://transmission.pastebin.com/m7a9fb400

At the time transmission running with 11 transfers (6 with peers) global max peer count 300, cpu usage round 8%, with spikes of 12+, on a 2.1GHz Macbbook. I had higher cpu usage before (15% avarage) with other torrents (count 12), but don't have those anymore, still no big difference in means of speed, encrypted peer count (very low) or file number (avarage) in torrents.

[qqlapraline]

You may have a point here. I usually have many encrypted peers (private trackers). Can this change anything ?

[charles]

repattila: d'oh, I forgot that the official 1.50 release has debugging information stripped out of it. Would it be possible for you to download a nightly build and re-run the shark sample?

[repattila]

Running r7896: ​http://transmission.pastebin.com/m5f9f6980

Same torrents as before and about the same cpu load.

[charles]

repattila: much better; thanks

It looks like most of the time in that shark trace is being spent in refillPulse(). That's something that'll be worked on for 1.51.

It would be nice to hear more in the way of profiling from the NMT users. I scanned the NMT forum's Transmission thread for more information on this subject -- and as an aside, I'm very happy to see NMT users using Transmission so much, and grateful to Ger for providing builds -- but the bug reports there are so vague as to be useless...

[charles]

Other users have also reported that refillPulse() is eating a lot of CPU cycles, so for now I'm going to work from the assumption that the same behavior is taking place on the NMT. As I mentioned above (maybe too grumpily ;) I'd be happy for any profiling information of Transmission on NMT.

Refining this ticket's topic so that it's a discrete problem that can be fixed. If there are other issues affecting CPU load please open another ticket and I'll dig into them as well.

[charles]

There is an experimental fix for the refillPulse() issue in the nightlies starting in r7909. Feedback on this would be great...

[repattila]

Looking better here, cpu load around 4% (no peaks) with torrents like before. Thx for the quick response.

Made a shark report too if its relevant: ​http://transmission.pastebin.com/m7195be21

[charles]

Sounds good. Thanks repattila :)

[DaveF1]

That's made quite a big difference on my NSLU2 - now running ~20-40% CPU (with odd peaks of 90%+) instead of constant 95-100%. T-remote is now far more responsive as is the web interface (doesn't take as long to load).

Running 7914 on NSLU2 Unslung 6.8

[DaveF1]

Oops, spoke too soon there. After a couple of hours running I'm back up to a fairly constant 95-100% CPU with odd dips to ~20%. Memory usage seems to creep up over time and d/l speeds seem to drop off. Anything I can do to help? Dave

[charles]

Yes: submit a profile of what Transmission is doing when it's eating 100% of your CPU. I'm not clairvoyant :)

I don't know what profilers (if any) are available on the NSLU2. If nothing's available you might have to do brute-force profiling with gdb... see the above comments.

[DaveF1]

Can you give me some pointers ... I tried running transmission-daemon under gdb but can't connect to it to start paused torrents. I worked out how to pass command line arguments when running T under gdb but it doesn't seem to create 3 processes as it does normally (I'm a bit of a newbie when it comes to profiling) Gdb 6.8 from ipkg (optware).

[charles]

You'll want to run the daemon in the foreground

% gdb transmission-daemon
(gdb) handle SIGPIPE nostop noprint nopass
(gdb) run --foreground --your-daemon-args-go-here
// then run transmission for a little while until the cpu load is high, and press ctrl-c
(gdb) thread apply all bt
// gdb will spit out the first backtrace that you'll provide
(gdb) continue
// repeat as necessary

[DaveF1]

Sorry, I can't get Transmission-daemon running under gdb. It appears to run (once I've continued through a breakpoint in libc.so.6) but looking at the process using top it isn't actually doing anything. There only appears to be one process when running under gdb but there are 3 normally - I take it its not running properly?

[Ger Teunis]

Sorry guys; wasn't tracking this one actively. Dowload the nonstripped trunk build (revision 7929) here ​http://members.tele2.nl/ger.teunis/downloads/transmission-trunk-nmt.zip

[DaveF1]

Ger - I've noticed similar high CPU usage on NSLU2 to the original poster's NMT device. Do you have a nonstripped trunk build for NSLU2 (armb)? Thanks Dave

[Ger Teunis]

Replying to DaveF1:

Ger - I've noticed similar high CPU usage on NSLU2 to the original poster's NMT device. Do you have a nonstripped trunk build for NSLU2 (armb)? Thanks Dave

Sorry, but I cannot provide that build for you. It took me a great while to get the MIPS (read NMT) build up and running. Perhaps somebody else can provide you this build.

[qqlapraline]

Here are a few tools I've found to profile. Unfortunately, as stated, I don't have any platform where I could build a MIPS version of these...Hopefully, somebody will have time to do so.

OProfile: ​http://www.linux-mips.org/wiki/Oprofile (On MIPS ? ​http://oprofile.sourceforge.net/news/) KernProf?: ​http://oss.sgi.com/projects/kernprof/ VProf: ​http://sourceforge.net/projects/vprof/ TSProf: ​http://freshmeat.net/projects/tsprof/

[qqlapraline]

After many, many (many !) tests, I think I have issues only with public trackers. With private trackers, I don't have CPU usage issues. Hope this helps. And I still can't get any profile information....

[charles]

Some questions that DaveF and qqlapraline can answer even without a profiler. Note that it's a given that CPU load will be high when verifying local data, so please don't verify local data while testing to answer these questions:

1. Is the CPU load high when only seeding?

2. Is the CPU load high when only downloading?

3. Is the CPU load high when only downloading from seeds?

4. Is the CPU load high when only downloading, from both seeds and peers?

5. Is the CPU load high when neither seeding nor downloading?

[KyleK]

I'm experiencing the same on my NAS. I really want to do profiling, but I simply can't find a way to do it on the device. oprofile compiles but there's no kernel support. Valgrind doesn't support ARM at all. KernProf? needs a modified kernel as well. VProf uses functions that are disabled on the device. And to top it all off, gdb gives me an "generic error" when I try to start Transmssion. I also tried to compile Transmission with gprof support (-pg), but that didn't work either (can't remember the error message right now).

If anyone has any other ideas, let me here them :)

[okouais]

Same problem on my NAS (Qnap TS409Pro). Transmission 1.50 installed via optware. Use with Transmission Remote Gui (Windows) and a ssh session to check the %cpu via "top" command. Tested on 10 torrents from two private trackers and one public.

The CPU load is high when only downloading (about 20-40%) from both seeds and peers. Cpu usage and download speed seems to be linked. But there is some random 95% peaks. When only seeding, I think cpu usage is normal (less than 5%).

Any ideas?

[livings124]

okouais: As stated in this ticket, feedback on the current source would be useful.

[charles]

KyleK, okouais: even without profiling, possibly the most helpful thing you could do would be to compare for me (1) the CPU load when downloading in a swarm where you're only connected to seeds, compared to (2) the CPU load when downloading in a swarm where you're connected to both seeds and other downloaders

[okouais]

Replying to charles:

KyleK, okouais: even without profiling, possibly the most helpful thing you could do would be to compare for me (1) the CPU load when downloading in a swarm where you're only connected to seeds, compared to (2) the CPU load when downloading in a swarm where you're connected to both seeds and other downloaders

Test : 1 torrent downloading from 1 seed and 1 peer : average cpu load<5% with frequent random peaks > 15-20% 1 torrent downloading from 3 seeds and no peer : average cpu load<3% with almost NO random peaks 1 torrent downloading from 3 seeds and 1 peer : average cpu load<5% with frequent random peaks > 15-20%

Cpu load peaks seems to be linked to peer exchange ...

[okouais]

Replying to livings124:

okouais: As stated in this ticket, feedback on the current source would be useful.

Do you mean I have to open a new ticket ?

[livings124]

Replying to okouais:

Replying to livings124:

okouais: As stated in this ticket, feedback on the current source would be useful.

Do you mean I have to open a new ticket ?

No, I mean your tests should be based off of current source, not 1.50 release version.

[okouais]

Sorry for my ignorance, but it's the first time I post in Trac. I'm not a developper. I'm only a Transmission user. I only have basic Linux knowledges. I don't know how to compile current source for my device (Qnap) so I only can run the released version. Sorry again.

[charles]

Test :

1 torrent downloading from 1 seed and 1 peer : average cpu load<5% with frequent random peaks > 15-20%

1 torrent downloading from 3 seeds and no peer : average cpu load<3% with almost NO random peaks

1 torrent downloading from 3 seeds and 1 peer : average cpu load<5% with frequent random peaks > 15-20%

This is very helpful. Thanks!

[DaveF1]

My CPU usage seems to run higher than okouais - 1 torrent U/L to 6, D/L from 11, CPU ~13-15% odd peaks to 20% and occasional peaks to 95%, Memory usage starts at 38% with torrents paused, increases to 57% with 1 torrent running, gradually increases to 101% over several days (OK my NSLU2 only has 32M memory). When the Memory usage is high the CPU usage seems to run high - ie runs at fairly constant 90%+.

Any suggestions for a seeds only torrent? I'm only using public trackers.

[charles]

There's another experimental improvement for the refillPulse() issue in the nightlies starting in r7909. Feedback on this would be great...

[DaveF1]

Charles - I think the changes you made 7 days ago (according to the revision log) are what appeared to make a big difference when I first posted. I've had some slow, large torrents running for several months which have a bad seed/peer ratio. I definitely noticed a speed increase when I started using 7909 and the overall processor usage seems to be down.

I've tried downloading a ubuntu release from the site mentioned in the "slow speeds" page of the wiki - this saturates my d/l (about 300Kb) with CPU running at an average of 80% (62% memory).

I'm not having much joy with profiling either, haven't yet tried compiling to run under gprof although I've downloaded and installed it.

[charles]

The revision in my last message was wrong... that's what I get for cutting and pasting. :)

What I was really looking for was feedback on r7942.

[KyleK]

With the latest SVN build I have a steady CPU usage of about 20% (with rare spikes to 40%). This is on a NAS (500MHz ARM CPU, 64MB RAM).

Before this latest checkin, Transmission would jump to 99% CPU within a minute after starting, and stay there as long as the torrent(s) would download. When seeding, all was fine.

So, I'd say this is a very welcome improvement :)

(On a side note: I had noticed very high mem usage with versions 1.50 and before, but this seems to have gone with this update as well)

[charles]

Backported to the 1.5x branch in r7949.

[sim]

(deleted spam)