Opened 6 years ago

Closed 6 years ago

Last modified 6 years ago

#2993 closed Bug (fixed)

"Downloaded" much greater than "Have" or "verified"

Reported by: dskchk Owned by: charles
Priority: Normal Milestone: 2.00
Component: libtransmission Version: 1.91
Severity: Normal Keywords:
Cc: trejkaz@…, krx@…

Description (last modified by charles)

This issue is about what has been reported in forum topic: http://forum.transmissionbt.com/viewtopic.php?f=2&t=9566

With both Transmission 1.83 and 1.91 I almost always have an overdownloading on my files.

Some examples:

EXAMPLE 1 (with 1.83):
Size: 699.8 MB
Progress: 88.52%
Have: 619.5 MB (618.4 MB verified)
Downloaded: 839.8 MB
Uploaded: 237.8 MB
Ratio: 0.28
Wasted: 0 b (0 hashfails)

EXAMPLE 2 (with 1.83):
Size: 1.37 GB
Progress: 100.00%
Have: 1.37 GB (1.37 GB verified)
Downloaded: 1.62 GB
Uploaded: 1.24 GB
Ratio: 0.76
Wasted: 10 MB (19 hashfails)

EXAMPLE 3 (with 1.83):
State: Downloading
Progress: 90.64%
Have: 634.3 MB (633.6 MB verified)
Downloaded: 810.8 MB
Uploaded: 586.9 MB
Ratio: 0.72
Wasted: 0 MB (0 hashfails)

EXAMPLE 4 (with 1.91):
Finished
Size: 700 MB
Downloaded: 726 MB
Wasted: 0 MB (0 hashfails)

EXAMPLE 5 (with 1.91):
Finished
Size: 700 MB
Downloaded: 815 MB
Wasted: 0 MB (0 hashfails)

I'm using transmission-daemon on a WD MyBook? World Edition NAS:
uname -a: "Linux MyBookWorld? 2.6.24.4 #1 Tue Feb 10 11:00:22 GMT 2009 armv5tejl unknown"
Installed through the cs05q1armel feed of NSLU2-Linux OptWare? packages.

Attachments (2)

overdownload.png (36.3 KB) - added by trejkaz 6 years ago.
Another example
Screen shot 2010-03-11 at 4.53.04 PM.png (16.7 KB) - added by jpvm 6 years ago.
Ruge difference

Download all attachments as: .zip

Change History (53)

Changed 6 years ago by trejkaz

Another example

comment:1 Changed 6 years ago by trejkaz

I have the same problem, and I attached a screenshot to show how bad the ratio can be. :-(

comment:2 Changed 6 years ago by trejkaz

  • Cc trejkaz@… added

comment:3 Changed 6 years ago by charles

It's very odd. Not many people seem to be experiencing this, and I'm not seeing it at all. I wonder what conditions cause this?

Can you confirm what version you used to download these torrents? If you started downloading it under one version and then upgraded, or if you downloaded the entire thing under a specific version, and if so, what version was it?

If this is something that's happening repeatedly, could you please try a sample torrent in 1.91 and record the have + downloaded + corrupt at various stages of downloading? I'm curious whether the overdownloading accumulates throughout the entire process, or if it shows up mostly at the end.

comment:4 Changed 6 years ago by dskchk

Hi Charles, I have torrents fully downloaded with 1.83, others fully downloaded with 1.91. I don't think I have any torrents partially downloaded with 1.83 and then with 1.91.

If I monitor the have + downloaded + corrupt, it seems that the overdownloading accumulates throughout the entire process.

For instance, now I'm downloading a 700MB file, have 209, downloaded 244, wasted 0 (0 hashfails). Progress is 29,9%.

I will sample the progress and let you know. By the way, this is something happening repeatedly, but the ratio total size/total downloaded differs from file to file.

Someone in the forum stated that this didn't happen with 1.76.

comment:5 Changed 6 years ago by dskchk

One thing I still can't explain: if the overdownloading is due to corrupted data, why isn't it appearing in the wasted/hashfails statistics? Where is going that overdownloading amount of data?

comment:6 Changed 6 years ago by dskchk

Now that file is at 45,5%, have 319MB, downloaded 385MB, wasted 0 (0 hashfails).

comment:7 Changed 6 years ago by bollywood

I am seeing this bug too.

1.91+ (10291)

comment:8 Changed 6 years ago by dskchk

That torrent has finished by downloading 806MB (while its total size is 700MB). 0 wasted byte (0 hashfails). The torrent contains two files.

comment:9 Changed 6 years ago by Rascal

I just wanted to confirm that I'm seeing this too, exactly what the others are describing. I'm not on my main computer but out of 7 recent torrents, all approx. 700MB, there was a discrepancy (wasted download) of about 100-160MB. Still using 1.83, some public torrents, some private trackers.

comment:10 Changed 6 years ago by Rascal

Oh, I also wanted to mention one thing that I noticed. The discrepancy happens throughout the download. As the torrent completes, I find that it often takes a while to get the last few KB. I have watched as the downloaded amount continues to rise, but the "have" and "verified" remain stuck at 99.something. Hope that's helpful, and thanks for the continued work on this great app.

comment:11 Changed 6 years ago by charles

Okay, I found an issue that could cause overdownloading in some circumstances. Could you please try a nightly build of r10301 or higher from http://transmission.xpjets.com/ and report back whether the problem persists in torrents newly-downloaded in that nightly build?

comment:12 Changed 6 years ago by dskchk

Hi Charles, great news!! Unfortunately I can't test it until the corresponding package for NSLU2-Linux (cs05q1armel feed) has been prepared... I do not have enough knowledge to compile it myself :-(

I hope someone else here in the bug report has this oportunity, otherwise we must wait for the next Transmission version to be released...

comment:13 follow-up: Changed 6 years ago by bollywood

Still happening with r10301

from the inspector file size: 2.70MB Downoaded: 2.82MB

which looks a lot better, however Global DL: 4.91MB

comment:14 follow-up: Changed 6 years ago by Rascal

Thanks for looking into this Charles!

Here are my results with r10301 on a new torrent, no other torrents active.

File total size:700MB Progress: 50% Have: 350.0 MB (347.9 verified) Downloaded: 377.3 MB Failed DL: 0 Bytes

So far it would appear that the overdownload quantity is less than in 1.83, but it's difficult to say without having a wider sample to look at. Will post final stats when the download is complete.

comment:15 in reply to: ↑ 13 Changed 6 years ago by charles

Replying to bollywood:

Still happening with r10301

from the inspector file size: 2.70MB Downoaded: 2.82MB

A few MB is not enough for a decent test. You would hit endgame nearly as soon as you started downloading.

comment:16 in reply to: ↑ 14 Changed 6 years ago by charles

  • Description modified (diff)

Replying to Rascal:

Thanks for looking into this Charles!

Here are my results with r10301 on a new torrent, no other torrents active.
File total size: 700MB
Progress: 50%
Have: 350.0 MB (347.9 verified)
Downloaded: 377.3 MB
Failed DL: 0 Bytes

This is a little better test -- the overall size is larger, and you're also showing me how far into the DL we are. We can't have hit endgame when there's still 350 MB left to go.

Still, it's much different than what I'm seeing. :(

I just downloaded two test torrents in r10301: Torrent Size: 573.7
Have: 583.7
Downloaded: 583.8

Maybe the swarm properties have something to do with it. What is your average download speed on these tests?

Hmmm

comment:17 Changed 6 years ago by charles

In that previous test, "Torrent Size" should be 583.7 --

Size: 583.7
Have: 583.7
Downloaded: 583.8

comment:18 Changed 6 years ago by Rascal

Just hit 75%

Have: 525.1 MB (523.0 verified)
Downloaded: 567.7 MB
Failed DL: 0 bytes

I'll check back in at 100%. The torrent is relatively new with a lot of peers, so it's moving quite slowly, 50-100 KB/s, but usually closer to 50 KB/s

comment:19 Changed 6 years ago by Rascal

100%

Have: 700.0 MB (700.0 verified)
Downloaded: 760.6 MB
Failed DL: 0 bytes

Let me know if you want any further info from me Charles, and thanks again!

comment:20 Changed 6 years ago by charles

I'm going to test out different swarms, and limit my upload and download caps in various combinations, to see if I can find some pattern to this. I've contacted a couple of tracker admins who have analyzed Transmission's swarm footprint and it doesn't seem to be a widespread issue. But even if it's only for a small percentage of users, 8% overhead is far too much on a 700 MB torrent IMO.

Rascal: thanks for describing the general speed you're getting... in addition, could you tell me (regardless of the actual speed you're getting) if you have either upload or download speed caps set, if so what they are, and what your ISP limits are?

comment:21 Changed 6 years ago by dskchk

In my case, I have the following:

  • normal mode (during the night): no limits down, 30KB/s up
  • turtle mode (during the day): 3000 KB/s down (that is, no limit in practice for my connection), 15KB/s up

My connection is an ADSL connection of about 1500 Kbit/s down, 384 KBit/s up.

comment:22 Changed 6 years ago by Rascal

My connection is ADSL2+, and I get 3.5 Mbit/s down, 600 kbit/s up.

On good torrents with fast seeders, downloads and uploads are consistently at max. There is no cap from the ISP on bandwidth, and I usually don't put a limit on transfers unless the leeching is ridiculous and affecting my uploads to sites which require good ratios.

comment:23 Changed 6 years ago by dskchk

I forgot to say that I don't have any cap on ISP side, too.

comment:24 Changed 6 years ago by charles

  • Component changed from Transmission to libtransmission
  • Owner set to charles
  • Status changed from new to assigned

Xref: #3014

comment:25 Changed 6 years ago by dskchk

Hi Charles! #3014 is great news, although you say that "contributes" to this. Does it mean that there are also other reasons for this?

comment:26 Changed 6 years ago by charles

Yes, that's what I mean. I think r10332 fixes the rest of it.

comment:27 Changed 6 years ago by bollywood

10334 is a big improvement, Ill keep testing and post back. I think i had around 1MB unaccounted for on a 700MB d/l.

great job!

comment:28 Changed 6 years ago by bollywood

r10334 File size: 412MB Have: 412MB Downloaded: 418.2MB Failed DL: 0MB

Global DL: 418.8MB

comment:29 Changed 6 years ago by Rascal

Hi Charles,
I'm having problems with r10334. Download speeds are extremely slow, and while the torrents are new and not very well seeded, I'm talking speeds of 0.5 to 10 KB/s. The torrents were started with r10301 so I noticed the dramatic drop in speed. Uploads speeds are not affected from what I see. I think it has to do with the changes recently made. It would appear that a lot of peers are not interested in sending data. For example, on one torrent I am connected to 44 peers, but downloading from only 2. On another downloading from 3 peers, connected to 36. I'm going back to r10301 for the time being. Thanks for your work on this issue, it's appreciated!

comment:30 Changed 6 years ago by charles

charles * r10346 libtransmission/peer-mgr.c: (trunk libT) #2993 "'Downloaded' much greater than 'Have' or 'Verified'" -- tweak the new throttle code based on slow download feedback from gn0s1s in irc, from Rascal @ http://trac.transmissionbt.com/ticket/2993#comment:29, and from AGSystem @ http://forum.transmissionbt.com/viewtopic.php?p=45631&f=1#p45631

comment:31 Changed 6 years ago by charles

r10346 also allowed some cleanup to take place in r10347

comment:32 Changed 6 years ago by Rolcol

I see the same thing as Rascal and I was up to r10353 when I added a new torrent to Transmission. The torrent itself had only been posted a few hours prior.

Changed 6 years ago by jpvm

Ruge difference

comment:33 Changed 6 years ago by jpvm

after updating to r10353 this issue seems to be solved or, at least, very improved.

My download (I'm still trying to finish the one used in the sample I've uploaded to this issue) is being completed faster and the amount downloaded seems to be correct (or less wring). I simply installed the r10353 version over the 1.9.1 release and relaunched Transmission and the issue seemed to be solved.

comment:34 follow-up: Changed 6 years ago by dskchk

Charles, what is the state of this in 1.92?

comment:35 in reply to: ↑ 34 Changed 6 years ago by charles

Replying to dskchk:

Charles, what is the state of this in 1.92?

#3014 is fixed in 1.92, so things should be better than 1.91.

However since the throttle code is still experimental -- some people are still reporting sluggish downloads as a result -- that didn't make the cut for 1.92.

comment:36 Changed 6 years ago by setek

Hi, new here so forgive me if I'm speaking out of turn.

Started torrent in Transmission 1.91:

Have 199.2mb
Verified 194.0mb
Downloaded 804.1mb
Failed 0 bytes

Switched to and completed in Transmission 1.92 r10364:

Have 700.0mb
Verified 700.0mb
Downloaded 1.29gb
Failed 0 bytes

Started torrent in Transmission 1.92 r10364, speed limited to 50kb/s:

Have 474.0mb
Verified 474.0mb
Downloaded 476.8mb
Failed 0 bytes

So it works much, much better! I did take some figures periodically for the first torrent, and noticed if you were speed limited quite drastically (e.g. 16kb/s) there was still a decent amount of wasted download without it showing up as failed, but it was a bit inconclusive.

Have not noticed any sluggish downloads as a result of this throttle code, will keep an eye out if I notice anything else.

comment:37 Changed 6 years ago by dskchk

I updated my Transmission installation to 1.92 (10363) and I started a new download with it to test. This is the current situation:

Have: 1,57 GB Verified: 1,57 GB Downloaded: 1,62 GB Wasted: 6 MB (6 hashfails) Progress: 19,78%

That is, about 3% of overdownloading. Surely better than before (although I would have to test more), but still quite high...

comment:38 Changed 6 years ago by charles

dskchk: Please see three comments up where I answered you on this question already. The nightlies are better than 1.92, and 1.92 is better than 1.91.

There are two branches, so the new code added to the nightlies in r10332 is not in 1.92 (r10363). Don't get confused by the revision numbers.

comment:39 Changed 6 years ago by dskchk

Ok, thank you. I just wanted to report my own experience with 1.92. As I said before, I cannot test the nightly builds on my system.

comment:40 Changed 6 years ago by krx

  • Cc krx@… added

comment:41 Changed 6 years ago by dskchk

Hi Charles, do you have any plan to target this for 2.0?

comment:42 Changed 6 years ago by Rolcol

After 5 weeks of my initial comment, this is still slow. I added 2 torrents that were released roughly 30 minutes ago. One is "Downloading from 2 of 214 peers" while the other one is "Downloading from 2 of 204 peers" (~3000 peers known in each).

comment:43 Changed 6 years ago by jamesk5

I've got this (or a similar problem) happening when I throttle torrents.

Transmission will download quite a lot of data, and I'll need to restart the app to get it to verify the download - at which point it wont download any more because it's already got more than enough.

An example of the numbers is something like the following:
Have: 183.5 MB (183.5 MB verified)
Downloaded: 1.04 GB

I'm running the latest official Transmission release (1.92) available from the the Ubuntu packages under a fully up-to-date install of Lucid.

comment:44 Changed 6 years ago by charles

jamesk5: you're saying that Transmission is overdownloading by 5x on your system??? I've never seen that behavior. Also I don't understand what you mean by "won't download any more because it's already got more than enough".

comment:45 follow-up: Changed 6 years ago by dskchk

Still reporting my own experience with 1.92 after quite a lot of testing. My overdownloading rate is between 3% and 8% (more or less) depending on the torrent. These percentages are calculated on the "good" bytes, that is after discarding the wasted bytes (hashfails). As I said, much better than 1.91, but I still need an improvement (on a 10GB download Transmission has downloaded about 800 MB more...).

Last edited 6 years ago by dskchk (previous) (diff)

comment:46 in reply to: ↑ 45 Changed 6 years ago by charles

Replying to dskchk:

Still reporting my own experience with 1.92 after quite a lot of testing. My overdownloading rate is between 3% and 8% (more or less) depending on the torrent.

Please reread comment:33 and comment:35 :)

comment:47 Changed 6 years ago by charles

05:12:34 < CIA-45> charles * r10510 libtransmission/peer-mgr.c: (trunk libT) #2993 '"Downloaded" much greater than "Have" or "verified"' -- tweak the download throttle algorithm a bit to try & address the slowness reported by Rolcol @ http://trac.transmissionbt.com/ticket/2993#comment:42

comment:48 Changed 6 years ago by Rolcol

Speeds have vastly improved since r10510. I tested only with two new torrents, but it looks consistent. Thanks, charles!

Last edited 6 years ago by Rolcol (previous) (diff)

comment:49 Changed 6 years ago by charles

  • Milestone changed from None Set to 2.00
  • Resolution set to fixed
  • Status changed from assigned to closed

comment:50 Changed 6 years ago by dskchk

I'm now using Transmission 2.00. Things seem to have been improved, but I still see cases of significant overdownloading. For instance:

Size: 1,31 GB
Downloaded: 782 MB
Have: 703 MB
Wasted: 256 KB (1 hashfail)

That is, about 11% of overdownloading. This download was started using Transmission 2.00 from scratch and it is veeery slow (because of few slow sources).

comment:51 Changed 6 years ago by dskchk

Another example:

Size: 1,36 GB
Downloaded: 450 MB
Have: 412 MB
Wasted: 0

That is about 9% of overdownloading.

Note: See TracTickets for help on using tickets.