Opened 13 years ago

Closed 10 years ago

Last modified 8 years ago

#1242 closed Bug (fixed)

Local data can be corrupted by bad peers when Transmission accepts duplicate blocks during endgame

Reported by: reptileman Owned by: charles
Priority: High Milestone: 1.92
Component: libtransmission Version: 1.91
Severity: Critical Keywords:
Cc: eisa01@…, grzegorz.dubicki@…, staze@…

Description

I have had problems with torrents that are at 100% not being fully downloaded. My files are corrupt until I do a verify and then it usually tells me its downloading at 98.x% again. Usually a verify will fix it first time. Other times I've had to import the torrent to azureus to finish it off properly.

Change History (28)

comment:1 Changed 13 years ago by smmalis

This has happened to me before. My theory is that for some reason Transmission will download a corrupt part from someone but not realize it until you do a manual verify. At this point T will recognize the bad part and redownload it from the same person, which just causes the problem again. This is my theory.

comment:2 Changed 13 years ago by charles

reptileman: do you see this behavior consistently with particular torrents, or does it vary? I'm not seeing this behavior at all, and am wondering what I need to try to do to reproduce it.

comment:3 Changed 13 years ago by charles

reptileman: ping

comment:4 Changed 13 years ago by livings124

  • Resolution set to worksforme
  • Status changed from new to closed

comment:5 Changed 12 years ago by eisa01

  • Cc eisa01@… added
  • Resolution worksforme deleted
  • Status changed from closed to reopened

I just had this happen to me.

I noticed the file sounded strange when playing in iTunes, so I tried to recheck it. Turns out it had some corrupt pieces. It downloaded anew, I took a new recheck and it still had a corrupt piece. It managed to get it correctly on the last download.

I was actually manage to reproduce this just now, so I'm opening the bug again. Send me an email Charles, and I'll give you the .torrent file.

Running 1.91 (10268)

comment:6 Changed 12 years ago by eisa01

Sent you an email about it, I'll add it here for completeness sake: How I reproduced this: Running 1.91 for OS X Open the torrent in Transmission Choose to only download the first track Wait for it to finish Choose "Verify Local Data" from the Transfers menu Discover that some pieces are corrupt.

If it matters, I'm on a fast internet connection currently. Transmission reports speeds up to 3 MB/s during the short download. I tested capping the speed at 100 KB/s, that lead to less pieces having to be redownloaded, so it seems to be affected by the download speed.

comment:7 Changed 12 years ago by nifoc

  • Version changed from 1.33 to 1.91

I can confirm this. While reading through this ticket a download finished, so I was just curious and verified it. As it turn out, it had a few corrupt pieces.

comment:8 Changed 12 years ago by grzegorzdubicki

  • Cc grzegorz.dubicki@… added

comment:9 Changed 12 years ago by Tantali

I have the same problem, I had opened a ticket for it too, so this is the same ticket as #2700.

comment:10 Changed 12 years ago by charles

Could someone mail me the URL of a public torrent that has this behavior?

comment:11 Changed 12 years ago by charles

Also, do you still see the behavior in the nightly builds of r10321 or higher from http://transmission.xpjets.com/ ?

comment:12 Changed 12 years ago by eisa01

Still there for me.

I'll mail you a link in one min.

comment:13 Changed 12 years ago by charles

  • Component changed from Transmission to libtransmission
  • Milestone changed from None Set to 1.92
  • Resolution set to fixed
  • Status changed from reopened to closed
  • Summary changed from Not fully downloading torrent despite 100% status to Local data can be corrupted by bad peers when Transmission accepts duplicate blocks during endgame

Okay, after downloading a specific single file again... and again... and again... and again... I've finally found a pattern:

In the swarm that was emailed to me, there's at least one Bad peer constantly sending bad blocks out to everyone. During endgame, Transmission can double-request a block from two peers. (This isn't the problem... all clients do this. If you're new to this idea, go Google "endgame") If one of those peers is the Bad peer, and if the good peer's block comes in first and if that block completes the piece, then we checksum the piece and mark it as good. Then when the bad peer sends in its bad version of the block, we wrongly save it too, instead of throwing it away. And since the piece is already flagged as complete, we don't re-checksum it.

I've committed a fix for this behavior to trunk in r10325 for 1.92. I think it's "the" fix and ran 10 trials after making the change, and can no longer trigger the corruption bug.

Since so many "ifs" were required to tickle this bug in the first place, I'd be happy if other people could test it out to try and confirm the fix.

comment:14 Changed 12 years ago by charles

  • Severity changed from Major to Critical

comment:15 Changed 12 years ago by eisa01

I haven't been able to reproduce it again, but I had to test on other torrents than my original as the peer that sent the bad data must have disappeared.

comment:16 Changed 12 years ago by Tantali

I've downloaded a bunch of torrents which should at least have generated this bug once, but it didn't. So as far as I'm concerned, this bug is solved.

comment:17 Changed 12 years ago by charles

reopened for attribution

comment:18 Changed 12 years ago by charles

  • Resolution fixed deleted
  • Status changed from closed to reopened

comment:19 Changed 12 years ago by charles

  • Owner set to charles
  • Status changed from reopened to new

comment:20 Changed 12 years ago by charles

  • Resolution set to fixed
  • Status changed from new to closed

comment:21 follow-up: Changed 10 years ago by staze

  • Cc staze@… added
  • Resolution fixed deleted
  • Status changed from closed to reopened

Please let me know if I should open a new ticket on this, but I seem to be seeing this same behavior in Transmission 2.33.

A torrent will "successfully" complete, then after doing a "verify local data", I'll see a chunk is missing/corrupt, and it'll download again. Usually this amounts to about 0.01-0.03%. Then reverifying results in the same thing, though usually after about 5-10 times of doing this over and over, things finally work and I get a fully verifiable torrent.

Could this fix have been rolled back for some reason? Or this "bug" reappear due to some other change?

comment:22 in reply to: ↑ 21 Changed 10 years ago by jordan

Replying to staze:

Could this fix have been rolled back for some reason?

Nope, it hasn't been rolled back.

Or this "bug" reappear due to some other change?

You're the first person to report it, but sure it's possible. Is the behavior repeatable? consistent? Can you give me a walkthrough of the exact steps I should follow to trigger the behavior?

comment:23 Changed 10 years ago by lm2s

I have experienced this bug too, in 2 distinct files.

I believe I might have ran out of disk space in the middle of the transfer and from that point on the data is all bad. This 2nd file stopped at around 50-60% because of disk space, I deleted some stuff and resumed, it downloaded till 99 or 98 and said to verify data, I just did resume and it went to 100%. Tried to open the file and it was damaged, so I did a Verify Local Data and the file went back to 50%...

comment:24 Changed 10 years ago by jordan

  • Resolution set to fixed
  • Status changed from reopened to closed

Thanks for the report, but it sounds like a separate issue unrelated to the endgame bug which this ticket was closed for 18 months ago.

I'm going to reclose this ticket, but please feel free to open a new ticket will a full description / walkthrough of the issue you're seeing. Thank you!

comment:25 Changed 10 years ago by staze

See, I'm not sure if it's the same bug or not. It SOUNDS like the same bug. You download something, and once Transmission says it's done, you, say, go to unrar everything, and that throws an error saying CRC failed. So you do a "Verify Local Data" with Trasmission, and it gets to be about 99-99.98% complete, then redownloads a chunk. So you do a Verify again, and it proceeds to download the "same" data again. But as you do this several times, the chunk it's redownloading seems to get smaller, until about 5-10 times later, you have something that's complete.

I'm not sure how reproducible it is, other than several of the torrents I've downloaded as of late have done this. However, I have not seen this issue on another machine.

Computer in question is a Mac Mini 2009, RAID1 HD setup, with 4GB of ram, running 10.6.8 1.1 server. If you could point me in the direction of how to adequately test something like this, I'd be happy to.

My issue isn't due to disk space. I have 200-300GB available on that drive at all times.

comment:26 Changed 9 years ago by reyiyo

Hi! I'm experiencing the same problem. Torrents finish 100% download but after some hours, become paused with this error and a completion of 99.x%. Verify local data does not change its state and when started again, the problem is repeated over and over, with several files (not all). I'm running Transmission 2.52 (13304) on Debian Sid. I've already checked that is not a disk problem, as it happens in different machines, with ext3, ext4 and ntfs filesystems. Could you please tell me how can I bring more information for this issue?

comment:27 Changed 8 years ago by williampoetra

I'm using Transmission 2.61 (13407) on Linux Mint 14 and I'm experiencing the same thing as reyiyo. dmesg doesn't show any disk/fs problem, and smartctl reports that my drives are fine. Verify local data sometimes makes the problem go away.

Note that I've only noticed this recently, with multiple torrents open (~20 of them) and mostly seeding, not downloading.

comment:28 Changed 8 years ago by x190

williampoetra: Please see #4649.

Note: See TracTickets for help on using tickets.