Opened 7 years ago

Last modified 3 years ago

#4649 reopened Enhancement

Option to Automatically Verify Local Data on Completion

Reported by: Renara Owned by:
Priority: Low Milestone: None Set
Component: Transmission Version: 2.33
Severity: Normal Keywords:
Cc:

Description

I realise that the bit torrent protocol is supposed to render this unnecessary, however, I still infrequently find torrents that have finished downloading with some corrupt blocks.

Now, I haven't the faintest clue why this is happening, as many torrents are just fine, while others complete with as much as 5% of the blocks failing verification. Of course if I can identify the problem I'll do my best to post a suitable bug report so that the real issue can be fixed.

However, I believe there's always going to be the remote possibility of bad blocks, for whatever reason. And so, I'd like to suggest a feature that some other torrent programs have, which is the ability to auto-verify all local data when the torrent is completed (prior to it being marked as complete, just in case).

This should be optional, as it can generate a lot of extra disk activity, but it allows users to add an extra layer of protection, rather than them having to discover the issue while trying to use whatever it is they've downloaded. I've had this happen to a Linux distribution .iso which was particularly annoying, as it means the disk I burned using it was unusable (installation failed).

Anyway, a simple enough advanced option I think, allowing users to defensively re-verify their torrents against any current, or future, issues, exploits or other integrity issues.

Attachments (1)

reverify4649.patch (4.0 KB) - added by cfpp2p 6 years ago.

Download all attachments as: .zip

Change History (60)

comment:1 Changed 7 years ago by jordan

  • Resolution set to invalid
  • Status changed from new to closed

IMO it would be more useful to get some information on what caused the torrent to be corrupt in the first place -- if we're only talking about "some remote possibility", surely those odds are better handled by the already-existing "verify local data" feature.

comment:2 Changed 7 years ago by Renara

  • Resolution invalid deleted
  • Status changed from closed to reopened

Considering there are no suitable debug tools that I'm aware of then where exactly is that information supposed to come from? I'd happily give more information if I could, but for the time being I'd much rather just have a way to auto-verify prior to completion, because let's face it, no system is perfect, and assuming it is (or even can be) would be a mistake. Especially when data errors could come from sources outside of Transmission just as easily (hard drive error or quirk, over-zealous virus-scanner, etc. etc.).

Don't get me wrong; I'd love a suite of debug tools, but I don't see how well they can really figure out where the problem lies; I mean, I could find out which blocks are corrupt, but that doesn't tell me anything about how they came to be corrupt in the first place. Might help figure out if there's an error with the check on blocks as they're downloaded (I've already checked my drives and any software I can think of is set to avoid Transmission's folders). But even so, there's no such thing as a perfect system, and the chances of an error creeping in only go up with bigger and bigger torrents.

So I think it should be an option; many users probably never have enough trouble to need to turn it on, but I know that I, as a heavy torrent user, definitely encounter problems often enough that having to remember to do it manually each time, or discover the error mid-way through using the downloaded data, is a huge pain. It's a tiny feature to add, default it to off, and everyone's happy.

comment:3 Changed 7 years ago by livings124

  • Resolution set to invalid
  • Status changed from reopened to closed

I'm going to agree with jordan, figuring out and stopping what corrupts the torrent is the better option, and the already-existing verify option can be used.

comment:4 Changed 7 years ago by Renara

  • Resolution invalid deleted
  • Status changed from closed to reopened

Seriously; do you even read any of the text or do you just glance at the title and skip to closing the issue?

To assume that every download is going to be, or even can be, perfect is not only bad-practise, it's foolish. Forcing your users to manually verify everything they download is even worse, especially when an auto-verify feature would be child's play to add (since all the difficult verification code is already there) and, just to keep everyone happy, it can be opt-in, so only user's aware of what they're doing will turn it on, the rest can just verify as required.

At least with such a feature in place it'd be possible to log and analyse what percentage of torrents are turning out to be corrupt, so we'd actually have more information than we do now. Could even log or otherwise gather the bad blocks and expected checksums so the whole thing could be analysed properly.

Currently the only info available is that torrents can download broken, and no-one knows how or why, and there's no good way to find out. Even if an issue is found, who's to say there won't be more, as yet undiscovered? Other torrent clients have this feature, because it's better to program defensively than assuming nothing can go wrong.

So either give us a way to find out where the problem lies, or give us a way to automatically verify the data on download. Ideally both, since there's always the possibility of another problem, due to other bug-fixes or new features having unexpected results, or outside causes such as over-zealous virus-scanners, OS or file-system oddities, hard-drive issues, magnetic storms, solar flares, computer gremlins, espionage… point being that no check during download can ever be perfect, so at least being able to enable an extra check on completion can allow a user to avoid errors if they want to.

comment:5 Changed 7 years ago by jordan

Other torrent clients have this feature

Which torrent clients have a preferences option to verify local data on completion, again?

comment:6 follow-up: Changed 7 years ago by jordan

  • Resolution set to invalid
  • Status changed from reopened to closed

Answering-my-own-question department: looks like rTorrent has it as an option disabled by default. I don't see the feature in Deluge or KTorrent.

It doesn't seem that this is a feature that many people are using in the wild, and I don't see any reason to think that Transmission's users would be more inclined to use this feature than other clients' users.

comment:7 Changed 7 years ago by Renara

  • Resolution invalid deleted
  • Status changed from closed to reopened

Well for one, Vuze has this feature enabled by default. If you can endure their shitty preferences window then it requires advanced view enabled, and it's under the files category as, I think, "Re-check pieces after download is done". So all Vuze users are currently using this feature.

While I don't like Vuze in general, it's a sound feature to have for all the reasons that I've already covered at length, though it's certainly debatable whether it should be opt-in as opposed to Vuze's opt-out.

comment:8 Changed 7 years ago by jordan

  • Resolution set to invalid
  • Status changed from reopened to closed

comment:9 in reply to: ↑ 6 Changed 7 years ago by marcus

  • Resolution invalid deleted
  • Status changed from closed to reopened

Replying to jordan:

Answering-my-own-question department: looks like rTorrent has it as an option disabled by default. I don't see the feature in Deluge or KTorrent.

It doesn't seem that this is a feature that many people are using in the wild, and I don't see any reason to think that Transmission's users would be more inclined to use this feature than other clients' users.

I can of course not comment on what "many people" are going to use, but I would definitely use this feature because I experience the exact same problems. I'm currently making the transmission-done script do a verify, which makes it kind of complicated to write good write good scripts that rely on the fact that a torrent is actually complete. That being said, it would be nice if transmission would make sure that the torrent is actually complete before acting as if it was.

comment:10 Changed 7 years ago by jordan

  • Resolution set to invalid
  • Status changed from reopened to closed

comment:11 follow-up: Changed 7 years ago by Renara

  • Resolution invalid deleted
  • Status changed from closed to reopened

Hey marcus, don't suppose you could share that script anywhere? It seems apparent that jordan has switched to livings124's poisonous attitude towards the trac system, which means there's no hope of anything like this ever being added, nor any useful new features by the looks of it, since neither seems interested in actually listening to the discussion, and would much rather re-close the issue so they can pretend they've actually done something here.

Seriously, why did you guys even set-up a trac system? Transmission would be better off with a burning barrel for people to put their complaints in.

comment:12 in reply to: ↑ 11 Changed 7 years ago by jordan

  • Resolution set to invalid
  • Status changed from reopened to closed

Replying to Renara:

better off with a burning barrel for people to put their complaints in.

Gotta admit, that one made me laugh. :)

comment:13 Changed 7 years ago by jaffakree

Renara: the trick is to rally support in the forums from normal users to show the devs that enough people would use the feature to justify spending their limited time implementing and maintaining the new feature.

comment:14 Changed 7 years ago by pathetic_loser

  • Resolution invalid deleted
  • Status changed from closed to reopened

I would use it as well.
Look, these days some files are several GiB or so! Even at very low failure rate of good hardware, when you'll have to deal with, say, 1010 bytes, many things can happen due to huge numbers of data to process. So I believe it is a very good idea to do a final pass when data are here. As for me, I'm usually forced to do such checks manually before burning linux ISOs, etc which is inconvenient enough. As for me, I would rather have some extra burden on my HDD rather than some extra burden on me. Most of other torrent clients would allow me such option. I think Transmission should do quite the same. While it's not a critical for videos, etc, it's absolute must have for sensitive data like ISO files, etc.

comment:15 Changed 7 years ago by livings124

  • Resolution set to wontfix
  • Status changed from reopened to closed

Again, the solution to this is to determine where the app is failing to make sure it always is 100%.

comment:16 Changed 7 years ago by Renara

  • Resolution wontfix deleted
  • Status changed from closed to reopened

livings124 you're completely missing the point; there is no such thing as 100% certainty, and never will be. No system is ever perfect, and even if the live verification were working perfectly (which it currently isn't) then you cannot guarantee that other unknown bugs, or future bugs caused unwittingly by other features or changes, will not reintroduce the same problem. To assume that any system is perfect is to invite disaster, that's just basic common sense.

This is why defensive features like automatic re-verification are a must, as while they may not be perfect either, they can at least give us a much greater degree of protection against errors. Besides which; I'd gladly help to fix the existing issue, but I haven't the faintest clue what may be causing it, or how to track it down. Even if it is fixed; when will that be exactly? It could be months before the current issue is found and resolved, during which time we have to suffer corrupted torrents. If a similar issue crops up in future then that's another several months of potentially corrupted blocks and having to manually re-verify everything just to be sure again.

The simple fact is this; the problem can never be solved such that it will work 100% of the time now, and forever. Which means that the burden falls to users to either manually re-verify everything they download, or take the risk of damaged files. For such a simple feature, Transmission can bear that burden for us, which is exactly what computer programs are supposed to do. Vuze is believed to have millions, possibly even tens of millions (though I'm dubious on that) of users, and it has automatic re-verification. While I don't agree with it being opt-out as opposed to opt-in, I do believe that it should be proof enough of the merit of such a simple additional layer of data-integrity.

comment:17 Changed 7 years ago by livings124

  • Resolution set to invalid
  • Status changed from reopened to closed

That's not true at all. Since there are mechanisms such as bad-block detection, this should not happen and verify on completion should not be needed. Saying that this is needed for unknown and random bugs is silly - those made-up bugs would need to be fixed instead of putting this in. Stop reopening tickets.

comment:18 Changed 7 years ago by Renara

  • Resolution invalid deleted
  • Status changed from closed to reopened

this should not happen

Yet it does! By all means, find the current cause of corrupt blocks and fix that; if I knew what it was I would happily post an issue asking for such a solution, but even if I did know the cause of the current issue, even if there weren't currently an issue with corrupted torrents, I would still want automatic re-verification.

And I would want for the simple reason that you cannot guarantee that the bug will not reappear in some form or another; or if you did you would be lying or incredibly foolish to do-so. Bad-block issues aside there are plenty of ways that data corruption can creep in on active downloads, such as uncleanly unmounted file-systems, application or system crashes; even hard-drive errors. There are a myriad of ways that downloading data can potentially become corrupt both inside and outside of Transmission itself, while completed data is relatively safe; it's just plain good practise to re-verify before declaring the download complete.

All that aside; fixing the bugs take time, and that's time during which people's torrents are becoming corrupt. Next time a bad-block bug occurs, that'll be another several weeks, or months, during which torrents are potentially being declared complete with corrupt blocks. Automatically re-verifying torrents is an ideal, and easy to implement, way to add an extra level of security to that data, so users can actually just continue using Transmission normally while the bugs are hunted down. Hell; if done right the feature could actually solve any such bugs in future because it could log, or even aggregate and report, the bad blocks that it finds that "should not" be there.

Stop reopening tickets.

Start actually reading them then, instead of repeating the "fix the bug instead" mantra; fixing the bug won't completely remove the problem, which is that Transmission has no kind of backup when it comes to verifying torrents prior to completion, a backup that is clearly needed.

comment:19 Changed 7 years ago by livings124

  • Resolution set to invalid
  • Status changed from reopened to closed

We're not adding features to get around imagined bugs instead of fixing actual bugs.

I am reading your posts, but they're just needlessly long. I've given you the concise answer. Do not reopen this - the decision has been made that this isn't being added.

comment:20 follow-up: Changed 7 years ago by x190

Renara: Couldn't this be accomplished by a script fired at completion? Maybe someone on the forum could help you.

comment:21 in reply to: ↑ 20 Changed 7 years ago by sergio

Replying to x190:

Renara: Couldn't this be accomplished by a script fired at completion? Maybe someone on the forum could help you.

#!/bin/bash

transmission-remote -t $TR_TORRENT_ID --verify

However, using the transmission-done script in this way is not really useful if you want to do anything meaningful besides verification because trorrent-verify over rpc never returns any status. You can either operate on the assumption that the torrent is complete or not, bot no "if complete then x else y".

comment:22 Changed 7 years ago by Renara

  • Resolution invalid deleted
  • Status changed from closed to reopened

Replying to livings124:

We're not adding features to get around imagined bugs instead of fixing actual bugs.

You claim that you are reading the posts, but you can't be if this is your stance on the issue; the bug is not imagined! How low do you intend to stoop if you're going to start to dismiss real bugs as being imaginary just so you can close an issue?

The current bug is real, and the possibility of future, similar bugs is ever-present; your unwillingness to see past the bloody obvious is only going to result in many more corrupt torrents in future. Unless you can tell me that the bug is going to be fixed tomorrow and there is a framework of debugging tools in Transmission that can make it easy to diagnose future issues, then re-verification is by far the best way to protect the integrity of torrents, as it would not only allow users to just keep using Transmission as normal, but can actually help to gather the information needed to solve the real issue. There is literally no good reason not to implement it except that you can't be arsed.

But to hell with it anyway, Transmission used to be miles ahead of other torrent programs for Mac, but if you're going to oppose every obvious feature to improve it, then there is just no reason to stick with Transmission anymore; uTorrent gets better and better, with far better feedback from mods/developers, real discussion of ideas, and actual progress all the time, meanwhile Transmission is stagnating, and it's a fucking tragedy that is entirely your own doing. I've wasted more than enough time now trying to get simple features in that would greatly improve Transmission; truth is you'd probably make better progress by abandoning the project and closing the trac system, as both seem to be superfluous.

comment:23 Changed 7 years ago by livings124

  • Resolution set to invalid
  • Status changed from reopened to closed

The goal is to fix the bug, not put in a hack around it. I am not going to repeat this anymore. Do not reopen this.

If you do not like Transmission because we are not implementing whatever you want, either implement the changes you want yourself, or use something else.

Last edited 7 years ago by livings124 (previous) (diff)

comment:24 Changed 7 years ago by cfpp2p

  • Priority changed from Normal to Lowest
  • Severity changed from Normal to Trivial

The problem with this ticket is not a bug but a general philosophical idea on what constitutes a bug and where the bug originates. Even a re-verify will not solve the idea of uncertainty in data transfer/storage since the re-verify routine itself could potentially be the problem, institute/be buggy or create an unknown/imaginary bug. So Renara's suggestions can never be accomplished since he himself describes the problem:

there is no such thing as 100% certainty, and never will be.

and since re-verify itself is subject to uncertainty...

this ticket is most certainly dead and invalid

comment:25 Changed 7 years ago by killemov

  • Priority changed from Lowest to Low
  • Resolution invalid deleted
  • Severity changed from Trivial to Normal
  • Status changed from closed to reopened
  • Version changed from 2.42 to 2.33

I have found at least one reproducible case for this. When a torrent download is halted because the target disk is full then that torrent is usually to be found corrupt after resuming download. Note that I too found some inexplicable cases of corruption. There are no signs of any hardware failure. Unrar has an additional way to verify a rar archive so I added --verify as a parameter to transmission-remote in my Unrar script when unrar fails.

My suggestion would be to check the data on disk for a block when it is completely downloaded.

Running transmission-daemon on Debian Squeeze.

comment:26 follow-up: Changed 7 years ago by jordan

  • Resolution set to invalid
  • Status changed from reopened to closed

If that situation is reproducible, please open a ticket with step-by-step instructions on how to reproducibly corrupt a torrent by downloading onto a disk that doesn't have enough free space for the torrent.

As stated in comment:1, IMO it's better to fix corruption bugs directly, than it is to add a bandaid preferences option that many people won't see.

comment:27 in reply to: ↑ 26 Changed 7 years ago by killemov

New ticket created.

The point with comment:1 is that corruption has to be detected manually. I firmly believe this ticket merits implementation even if it may be a case of "Better safe than sorry."

comment:28 Changed 7 years ago by JJR

Okay, there is an option to have completed torrents in one directory, uncompleted in another. But if a torrent is completed (but corrupt) it gets moved to the completed directory (because it is). Then you verify and it's not completed, it's 99.9%, but it doesn't move back to the completed directory.

comment:29 Changed 6 years ago by sizzlemctwizzle

For those who want something better than that simple script: transmission-verify-torrent

comment:30 Changed 6 years ago by stainger

  • Resolution invalid deleted
  • Status changed from closed to reopened

Please reconsider adding the option, I've just experienced the same problem that killemov had:

A torrent was automatically paused because the disk was full, I freed some space and then the torrent was restarted and finished 100%. After a while I checked the file and found out it to be corrupted!

comment:31 Changed 6 years ago by mudders

I'm relatively new to Linux (headless archlinux specifically) and so new to transmission. This has happened to every torrent ive downloaded since installing 3 days ago. I have to verify each one 2 or 3 time before hitting 100%.

Im not saying auto verification would be a great thing to do, but it would be a handy workaround until i can figure out what is causing it.

maybe its because of the samba share hanging off my wdtvlive that i'm connecting to via cifs and downloading to.

comment:32 Changed 6 years ago by dicktyr

As an otherwise happy user of Transmission, I was shocked to discover that some "completed" torrents were incomplete/corrupt. I wish I could file a report for a reproducible bug but the problem is sporadic (occurring perhaps one in 40-50 torrents). I'm also disappointed to see this is still occurring with 2.77.

Clearly, I am not the only one experiencing this problem, and I'm writing now to say please take this seriously. After all, Transmission does not communicate directly with the storage device, and things may go wrong in between.

Even in the absence of any bugs in Transmission, verification after completion is critical for those who need to be sure, and such an option would not be superfluous.

comment:33 Changed 6 years ago by levrin

From a practical standpoint, while as mentioned above there is no way of providing 100% confidence that the data is correct, it does seem reasonable to at least minimize the potential for data corruption via hardware failure for those of us who aren't running ZFS on ECC RAM. A bad sector failing on write may well be transparently reallocated, but one failing on read after a successful write can't be because the drive doesn't know what data should go there.

comment:34 Changed 6 years ago by superwookee

  • Priority changed from Low to Normal
  • Severity changed from Normal to Major
  • Version changed from 2.33 to 2.77+

This is ridiculous and makes the open source community look childish. livings124 appears to be a lead or moderator of some sort but comes across as inept and easily offended.

I've used seven different clients across three platforms and never had a problem with completed torrents being corrupt until Transmission. The only reason I gave it a chance was because it was the only easily installable client on freenas, but I have been sorely disappointed with its performance.

Now while searching for a solution I find this?

Torrents being corrupted should be a huge issue. It doesn't work correctly. It's a bug. There solved your problem with the definition of is . . . I mean bug.

Fix the bug. That's what this forum is for right? Users to report bugs?

I'm using 2.77 on freenas running on ZFS and it is still an issue. I see no report that newer versions fix this bug but I am willing to upgrade if someone can verify that this has been addressed.

Otherwise I'll set up a linux jail and run a torrent client that actually performs it's primary function.

comment:35 Changed 6 years ago by livings124

  • Priority changed from Normal to Low
  • Severity changed from Major to Normal
  • Version changed from 2.77+ to 2.33

comment:36 Changed 6 years ago by Renara

In fairness, this issue isn't about fixing any particular corruption bug(s), but then that's kind of the point; these bugs can be very difficult to track down as there simply isn't enough information available as to what is causing them. It's for this reason that I proposed this feature; although I'd of course prefer that corrupt blocks wasn't an issue in the first place, the fact is that they can (and do) happen, so auto (re)verification is a desirable feature so that Transmission users can just carry on using Transmission without having to worry about whether a torrent is corrupt or not.

Yes any bug(s) should be fixed, but the majority of Transmission users simply don't have the information available to find what is causing the bugs, nor do they have the time to wait for a fix as by that point their torrent(s) are already corrupt. This is why auto verification on completion is a good compromise, and is present in many other clients, as it makes it less likely that a torrent file will complete in a damaged state. Fact is it's unlikely that Transmission (or other clients) can ever truly guarantee that the file(s) are error free, either due to bugs in the client itself, or some form of external issue, so verifying is just plain good practice, especially on larger, longer running downloads.

Even more annoying though is that I wouldn't even have to ask for the feature if all versions of Transmission actually came with transmission-remote, as with that tool it's fairly easy to just use a script to auto-verify, but that's not possible on OS X (and other platforms?) without having to keep compiling Transmission from source, which is hardly user friendly. Even if transmission-remote were bundled I still think I'd prefer to see this added as a simple checkbox, as it makes it much easier for users to enable if they want to.

comment:37 follow-up: Changed 6 years ago by eldonmcguinness

I just wanted to add that I also see this issue, especially with larger torrents in the GB range. I did try to mitigate the occurrence with a post finish script but that has its own issues as well which make it unusable. I would love to see this be a simple bool option in the settings.json that could be toggled by the user.

comment:38 in reply to: ↑ 37 ; follow-up: Changed 6 years ago by sizzlemctwizzle

Replying to eldonmcguinness:

I just wanted to add that I also see this issue, especially with larger torrents in the GB range. I did try to mitigate the occurrence with a post finish script but that has its own issues as well which make it unusable. I would love to see this be a simple bool option in the settings.json that could be toggled by the user.

Have you tried the app I wrote in C for this: transmission-verify-torrent

I had similar issues (specifically large files) myself, but since I've been using this I haven't come across a single corrupt torrent yet.

comment:39 in reply to: ↑ 38 ; follow-up: Changed 6 years ago by eldonmcguinness

Replying to sizzlemctwizzle:

Replying to eldonmcguinness:

I just wanted to add that I also see this issue, especially with larger torrents in the GB range. I did try to mitigate the occurrence with a post finish script but that has its own issues as well which make it unusable. I would love to see this be a simple bool option in the settings.json that could be toggled by the user.

Have you tried the app I wrote in C for this: transmission-verify-torrent

I had similar issues (specifically large files) myself, but since I've been using this I haven't come across a single corrupt torrent yet.

Apparently the error I am getting with the RPC is part of another bug that is slated to be fixed in 2.90.

Forum Ticket

Cheers for the app, but really a bash script should suffice :-)

Changed 6 years ago by cfpp2p

comment:40 Changed 6 years ago by x190

---

Last edited 6 years ago by x190 (previous) (diff)

comment:41 follow-up: Changed 6 years ago by cfpp2p

patch adds settings.json element "reverify-torrents-tries" which if greater than zero will reverify on completion until successful 100% verify or reverify-torrents-tries times at which time torrent is paused with error 'Reverify count exceeded - pausing torrent -- resetting retries'. After this error message restarting the torrent then again starts with original reverify-torrents-tries count. Default is 0, edit settings.json "reverify-torrents-tries": 0, to greater than 0 to enable reverify.

reverify4649.patch additionally needs:

--- libtransmission/quark.h.rev   2013-10-19 21:59:37.908980224 -0400
+++ libtransmission/quark.h   2013-10-19 22:00:22.002316492 -0400
@@ -287,6 +287,7 @@ enum
   TR_KEY_rename_partial_files,
   TR_KEY_reqq,
   TR_KEY_result,
+  TR_KEY_reverify_torrents,
   TR_KEY_rpc_authentication_required,
   TR_KEY_rpc_bind_address,
   TR_KEY_rpc_enabled,

thanks Vallimar ! https://forum.transmissionbt.com/viewtopic.php?f=1&t=15144#p66309

x

Last edited 4 years ago by cfpp2p (previous) (diff)

comment:42 in reply to: ↑ 41 Changed 6 years ago by mike.dld

Torrent completion script is still being run each time Transmission thinks it has been completed while it could theoretically be not, as revealed afterwards when verification completes.

comment:43 Changed 6 years ago by cfpp2p

reverify4649.patch additionally needs: https://forum.transmissionbt.com/viewtopic.php?f=1&t=15144#p66309

--- libtransmission/quark.h.rev   2013-10-19 21:59:37.908980224 -0400
+++ libtransmission/quark.h   2013-10-19 22:00:22.002316492 -0400
@@ -287,6 +287,7 @@ enum
   TR_KEY_rename_partial_files,
   TR_KEY_reqq,
   TR_KEY_result,
+  TR_KEY_reverify_torrents,
   TR_KEY_rpc_authentication_required,
   TR_KEY_rpc_bind_address,
   TR_KEY_rpc_enabled,

thanks Vallimar !

comment:44 Changed 6 years ago by vallimar

I just wanted to add a +1 vote to this. For several versions I've encountered very sporadic issues of torrents saying they completed only to later find they aren't. I've gotten in the habit of manually re-verifying everything and I still find issues where it will come up at 99% after and then a re-start will do an actual complete.

The patch by cfpp2p is minimal and non-invasive and seems to be working for me. It's a great feature for paranoid people and does nothing to impede others or detract from other options so I don't understand why there would be resistance against it.

comment:45 Changed 5 years ago by jager

Logging in to +1 also. I built a freenas system with ZFS/ECC specifically to avoid this sort of thing, I'm pretty sure it's not the hardware. I'm going to try ktorrent in a linux jail, which is what I use on the desktop, and one of the big reasons I do is because it has an auto verify feature.

comment:46 Changed 5 years ago by SandJ

I am reasonably sure I have had files downloaded with Transmission which, when I have manually verified them, have had additional blocks downloaded which 'overlay' whole sections of 0x00 content, despite them already being marked as 100%.

Automatic verification would give me confidence that Transmission could be trusted. (And, if it wrote to the log file whether it did make changes, that would be a great piece of evidence either way.)

Alternatively, instead of showing "100%" show "?% - needs verification".

comment:47 Changed 5 years ago by cfpp2p

Automatic verification would give me confidence that Transmission could be trusted. (And, if it wrote to the log file whether it did make changes, that would be a great piece of evidence either way.)

reverify4649.patch plus comment:41 comment:43 gives error upon reverify-torrents-tries exceeded and the log file shows the normal torrent verification statuses.

THIS IS NOT AN EDIT BUT A POST THAT BELONGS AT NEW but the recent Trac spam bot bug has become severely pronounced such that I can no longer post new comments or tickets. *

ticket #5912 created 04/03/15

March 31, 2015

I can verify that there is indeed a bug in webseed.c where corrupt blocks overwrite previously deemed verified OK block of a piece. I have test torrents that will consistently reproduce verified corrupt blocks after 100% completion, and then a state of less than 100%. The fix is having tr_torrentPieceIsComplete() test before the two instances of tr_cacheWriteBlock() in webseed.c

This fault was introduced with r12539.

something like this will do it

static void
write_block_func (void * vdata)
{
  struct write_block_data * data = vdata;
  struct tr_webseed * w = data->webseed;
  struct evbuffer * buf = data->content;
  struct tr_torrent * tor;

  tor = tr_torrentFindFromId (data->session, data->torrent_id);
  if (tor != NULL)
    {
      const uint32_t block_size = tor->blockSize;
      uint32_t len = evbuffer_get_length (buf);
      const uint32_t offset_end = data->block_offset + len;
      tr_cache * cache = data->session->cache;
      const tr_piece_index_t piece = data->piece_index;
     
      if (!tr_torrentPieceIsComplete (tor, piece))
   {
     while (len > 0)
       {
         const uint32_t bytes_this_pass = MIN (len, block_size);
         tr_cacheWriteBlock (cache, tor, piece, offset_end - len, bytes_this_pass, buf);
         len -= bytes_this_pass;
       }
      
     tr_bitfieldAdd (&w->blame, piece);
     fire_client_got_blocks (tor, w, data->block_index, data->count);
   }
    }

  evbuffer_free (buf);
  tr_free (data);
}
static void
web_response_func (tr_session    * session,
         char       * task_addr,
         bool         aborted,
                   bool            did_connect UNUSED,
                   bool            did_timeout,
                   long            response_code,
                   const void    * response UNUSED,
                   size_t          response_byte_count UNUSED,
                   void          * vtask)
{
...

           else
            {
              if (buf_len && !tr_torrentPieceIsComplete (tor, t->piece_index))
                {
                  /* on_content_changed () will not write a block if it is smaller than
                     the torrent's block size, i.e. the torrent's very last block */
                  tr_cacheWriteBlock (session->cache, tor,
                                      t->piece_index, t->piece_offset + bytes_done,
                                      buf_len, t->content);
        tr_bitfieldAdd (&w->blame, t->piece_index);
                  fire_client_got_blocks (tor, t->webseed,
                                          t->block + t->blocks_done, 1);
                }

Last edited 4 years ago by cfpp2p (previous) (diff)

comment:48 Changed 5 years ago by ravenexp

  • Version changed from 2.33 to 2.82

I too have this problem with torrents reported complete, when in fact they aren't and it's throwing sporadic 'Corrupt piece' errors when seeding. Verifying such torrents manually reveals that they are most often 70-99% complete. So as any new transmission user I'm getting into habit of manually verifying every torrent I download (I used rtorrent before which did it automatically). Verification on completion would be much appreciated.

My setup is: a single btrfs partition on an SSD for both torrent downloads and the system.No read checksum errors were ever reported and I ran btrfs scrub several times on the whole disk with no problems detected. I've also gone through several major system updates without anything falling apart. I run a source-based distro, so everything I install is also compiled on my box, which I believe tests my hardware more than enough. Running memtest86 for hours also detected no problems.

I'm planning to try and disable 'Move files to another folder after completion feature' to see if it makes any difference.

comment:49 Changed 5 years ago by livings124

  • Version changed from 2.82 to 2.33

comment:50 Changed 5 years ago by eldonmcguinness

I just wanted to add that the reverify option on post completion is a viable option as of 2.83. Ticket #5352 was addressed and you are now able to use RPC to do this without an issue.

*NOTE This comes from a much larger script, my apologies if I forgot something, but this should be more than enough to fix the verification issue.

Cheers all!

#!/bin/bash

# Change this if you wish to log the verification
LOG_FILE=/dev/null

# Settings needed
# RPC IP ADDRESS
TC_ADDR = "127.0.0.1"

# RPC PORT NUMBER
TC_PORT = "48000"

# Transmission Username
TC_USER = "trans_user"

# Transmission Password
TC_PASS = "trans_pass"

renice -n 1 $$

function getTorrentStatus {
    status=$( transmission-remote ${TC_ADDR}:${TC_PORT} -n ${TC_USER}:${TC_PASS} -t ${TR_TORRENT_HASH} -i | grep State | sed 's/State: \(.\+\)/\1/' | sed 's/[ |\t]*\(.\+\) ([0-9]\+%)/\1/' )
    echo "${TR_TORRENT_NAME} Status: $status" >> $LOG_FILE
    echo "$status"
}

function verifyTorrent {
        echo "${TR_TORRENT_NAME} Verification Started" >> $LOG_FILE
        transmission-remote  ${TC_ADDR}:${TC_PORT} -n ${TC_USER}:${TC_PASS} -t ${TR_TORRENT_HASH} -v
}

function startTorrent {
        echo "${TR_TORRENT_NAME} Re-Starting" >> $LOG_FILE
        transmission-remote  ${TC_ADDR}:${TC_PORT} -n ${TC_USER}:${TC_PASS} -t ${TR_TORRENT_HASH} -s
}

function getTorrentPercentage {
        percentage=$( transmission-remote ${TC_ADDR}:${TC_PORT} -n ${TC_USER}:${TC_PASS} -t ${TR_TORRENT_HASH} -i | grep 'Percent Done' | sed 's/Percent Done: \(.\+\)/\1/' | sed 's/[ |\t]*\([0-9]\+\)%/\1/' )
        echo "${TR_TORRENT_NAME} PERC: $percentage" >> $LOG_FILE
        echo "$percentage"
}

echo "${TR_TORRENT_NAME} Processing" >> $LOG_FILE
# Initiate the torrent validation
sleep 10
verifyTorrent

# Get the current torrent status
sleep 10
verificationStatus=$( getTorrentStatus )

while [ "$verificationStatus" == "Will Verify" ]; do
# Sleep to give the torrent a moment to cool down
  
  sleep $(($RANDOM%60 + 61))
  verificationStatus=$( getTorrentStatus )
done

# Keep checking the torrent status every 10 seconds
while [ "$verificationStatus" == "Verifying" ]; do
# Sleep to give the torrent a moment to cool down
  sleep 10
  verificationStatus=$( getTorrentStatus )
done

# Ensure the torrent is started
sleep 10
startTorrent

# Check to see if we have 100% of the torrent still
# if not then we need to exit the app
sleep 10
if [ "$( getTorrentPercentage )" != "100" ] && [ "$( verificationStatus )" != "Seeding" ]; then
  echo "${TR_TORRENT_NAME} Verification Failed [${getTorrentPercentage}|${verificationStatus}]" >> $LOG_FILE
  exit
fi

echo "${TR_TORRENT_NAME} Verification Passed [${getTorrentPercentage}|${verificationStatus}]" >> $LOG_FILE
Last edited 5 years ago by eldonmcguinness (previous) (diff)

comment:51 Changed 4 years ago by cfpp2p

comment:52 Changed 4 years ago by Ovidiu

@eldonmcguinness

would you mind explaining your script? I'm not very good at scripting but trying to figure out in general terms what the script does if it finds a "broken" torrent and what it does if the verification went well?

comment:53 Changed 4 years ago by eldonmcguinness

@Ovidu, that is up to you! :D I intentionally left the bottom bit without definition so the end-user can decide what they would like to do and fill it in. In the case the the torrent is detected as *broken*, a previous part of the code tries to start the torrent again so it can re-grab any broken pieces.

If you notice this codeblock:

# Check to see if we have 100% of the torrent still
# if not then we need to exit the app
sleep 10
if [ "$( getTorrentPercentage )" != "100" ] && [ "$( verificationStatus )" != "Seeding" ]; then
  echo "${TR_TORRENT_NAME} Verification Failed [${getTorrentPercentage}|${verificationStatus}]" >> $LOG_FILE
  exit
fi

echo "${TR_TORRENT_NAME} Verification Passed [${getTorrentPercentage}|${verificationStatus}]" >> $LOG_FILE

You can do any other processing you would like to do in this block.

comment:54 Changed 3 years ago by pastebincome

  • Priority changed from Low to Highest
  • Severity changed from Normal to Major

how can i use this script with utorrent 2.2.1, id appreciate it

comment:55 Changed 3 years ago by mike.dld

  • Priority changed from Highest to Low
  • Severity changed from Major to Normal

comment:56 in reply to: ↑ 39 Changed 3 years ago by sizzlemctwizzle

Replying to eldonmcguinness:

Replying to sizzlemctwizzle:

Replying to eldonmcguinness:

I just wanted to add that I also see this issue, especially with larger torrents in the GB range. I did try to mitigate the occurrence with a post finish script but that has its own issues as well which make it unusable. I would love to see this be a simple bool option in the settings.json that could be toggled by the user.

Have you tried the app I wrote in C for this: transmission-verify-torrent

I had similar issues (specifically large files) myself, but since I've been using this I haven't come across a single corrupt torrent yet.

Apparently the error I am getting with the RPC is part of another bug that is slated to be fixed in 2.90.

Forum Ticket

Cheers for the app, but really a bash script should suffice :-)

I wrote it in C for reliability/precision, and so I could abstract it all away from the rest of the logic within my own Bash completion script. It provides the behavior

transmission-remote -t $TR_TORRENT_ID --verify

should have. Of course the need for compiling is an obvious downside, but if you already have all the build tools setup on you computer it's dead simple.

comment:57 Changed 3 years ago by tuxolero

I'd also like to vote for an option to do this internally in transmission.

Possible reasons for corrupted blocks are many. On any long wire, bits can tip. Also during the writing process to the harddisk and so on ... this does not necessarily mean that the hardware is defective and needs replacement.

As a user who downloads many old torrents, I sometimes need to wait for months until a seeder appears. In these cases, it is important to auto-verify the downloaded data directly after completion. Otherwise it happens regularly that I notice the progress and the corrupted blocks on the next day, when the seeder is unavailable again. With auto-verify, it would have been possible to re-request the corrupted blocks during the seeder was online. Now, I have to wait another several months. :-(

comment:58 Changed 3 years ago by mike.dld

Closed #6154 as duplicate of this ticket.

comment:59 Changed 3 years ago by Tomato

KTorrent client (Linux based, KDE) has an auto-verify option for each finished torrent. If something wrong is found, it downloads automatically the corrupt pieces and then it verifies again until the torrent finishes properly.

This is a very useful feature that must be on Transmission.

Last edited 3 years ago by Tomato (previous) (diff)
Note: See TracTickets for help on using tickets.