Opened 9 years ago

Closed 9 years ago

Last modified 9 years ago

#4644 closed Bug (fixed)

Transmission can't download big files from webseed.

Reported by: Altimit Owned by: jordan
Priority: Normal Milestone: 2.50
Component: libtransmission Version: 2.42
Severity: Normal Keywords: backport-2.4x webseed
Cc:

Description

While downloading torrent with size greater than 4 Gb from webseed I get a huge amount of "Piece ####, which was just downloaded, failed its checksum test (peer-mgr.c:1791)" errors in log file. There are different pieces but all of them have ID greater than 1024 (torrent has 4Mb piece size). In the same time webserver gets query from transmission but the requested byte range is wrong. Requested byte offset is 4294967296 less than it's supposed to be for the piece. It seems like there is a 32bit variable in transmission that is overflowed.

Attachments (4)

test.torrent (39.3 KB) - added by Altimit 9 years ago.
transmission-log.txt (16.5 KB) - added by cfpp2p 9 years ago.
transmission.test.log (43.1 KB) - added by Altimit 9 years ago.
transmission_get.pcap (24.0 KB) - added by Altimit 9 years ago.

Download all attachments as: .zip

Change History (24)

comment:1 Changed 9 years ago by jordan

do you have a .torrent that has a > 4 GB webseed that I can test with?

Changed 9 years ago by Altimit

Changed 9 years ago by cfpp2p

comment:2 Changed 9 years ago by cfpp2p

verified: Transmission can't download big files from webseed

tried 5.0 GB torrent 4194304 piece size from webseed: many 'failed its checksum test'

same with 5.0 GB torrent 8388608 piece size

same with 3.0 GB, 3.9 GB torrents

1.0 and 1.99 GB torrents downloaded from webseed perfectly, ZERO hash check fails

comment:3 Changed 9 years ago by cfpp2p

addendum--

actually 3.0 & 3.9 GB webseed torrents actually didn't show 'failed its checksum test' errors but the downloads just seem to stall after a very short time../

Couldn't read "/root/.config/transmission-daemon/resume/test39gb.a7819defbfcc2ce7.resume": No such file or directory (utils.c:443)

test39gb Couldn't read "/root/.config/transmission-daemon/resume/test39gb.a7819defbfcc2ce7.resume" (resume.c:686) Saved "/root/.config/transmission-daemon/torrents/test39gb.a7819defbfcc2ce7.torrent" (bencode.c:1721) test39gb Queued for verification (verify.c:260) test39gb Verifying torrent (verify.c:218) test39gb verifying torrent... (verify.c:61) test39gb Verification is done. It took 0 seconds to verify 4193255424 bytes (4193255424 bytes per second) (verify.c:155)

then nothing more?????

comment:4 Changed 9 years ago by x190

worksforme with Altimit's "test.torrent" (Sorry, that's with 2.21+).

From log:
2011-11-30 22:49:02 -0700 verify.c:170 [Debug] test: Verification is done. It took 0 seconds to verify 8388608000 bytes (8388608000 bytes per second) (This is total size not piece size).
2011-11-30 22:55:20 -0700 peer-mgr.c:1530 [Error] test: Piece 1999, which was just downloaded, failed its checksum test

How do we know your server isn't sending corrupt pieces? (Encoding issues perhaps?) [Ïðèâåò. Ñïàñèáî, ÷òî çàãëÿíóëè, íî òóò íè÷åãî íåò. À âû ÷òî îæèäàëè óâèäåòü? Î.î = Привет. Спасибо, что заглянули, но тут ничего нет. А вы что ожидали увидеть? О.о = Regards. Thanks, that they glanced, but here there is nothing. But you that did expect to see? [O].[o] :-)]

cfpp2p: I see nothing unusual in your logs. Transmission can't force peers/webseeds to connect or stop corrupt pieces from being sent.

I am currently d/ling a 34 GB webseeded torrent with no issues. (Only 4 bad pieces out of 5 GB so far).

Last edited 9 years ago by x190 (previous) (diff)

Changed 9 years ago by Altimit

Changed 9 years ago by Altimit

comment:5 Changed 9 years ago by Altimit

How do we know your server isn't sending corrupt pieces

Just believe me it isn't :)

Encoding issues perhaps?

Nope. It's just "welcome page". It's .txt file with russian text in CP1251 charset. If it does matter it means "Hi there. Thanks for yout visit but there is nothing here. What did you expect to see? O.o". But your browser thinks that it's in UTF-8. That's all. Web server dosen't change original file charset. Besides the test file you are downloading is binary.

I am currently d/ling a 34 GB webseeded torrent with no issues. (Only 4 bad pieces out of 5 GB so far)

Are there any non-webseed peers seeding this torrent? Only pieces with offset greater than 4294967296 bytes asked from webseed gives you 'failed its checksum test' error. So in order to repeat the issue you should download a torrent containing 4Gb+ files (not a 4Gb+ multifile torrent with small files) w/o or with a small amount of non-webseed peers.

comment:6 Changed 9 years ago by Altimit

Here are transmission log file and traffic dump file (HTTP GET requests only). As you can see, when transmission writes in log: "test Piece 1632, which was just downloaded, failed its checksum test (peer-mgr.c:1791)" the real HTTP request is "Range: bytes=2550136832-2554331135\r\n". 2550136832/4*(1024*1024)=608th piece. Also you can notice that 608=1632-1024.

comment:7 Changed 9 years ago by cfpp2p

I really don't believe corrupt pieces are being sent by my servers as uTorrent downloaded all of my tests perfectly and with the 1.9 GB webseed torrent that I tested and downloaded 100% via transmission there were ZERO 'failed its checksum' errors. You've got to have zero non-webseed peers for the checksum errors to appear. There is not a problem with large files as long as there are some non-webseed peers. I put a lot of time into testing this and I sincerely believe that there is a problem with large files from 100% webseed...

comment:8 Changed 9 years ago by Altimit

There is not a problem with large files as long as there are some non-webseed peers.

If there are some non-webseed peers there is still a chance to have this problem. When transmission asks webseed for the piece with offseat greater than 232 bytes it tries downloading it again and again. Transmission has limited amount of concurrent connections (number of pieces it's downloading in one time per torrent), e.g. 10. If 5 of these connection would stuck trying downloading pieces from webseed and the other connections would succeed downloading the rest of the file then we'll enter the "endgame mode" and the 5 "unlucky" pieces would be downloaded from non-webseed other peers. But if all of 10 connections by chance will stuck downloading from webseed we'll never have the file downloaded. So the less non-webseed peers we have and the more pieces with offset greater than 232 bit are in the torrent the greater is the chance to run into this problem.

comment:9 Changed 9 years ago by cfpp2p

If there are some non-webseed peers there is still a chance to have this problem. When transmission asks webseed for the piece with offseat greater than 232 bytes it tries downloading...

noted, and agree

comment:10 Changed 9 years ago by cfpp2p

[edit correction (232 --> 232)]

...with offset greater than 232 bytes it...

comment:11 Changed 9 years ago by and_cesbo

Patch to fix this bug

--- libtransmission/webseed.c	(revision 13103)
+++ libtransmission/webseed.c	(working copy)
@@ -481,19 +481,21 @@
         char ** urls = t->webseed->file_urls;
 
         const tr_info * inf = tr_torrentInfo( tor );
-        const uint32_t remain = t->length - t->blocks_done * tor->blockSize
+        const uint64_t remain = t->length - t->blocks_done * tor->blockSize
                                 - evbuffer_get_length( t->content );
 
-        const uint64_t total_offset = inf->pieceSize * t->piece_index
-                                    + t->piece_offset + t->length - remain;
+        uint64_t total_offset = inf->pieceSize;
+        total_offset *= t->piece_index;
+        total_offset += (t->piece_offset + t->length - remain);
+
         const tr_piece_index_t step_piece = total_offset / inf->pieceSize;
-        const uint32_t step_piece_offset
+        const uint64_t step_piece_offset
                                = total_offset - ( inf->pieceSize * step_piece );
 
         tr_file_index_t file_index;
         uint64_t file_offset;
         const tr_file * file;
-        uint32_t this_pass;
+        uint64_t this_pass;
 
         tr_ioFindFileLocation( tor, step_piece, step_piece_offset,
                                     &file_index, &file_offset );

comment:12 Changed 9 years ago by jordan

  • Milestone changed from None Set to 2.50
  • Status changed from new to assigned

comment:13 Changed 9 years ago by jordan

and_cesbo: thanks for the patch!

There's already some simple code for the total_offset calculation that we might reuse here. What do you think of:

Index: libtransmission/webseed.c
===================================================================
--- libtransmission/webseed.c	(revision 13103)
+++ libtransmission/webseed.c	(working copy)
@@ -481,19 +481,19 @@
         char ** urls = t->webseed->file_urls;
 
         const tr_info * inf = tr_torrentInfo( tor );
-        const uint32_t remain = t->length - t->blocks_done * tor->blockSize
+        const uint64_t remain = t->length - t->blocks_done * tor->blockSize
                                 - evbuffer_get_length( t->content );
 
-        const uint64_t total_offset = inf->pieceSize * t->piece_index
-                                    + t->piece_offset + t->length - remain;
+        const uint64_t total_offset = tr_pieceOffset( tor, t->piece_index,
+                                      t->piece_offset, t->length - remain );
+
         const tr_piece_index_t step_piece = total_offset / inf->pieceSize;
-        const uint32_t step_piece_offset
+        const uint64_t step_piece_offset
                                = total_offset - ( inf->pieceSize * step_piece );
 
         tr_file_index_t file_index;
         uint64_t file_offset;
         const tr_file * file;
-        uint32_t this_pass;
+        uint64_t this_pass;
 
         tr_ioFindFileLocation( tor, step_piece, step_piece_offset,
                                     &file_index, &file_offset );

comment:14 Changed 9 years ago by and_cesbo

it's better solution :)

comment:15 Changed 9 years ago by and_cesbo

patch for patch:

in your patch need set position of the target file to

@@ -481,19 +481,20 @@

or remove empty line after total_offset initialization

comment:16 Changed 9 years ago by cfpp2p

??

webseed.c

36       uint32_t             piece_offset;
37       uint32_t             length;

??

in

+        const uint64_t total_offset = tr_pieceOffset( tor, t->piece_index,
+                                      t->piece_offset, t->length - remain );

or

+        total_offset += (t->piece_offset + t->length - remain);

comment:17 Changed 9 years ago by jordan

  • Keywords backport-2.4x added
  • Resolution set to fixed
  • Status changed from assigned to closed

Patch added in r13105; closing ticket as fixed. Please reopen the ticket if the problem persists. Thanks!

comment:18 Changed 9 years ago by cfpp2p

tested and no problems, fixed!

comment:19 Changed 9 years ago by cfpp2p

note that pause/stop of webseed torrent can be slow as it seems a wait until each currently incomplete block is complete before so. This is also related to #1079. The bigger the piece size the longer it may take, especially if there are a few incomplete pieces at once.

comment:20 Changed 9 years ago by x190

It ain't fixed until everybody's using the updated code. I think this is messing up real torrents in the wild. How about a 2.50 release already!!!!!

Note: See TracTickets for help on using tickets.