Opened 12 years ago

Closed 12 years ago

Last modified 11 years ago

#3231 closed Enhancement (fixed)

"Verify Local Data" can be faster

Reported by: eridius Owned by: charles
Priority: Normal Milestone: 2.00
Component: libtransmission Version: 1.93
Severity: Normal Keywords: performance


Checking existing data on a large file is incredibly slow. Activity Monitor is reporting that Transmission is reading between 2 and 4MBps from my media drive, when it is capable of nearly 20MBps. Transmission is using roughly 4% CPU during this time, and the relevant thread is just doing preads and SHA1 hashes:

    2723 Thread_167599
      2723 thread_start
        2723 _pthread_start
          2723 0x10006ae57
            2301 0x10009023a
              2301 pread
            372 0x100090311
              372 usleep
                372 nanosleep
                  372 __semwait_signal
            46 0x100090255
              46 SHA1_Update
                46 sha1_block_data_order
            4 0x100090250

sudo iosnoop shows Transmission is doing a lot of sequential reads of 20k blocks from the disk. I'm guessing this is the culprit right here. It should be reading far more data at once. As a test, copying a 1.4GB file off of this drive using the Finder does reads of 1 and 2MB at a time, and this allows for a much higher throughput (roughly 18MBps).

Change History (8)

comment:1 Changed 12 years ago by charles

  • Milestone changed from None Set to 2.00
  • Owner set to charles
  • Status changed from new to assigned
  • Type changed from Bug to Enhancement

Well I didn't think there would be that much of a boost, but increasing the buffer size really does improve things. I've spent awhile this afternoon testing verifies with various buffer sizes, and at least in my testing the "sweet spot" seems to be a 128 KiB buffer. Going past that resulted in almost no improvements.... but growing from the current buffer size to 128 KiB had a distinct improvement.

I'm going to milestone this for 2.00 despite the freeze because the change is extremely small (changing the size of the buffer we alloc from 16 KiB to 128 KiB) and because the improvements were so distinct.

Last edited 11 years ago by charles (previous) (diff)

comment:2 Changed 12 years ago by charles

Implemented in trunk for 2.00 by r10689.

Just to expand on the last message -- I tested by doing a "verify local data" on a Fedora 13 iso. Growing the buffer from 16 KiB to 128 KiB cut the time from an average of 130 seconds to 62 seconds... less than half the time. :)

Growing the buffer from 128 KiB to 2 MiB brought it down to 59 seconds, which IMO is not enough of an improvement over 62 seconds to make the extra memory allocation worthwhile.

Great suggestion though. Thanks for the ticket :)

comment:3 Changed 12 years ago by charles

  • Resolution set to fixed
  • Status changed from assigned to closed

comment:4 Changed 12 years ago by eridius

Wow, talk about fast response time! It looks like r10689 is actually using a 1MiB buffer though, the 128KiB version is commented out.

comment:5 Changed 12 years ago by charles


Thanks for that. :) Fixed now in r10690

comment:6 Changed 11 years ago by charles

  • Keywords performance added

comment:7 Changed 11 years ago by charles

  • Component changed from Transmission to libtransmission

comment:8 Changed 11 years ago by charles

  • Summary changed from Check existing data is extremely slow to "Verify Local Data" can be faster
Note: See TracTickets for help on using tickets.