Opened 12 years ago

Closed 12 years ago

Last modified 12 years ago

#2196 closed Enhancement (fixed)

Special preallocation for small headless devices - to prevent fragmentation

Reported by: Iznakurnog Owned by: charles
Priority: Normal Milestone: 1.72
Component: libtransmission Version: 1.71
Severity: Normal Keywords: Preallocate, posix_fallocate, fragmentation
Cc:

Description

That is continuation of closed ticket http://trac.transmissionbt.com/ticket/849

Charles I quite understand your last answer in former ticket. It is best choice to have fallocation() and posix_fallocation() well configured ... but there are plenty of small headless devices (Media players, NASes, routers and so on) where impossible to have it done. For example my NAS DNS-323 uses Linux 2.6.12.6, while fallocate() was added to the kernel in version 2.6.23. we cannot change kernel without breaking firmware and I don't see any way of somehow adding the functionality manually. Even newest firmware goes with that same old kernel ...

Why cant we introduce small easy and painless patch which could make relaxation and smooth life for all owners of such devices ?

I take the patch 1.diff from previous ticket and little bit adopt it for 1.71. Built experimental Transmission and see what result is excellent ! No more fragmentation for me since the moment. I'm going to distribute my special build in our DNS-323 comunity for proper testing

I am not developer of the patch and I believe author - hdfan2 help us support this request if necessary. As well i'm not much familar with development process and diff/patch utilities. so I attach here just three putched files for start - fdlimit.c, transmission.h, session.h

Attachments (3)

Patched_files_Transmission1.71.zip (20.7 KB) - added by Iznakurnog 12 years ago.
fill_correct.diff (2.9 KB) - added by hdfan2 12 years ago.
Reviewed the patch and noticed that it has one change that wasn't in my original path. Attachet the correct version of the patch (couldn't replace existing - no priveleges)
preallocate.diff (726 bytes) - added by charles 12 years ago.
example code for doing the patch without adding a new preallocation mode

Download all attachments as: .zip

Change History (17)

Changed 12 years ago by Iznakurnog

comment:1 Changed 12 years ago by Iznakurnog

Experimental transmission build for testing the patch on DNS-323 with ffp-05 http://slil.ru/27748402

comment:2 Changed 12 years ago by charles

  • Severity changed from Major to Normal

Changed 12 years ago by hdfan2

Reviewed the patch and noticed that it has one change that wasn't in my original path. Attachet the correct version of the patch (couldn't replace existing - no priveleges)

comment:3 Changed 12 years ago by Iznakurnog

hdfan2 do you mean fdlimit.c

613: if( ( s >= 0 ) && gFd->socketCount > getSocketMax( gFd ) ) 
changes to 
627: if( ( s >= 0 ) && gFd->socketCount < getSocketMax( gFd ) )

in fill.diff

I don't know where it came from. Both in original stable 1.71 sources and in my attachment already is

if( ( s >= 0 ) && gFd->socketCount < getSocketMax( gFd ) )

So your diff seemed to be right BTW going on with testing ... Working excellent for me !

comment:4 follow-ups: Changed 12 years ago by charles

Why do we need a new preallocation type? What if we were to just add another "if( !success )" test that contained the write-in-a-loop code?

Changed 12 years ago by charles

example code for doing the patch without adding a new preallocation mode

comment:5 in reply to: ↑ 4 Changed 12 years ago by Iznakurnog

  • Keywords fragmentation added

Replying to charles:

Why do we need a new preallocation type? What if we were to just add another "if( !success )" test that contained the write-in-a-loop code?

I don't mind if it is same robust and effective. Will that new condition test fire only with preallocation: 2 and not with 1 ? How can we make sure that indeed "write-in-a-loop" branch of code is working ? Look at files and check if they become large immediate or growing up gradually ?

I'm going to compile that too and try.

comment:6 Changed 12 years ago by Iznakurnog

New experimental build ready for testing on DNS-323 ! Transmission-1.71_test2.tgz http://slil.ru/27758044 everything fine at fist glance. Now perform zero-padding preallocation at preallocation:2 mode.

comment:7 in reply to: ↑ 4 Changed 12 years ago by hdfan2

Replying to charles:

Why do we need a new preallocation type? What if we were to just add another "if( !success )" test that contained the write-in-a-loop code?

I've done it as a separate option for two reasons: 1) I wanted everything to work as before with old settings, to be sure that nobody will be harmed by this patch; 2) More importantly, I'm not completely sure that fallocate and/or posix_fallocate are absent on our platforms or that they return non-zero (fail). I'm even quite sure that (one of them or both) return 0 because files ARE preallocated (however, in a sparse mode, prone to fragmentation). However, I may be wrong here.

I'll ask Ger Teunis to build one more version, so that I could test this patch on NMT as well.

comment:8 Changed 12 years ago by Iznakurnog

Well Charles patch works fine for DNS-323 but is it possible what on some platforms (Readynas NV+ maybe (???)) fallocate exists but do not work so effective as direct zero-padding approch ? In such case option preallocation:3 would be still usefull in addition to zero-padding fallback (Charles patch) Anyway IMHO it would be rather well to add some diagnostic output to transmission log to clear up which way of preallocation was used exactly .... For instance File ... was preallocated by zero-padding fallback or file ... was preallocated by posix_fallocation call and so on

comment:9 Changed 12 years ago by charles

fallocate()'s been removed from svn trunk for 1.72. <http://www.lucas-nussbaum.net/blog/?p=332> says: "Glibc does have posix_fallocate(), which implements the POSIX interface. posix_fallocate() is wired to use the fallocate system call, for sufficiently modern versions of glibc."

comment:10 follow-ups: Changed 12 years ago by charles

what on some platforms (Readynas NV+ maybe (???)) fallocate exists but do not work so effective as direct zero-padding approach

Do you know of an instance where a filesystem-specific call to preallocate a file exists but is less effective than zero-padding? :)

comment:11 Changed 12 years ago by charles

  • Milestone changed from None Set to 1.72
  • Resolution set to fixed
  • Status changed from new to closed

Added to trunk in r8707 for 1.72.

comment:12 in reply to: ↑ 10 Changed 12 years ago by Iznakurnog

Do you know of an instance where a filesystem-specific call to preallocate a file exists but is less effective than zero-padding? :)

I don't know of course. Just assumtion :) Thanks a lot for 1.72 !

comment:13 in reply to: ↑ 10 Changed 12 years ago by bloodflower

Replying to charles:

what on some platforms (Readynas NV+ maybe (???)) fallocate exists but do not work so effective as direct zero-padding approach

Do you know of an instance where a filesystem-specific call to preallocate a file exists but is less effective than zero-padding? :)

find some information from internet, think it maybe helpful for this ticket: https://www.redhat.com/archives/libvir-list/2009-March/msg00006.html

"I compiled results for ext3, ext4, xfs and btrfs. I used the following methods to allocate a file (1 GB in size) and zero it:

  • posix_fallocate()
  • mmap() and memset()
  • write chunks, sized 4k and 8k.

Results:

--- ext4: posix-fallocate run time:

(approx 0s)

mmap run time:

(approx 13s)

4096-sized chunk run time:

(approx 15s)

8192-sized chunk run time:

(approx 18s)

$ sudo filefrag /mnt/ext4/* /mnt/ext4/file-chunk4: 29 extents found /mnt/ext4/file-chunk8: 20 extents found /mnt/ext4/file-mmap: 38 extents found /mnt/ext4/file-pf: 1 extent found

--- xfs: posix-fallocate run time:

(approx 0s)

mmap run time:

(approx 14s)

4096-sized chunk run time:

(approx 18s)

8192-sized chunk run time:

(approx 19s)

$ sudo filefrag /mnt/xfs/* /mnt/xfs/file-chunk4: 3 extents found /mnt/xfs/file-chunk8: 4 extents found /mnt/xfs/file-mmap: 2 extents found /mnt/xfs/file-pf: 1 extent found

--- ext3: posix-fallocate run time:

(approx 18s)

mmap run time:

(approx 20s)

4096-sized chunk run time:

(approx 22s)

8192-sized chunk run time:

(approx 24s)

$ sudo filefrag /mnt/ext3/* /mnt/ext3/file-chunk4: 38 extents found, perfection would be 9 extents /mnt/ext3/file-chunk8: 9 extents found /mnt/ext3/file-mmap: 44 extents found, perfection would be 9 extents /mnt/ext3/file-pf: 9 extents found

--- btrfs: posix-fallocate run time:

(approx 0s)

mmap run time:

(approx 18s)

4096-sized chunk run time:

(approx 17s)

8192-sized chunk run time:

(approx 19s)

$ sudo /mnt/btrfs/* FIBMAP: Invalid argument "

comment:14 Changed 12 years ago by charles

That's pretty helpful information. Thanks for the link.

Note: See TracTickets for help on using tickets.