files copied via kde sftp into watch dir not loaded

[reardon]

[using r12929]

Torrent files copied to daemon machine are not picked up properly in the watch folder. Bizarrely, files placed into the watch dir by commandline (from debian, to debian) sftp or scp work properly.

But files copied using KDE Dolphin (and it's sftp engine) are not picked up. I presume something in inotify does not work in this case?

This _may_ be a transmission problem, hard to tell. 'touch'ing the file causes it to be picked up.

[reardon]

Well, looking at it some more, I think this is a transmission problem, subtle as it is. The problem is with the copy semantics that KDE follows, and actually I believe other copiers do this as well, rsync eg. The file is copied as a ".part" and then renamed, but not via rename(). transmission-daemon does not handle this.

It seems this is done via a hardlink to the .part file and then removing the original .part file, leaving behind the renamed file. transmission-daemon should catch this by screening IN_CREATE and stat() the file to see what the link count is. If links > 1, then we are in this copy-link-delete situation. Normally I would say the daemon should just grab the file now and get to work, but the problem is that a few more attr calls are coming over the wire.

Really, the best case scenario is for the daemon to delay all watch events for, say, 5 seconds to let everything settle. This would be good karma for regular CLOSE_WRITE events as well. Regardless, the daemoin will still need to detect this specific IN_CREATE/hardlink situation.

Here is the inotify trail when KDE copies the file:

SFTP copies file (watch is turned off):

watch/ CREATE FILENAME.torrent
watch/ OPEN FILENAME.torrent
watch/ MODIFY FILENAME.torrent
watch/ ATTRIB FILENAME.torrent
watch/ ATTRIB FILENAME.torrent
watch/ CLOSE_WRITE,CLOSE FILENAME.torrent

SFTP copies file, (watch is turned on: SUCCESS)
watch/ CREATE FILENAME.torrent
watch/ OPEN FILENAME.torrent
watch/ MODIFY FILENAME.torrent
watch/ ATTRIB FILENAME.torrent
watch/ ATTRIB FILENAME.torrent
watch/ CLOSE_WRITE,CLOSE FILENAME.torrent
watch/ OPEN FILENAME.torrent                <--- where transmission picks up
watch/ ACCESS FILENAME.torrent
watch/ CLOSE_NOWRITE,CLOSE FILENAME.torrent
watch/ MOVED_FROM FILENAME.torrent
watch/ MOVED_TO FILENAME.torrent.added


KDE copies file: (watch is turned on: FAIL)

watch/ CREATE FILENAME.torrent.part
watch/ OPEN FILENAME.torrent.part
watch/ MODIFY FILENAME.torrent.part
watch/ CLOSE_WRITE,CLOSE FILENAME.torrent.part
watch/ CREATE FILENAME.torrent
watch/ DELETE FILENAME.torrent.part
watch/ ATTRIB FILENAME.torrent
watch/ ATTRIB FILENAME.torrent
watch/ OPEN,ISDIR 
watch/ CLOSE_NOWRITE,CLOSE,ISDIR 

'touch' the KDE-copied file (now SUCCESS)
watch/ OPEN FILENAME.torrent
watch/ ATTRIB FILENAME.torrent
watch/ CLOSE_WRITE,CLOSE FILENAME.torrent
watch/ OPEN FILENAME.torrent                <--- where tranmission picks up
watch/ ACCESS FILENAME.torrent
watch/ CLOSE_NOWRITE,CLOSE FILENAME.torrent
watch/ MOVED_FROM FILENAME.torrent
watch/ MOVED_TO FILENAME.torrent.added

[jordan]

reardon, do you want to cook up a patch for this in daemon/watchdir.c?

[reardon]

I will attempt one...

[reardon]

Grr...just tried to do this by extending the inotify() handler to IN_CREATE events and checking for hardlink>1, but that fails because the original filename/link is deleted by the time we can read it. Checking for IN_ATTRIB is dangerous as well, as the standard copy mechanism might be mucking with attributes before finishing it's CLOSE_WRITE.

Basically, the best solution is to scrap the inotify() handler as is and instead have it trigger a directory scan. It's only a slight improvement on the pure polling solution already implemented. Yet another example of the hackery and poor implementation thinking of inotify... a filesystem call without any atomic gaurantee.

[reardon]

Ok, I've tested some more. Even with a very large torrent file, it appears to be safe to just add IN_CREATE to the existing INOTIFY_MASK. transmission appears to be resilient to files in an open-for-writing but not yet completed state. I tested with a large .torrent file (>1MB), ie one which was created well before it was written and the daemon processed it fine even if triggered on IN_CREATE.

A one line change...

#define DTR_INOTIFY_MASK (IN_CLOSE_WRITE|IN_MOVED_TO|IN_CREATE|IN_ONLYDIR)

[jordan]

Yet another ticket that got lost in the passage of time.

Fixed in r13726.