Opened 10 years ago

Closed 9 years ago

Last modified 9 years ago

#4523 closed Bug (fixed)

files copied via kde sftp into watch dir not loaded

Reported by: reardon Owned by: jordan
Priority: Normal Milestone: 2.76
Component: Daemon Version: 2.33
Severity: Minor Keywords:
Cc:

Description

[using r12929]

Torrent files copied to daemon machine are not picked up properly in the watch folder. Bizarrely, files placed into the watch dir by commandline (from debian, to debian) sftp or scp work properly.

But files copied using KDE Dolphin (and it's sftp engine) are not picked up. I presume something in inotify does not work in this case?

This _may_ be a transmission problem, hard to tell. 'touch'ing the file causes it to be picked up.

Change History (10)

comment:1 Changed 10 years ago by livings124

  • Version changed from 2.33 to 2.33+

comment:2 Changed 10 years ago by reardon

Well, looking at it some more, I think this is a transmission problem, subtle as it is. The problem is with the copy semantics that KDE follows, and actually I believe other copiers do this as well, rsync eg. The file is copied as a ".part" and then renamed, but not via rename(). transmission-daemon does not handle this.

It seems this is done via a hardlink to the .part file and then removing the original .part file, leaving behind the renamed file. transmission-daemon should catch this by screening IN_CREATE and stat() the file to see what the link count is. If links > 1, then we are in this copy-link-delete situation. Normally I would say the daemon should just grab the file now and get to work, but the problem is that a few more attr calls are coming over the wire.

Really, the best case scenario is for the daemon to delay all watch events for, say, 5 seconds to let everything settle. This would be good karma for regular CLOSE_WRITE events as well. Regardless, the daemoin will still need to detect this specific IN_CREATE/hardlink situation.

Here is the inotify trail when KDE copies the file:

SFTP copies file (watch is turned off):

watch/ CREATE FILENAME.torrent
watch/ OPEN FILENAME.torrent
watch/ MODIFY FILENAME.torrent
watch/ ATTRIB FILENAME.torrent
watch/ ATTRIB FILENAME.torrent
watch/ CLOSE_WRITE,CLOSE FILENAME.torrent

SFTP copies file, (watch is turned on: SUCCESS)
watch/ CREATE FILENAME.torrent
watch/ OPEN FILENAME.torrent
watch/ MODIFY FILENAME.torrent
watch/ ATTRIB FILENAME.torrent
watch/ ATTRIB FILENAME.torrent
watch/ CLOSE_WRITE,CLOSE FILENAME.torrent
watch/ OPEN FILENAME.torrent                <--- where transmission picks up
watch/ ACCESS FILENAME.torrent
watch/ CLOSE_NOWRITE,CLOSE FILENAME.torrent
watch/ MOVED_FROM FILENAME.torrent
watch/ MOVED_TO FILENAME.torrent.added


KDE copies file: (watch is turned on: FAIL)

watch/ CREATE FILENAME.torrent.part
watch/ OPEN FILENAME.torrent.part
watch/ MODIFY FILENAME.torrent.part
watch/ CLOSE_WRITE,CLOSE FILENAME.torrent.part
watch/ CREATE FILENAME.torrent
watch/ DELETE FILENAME.torrent.part
watch/ ATTRIB FILENAME.torrent
watch/ ATTRIB FILENAME.torrent
watch/ OPEN,ISDIR 
watch/ CLOSE_NOWRITE,CLOSE,ISDIR 

'touch' the KDE-copied file (now SUCCESS)
watch/ OPEN FILENAME.torrent
watch/ ATTRIB FILENAME.torrent
watch/ CLOSE_WRITE,CLOSE FILENAME.torrent
watch/ OPEN FILENAME.torrent                <--- where tranmission picks up
watch/ ACCESS FILENAME.torrent
watch/ CLOSE_NOWRITE,CLOSE FILENAME.torrent
watch/ MOVED_FROM FILENAME.torrent
watch/ MOVED_TO FILENAME.torrent.added

comment:3 Changed 10 years ago by jordan

  • Component changed from Transmission to Daemon
  • Version changed from 2.33+ to 2.33

reardon, do you want to cook up a patch for this in daemon/watchdir.c?

comment:4 Changed 10 years ago by reardon

I will attempt one...

comment:5 Changed 10 years ago by reardon

Grr...just tried to do this by extending the inotify() handler to IN_CREATE events and checking for hardlink>1, but that fails because the original filename/link is deleted by the time we can read it. Checking for IN_ATTRIB is dangerous as well, as the standard copy mechanism might be mucking with attributes before finishing it's CLOSE_WRITE.

Basically, the best solution is to scrap the inotify() handler as is and instead have it trigger a directory scan. It's only a slight improvement on the pure polling solution already implemented. Yet another example of the hackery and poor implementation thinking of inotify... a filesystem call without any atomic gaurantee.

comment:6 Changed 10 years ago by reardon

Ok, I've tested some more. Even with a very large torrent file, it appears to be safe to just add IN_CREATE to the existing INOTIFY_MASK. transmission appears to be resilient to files in an open-for-writing but not yet completed state. I tested with a large .torrent file (>1MB), ie one which was created well before it was written and the daemon processed it fine even if triggered on IN_CREATE.

A one line change...

#define DTR_INOTIFY_MASK (IN_CLOSE_WRITE|IN_MOVED_TO|IN_CREATE|IN_ONLYDIR)

comment:7 Changed 10 years ago by jordan

  • Milestone changed from None Set to Sometime
  • Severity changed from Normal to Minor

comment:8 Changed 9 years ago by jordan

  • Milestone changed from Sometime to 2.80
  • Owner set to jordan
  • Status changed from new to assigned

comment:9 Changed 9 years ago by jordan

  • Resolution set to fixed
  • Status changed from assigned to closed

Yet another ticket that got lost in the passage of time.

Fixed in r13726.

comment:10 Changed 9 years ago by jordan

  • Milestone changed from 2.80 to 2.76
Note: See TracTickets for help on using tickets.