daemon's non-inotify watchdir fails on .torrents with old timestamps

[blackjancsi]

Hey guys,

I am afraid the watch folder does not work for me. These are my params:

"watch-dir": "\/shares\/internal\/PUBLIC\/downloads\/_watch",

"watch-dir-enabled": true

It does not pick up the torrent either on the fly, or at startup.

I have a WD MyBook? World Edition II, and have upgraded transmission from 1.52 to 1.61 directly with optware.

Thanks, blackjancsi

[woodzu123]

In my MyBook? II, transmission 1.60 is working great with that option. I mean: by setting-up "--watch-dir <directory>" when transmission-daemon starting up.

$ uname -a
Linux MyBook 2.6.17.14 #1 PREEMPT Mon Mar 26 12:36:33 BST 2007 armv5tejl GNU/Linux

[charles]

I don't understand the comment vs. the summary. Are you saying that it works in 1.60 but not in 1.61? Or that it works when you start it up by hand? What are the differences between how you you start the sessions that work, and those that don't?

[blackjancsi]

Replying to blackjancsi:

Hey guys,

I am afraid the watch folder does not work for me. These are my params:

"watch-dir": "\/shares\/internal\/PUBLIC\/downloads\/_watch",

"watch-dir-enabled": true

It does not pick up the torrent either on the fly, or at startup.

I have a WD MyBook? World Edition II, and have upgraded transmission from 1.52 to 1.61 directly with optware.

Thanks, blackjancsi

At the risk of looking very stupid, I found out this morning that it DID pick up that last torrent after all, but it did not delete it from the watch folder (so once I would remove it from the queque, it would start up again, I guess). This is very strange, as 2 days ago, I put 4 torrents there and they were not started, even after a day of waiting. Sooooo, it appears that I need more statistics to make relevant observations. Can anyone tell me how to log the daemon's activity in the long run? I have tried running echo $sysvar >> output.textfile commands, but the file was completely empty afterwards. :-( If possible, I would like to avoid keeping a terminal running for a day or so, as it tends to log off automatically / loose connection after some time.

Cheers, blackjancsi

ps.: I would suggest adding another directory for torrent file archival (i.e. once a torrent gets added, the file gets moved from the watch folder to the torrent archive - this way, it can be started again).

[charles]

At the risk of looking very stupid, I found out this morning that it DID pick up that last torrent after all, but it did not delete it from the watch folder (so once I would remove it from the queque, it would start up again, I guess).

Okay, so I'll close this ticket.

This is very strange, as 2 days ago, I put 4 torrents there and they were not started, even after a day of waiting. Sooooo, it appears that I need more statistics to make relevant observations. Can anyone tell me how to log the daemon's activity in the long run? I have tried running echo $sysvar >> output.textfile commands, but the file was completely empty afterwards. :-( If possible, I would like to avoid keeping a terminal running for a day or so, as it tends to log off automatically / loose connection after some time.

there's already a ticket open for the daemon to use syslog, but until then, you could run it in the foreground inside of "screen" so that the session will persist.

ps.: I would suggest adding another directory for torrent file archival (i.e. once a torrent gets added, the file gets moved from the watch folder to the torrent archive - this way, it can be started again).

​http://trac.transmissionbt.com/wiki/ConfigFiles#torrents

[blackjancsi]

ps.: I would suggest adding another directory for torrent file archival (i.e. once a torrent gets added, the file gets moved from the watch folder to the torrent archive - this way, it can be started again).

​http://trac.transmissionbt.com/wiki/ConfigFiles#torrents

Not quite, as this only shows the torrents that are "inside" transmission. Suppose I removed a torrent and wanted to resume seeding later on - it would be really nice to have the torrent file still at disposal (and they really don't take up too much space, do they?)

Question: Are the torrent files in the watch folder deleted once the file gets added to the queue? If no, I could simply leave them there, right (but of course, if they got moved automatically to my proposed "archive", then one knew just from that that the download started, which is nice - I would not have to fire up the browser to check that.

[blackjancsi]

Replying to woodzu123:

In my MyBook? II, transmission 1.60 is working great with that option. I mean: by setting-up "--watch-dir <directory>" when transmission-daemon starting up.

$ uname -a
Linux MyBook 2.6.17.14 #1 PREEMPT Mon Mar 26 12:36:33 BST 2007 armv5tejl GNU/Linux

I have exactly the same OS:

[root@MarkiWD64 ~]# uname -a
Linux MarkiWD64 2.6.17.14 #1 PREEMPT Mon Mar 26 12:36:33 BST 2007 armv5tejl GNU/Linux

I start up the daemon with these commands (inside a script, but these are the important ones):

DAEMON=/opt/bin/transmission-daemon
DAEMON_PORT="51413"
DAEMON_OPTS="-c /shares/internal/PUBLIC/downloads/_watch -t -u *user* -v *passwd*"

* * * and then the start part:

NAME=Transmission
test -x $DAEMON || exit 0
start() {
        echo -e "Starting $NAME\r"
        $DAEMON $DAEMON_OPTS
        if [ -d $TRANSMISSION_WORKDIR ]; then
                sleep 10
                $REMOTE $AUTH -w $TRANSMISSION_WORKDIR
        else
                if [ ! -z "$TRANSMISSION_WORKDIR" ]; then
                        echo -e "note: $TRANSMISSION_WORKDIR doesn't exist, using the default work dir\r"
                fi
        fi
        if [ ! -z "$REMOTE_OPTS" ]; then
                $REMOTE $AUTH $REMOTE_OPTS
        fi
        if [ "$DAEMON_PORT" != "" ]; then
                $REMOTE $AUTH --port $DAEMON_PORT
        fi
        sleep 2
        $REMOTE $AUTH -t all --start
}

What can possibly be wrong here???

[blackjancsi]

Update

At the risk of looking very stupid, I found out this morning that it DID pick up that last torrent after all, but it did not delete it from the watch folder (so once I would remove it from the queque, it would start up again, I guess).

This is very strange, as 2 days ago, I put 4 torrents there and they were not started, even after a day of waiting. Sooooo, it appears that I need more statistics to make relevant observations. Can anyone tell me how to log the daemon's activity in the long run?

I have another case of non-starting - json parameters are (still):

    "watch-dir": "\/shares\/internal\/PUBLIC\/downloads\/_watch",
    "watch-dir-enabled": true

Added a torrent file yesterday night (2 am), still has not started it (8 hours later). Daemon has been running all this time, but debug shows some weird stuff as of this morning (which I will mark with ###WEIRD STUFF):

[root@MarkiWD64 ~]#  transmission-remote -si --debug --auth=*user*:*passwd*
posting:
--------
{"arguments":{},"method":"session-get","tag":0}

--------
> POST /transmission/rpc HTTP/1.1
User-Agent: transmission-remote/1.61 (8420)
Host: localhost:9091
Accept: */*
Accept-Encoding: deflate
Content-Length: 48
Content-Type: application/x-www-form-urlencoded

###WEIRD STUFF #1:
< HTTP/1.1 401 Unauthorized
< Server: Transmission
< WWW-Authenticate: Basic realm="Transmission"
< Date: Wed, 03 Jun 2009 08:16:32 GMT
< Content-Length: 43
< Content-Type: text/html; charset=ISO-8859-1
<
> POST /transmission/rpc HTTP/1.1
Authorization: Basic dHJhbnNtaXNzaW9uOmQ4NmQ2ZThRMmY=
User-Agent: transmission-remote/1.61 (8420)
Host: localhost:9091
Accept: */*
Accept-Encoding: deflate
Content-Length: 48
Content-Type: application/x-www-form-urlencoded

###WEIRD STUFF #2:
< HTTP/1.1 409 Conflict
< Server: Transmission
< X-Transmission-Session-Id: EUFmDC4L52lQ1DKTlf5CCPQ8Fq5s0jMGecpuE4iEIK74Jl1k
< Date: Wed, 03 Jun 2009 08:16:32 GMT
< Content-Length: 762
< Content-Type: text/html; charset=ISO-8859-1
<
posting:
--------
{"arguments":{},"method":"session-get","tag":0}

--------
> POST /transmission/rpc HTTP/1.1
User-Agent: transmission-remote/1.61 (8420)
Host: localhost:9091
Accept: */*
Accept-Encoding: deflate
X-Transmission-Session-Id: EUFmDC4L52lQ1DKTlf5CCPQ8Fq5s0jMGecpuE4iEIK74Jl1k
Content-Length: 48
Content-Type: application/x-www-form-urlencoded

< HTTP/1.1 401 Unauthorized
< Server: Transmission
< WWW-Authenticate: Basic realm="Transmission"
< Date: Wed, 03 Jun 2009 08:16:32 GMT
< Content-Length: 43
< Content-Type: text/html; charset=ISO-8859-1
<
> POST /transmission/rpc HTTP/1.1
Authorization: Basic dHJhbnNtaXNzaW9uOmQ4NmQ2ZThRMmY=
User-Agent: transmission-remote/1.61 (8420)
Host: localhost:9091
Accept: */*
Accept-Encoding: deflate
X-Transmission-Session-Id: EUFmDC4L52lQ1DKTlf5CCPQ8Fq5s0jMGecpuE4iEIK74Jl1k
Content-Length: 48
Content-Type: application/x-www-form-urlencoded

< HTTP/1.1 200 OK
< Server: Transmission
< Content-Encoding: deflate
< Content-Type: application/json; charset=UTF-8
< Date: Wed, 03 Jun 2009 08:16:32 GMT
< Content-Length: 333
<
got response (len 705):
--------
{"arguments":{"alt-speed-down":300,"alt-speed-enabled":true,"alt-speed-time-begin":30,"alt-speed-time-day":62,"alt-speed-time-enabled":true,"alt-speed-time-end":1080,"alt-speed-up":24,"blocklist-enabled":false,"blocklist-size":0,"download-dir":"\/shares\/internal\/PUBLIC\/downloads","encryption":"tolerated","peer-limit-global":240,"peer-limit-per-torrent":60,"peer-port":51413,"peer-port-random-on-start":0,"pex-enabled":true,"port-forwarding-enabled":true,"rpc-version":6,"rpc-version-minimum":1,"seedRatioLimit":1.2000,"seedRatioLimited":true,"speed-limit-down":150,"speed-limit-down-enabled":true,"speed-limit-up":10,"speed-limit-up-enabled":true,"version":"1.61 (8420)"},"result":"success","tag":0}

--------
VERSION
  Daemon version: 1.61 (8420)
  RPC version: 6
  RPC minimum version: 1

TRANSFER
  Download directory: /shares/internal/PUBLIC/downloads
  Portforwarding enabled: Yes
  Encryption: tolerated

LIMITS
  Peer limit: 240
  Downloadlimit enabled: Yes
  Downloadlimit:    150 KB/sec
  Uploadlimit enabled:   Yes
  Uploadlimit:       10 KB/sec

The reason I thought these were weird was that I only started doing stuff on it just 20 minutes ago - at 8 am of this morning, it could have been only the daemon. Before you ask, the PC I use to access the box was off, no other PC was ON on the network, let alone have a browser open trying to access that port. The network is WPA2 protected with a long password (random alphanumeric), however, the transmission port is routed through the router (but from a different one, so if I access it from the outside, it routes e.g. 48721 -> 9091 (so it is not 100% sure that I did not have some outside "tentative" at that time - just a little unlikely in my opinion)

I have restarted the daemon since, 20 minutes passed, the torrent has still not started (but as you can see, it did start them yesterday).

there's already a ticket open for the daemon to use syslog, but until then, you could run it in the foreground inside of "screen" so that the session will persist.

To solve this, I needed to log the daemon better. Sorry, but your lines just above are not 100% clear to me - do you suggest that I keep the terminal window I use to access my NAS storage box open for an extended amount of time, or installing a program called "screen"?

[blackjancsi]

Sorry, I was stupid, I did not check "reopen"...

[wishyou]

I have this exact same problem on the Netgear ReadyNAS NV+ Picking up torrents from the wathcfolder seems random, sometimes it works, sometimes it doesn't. So far it seems to work when I copy a torrent file internally with an SSH-shell, but dropping in files over Samba or wget-ing them with PHP script fails.

Wish

[charles]

Comment #7's "WEIRD STUFF" is an unrelated red herring. The daemon gives 409 the first time around to make the client re-request with the X-Transmission-Session-Id header, which is required to prevent ​CSRF exploits of our HTTP/JSON RPC. So that log is perfectly OK, and unrelated to the watch directory.

Summary of ticket so far:

• Works in 1.60 when started by hand
• Works in 1.61 but doesn't delete the .torrent file
• Extra information about the script you're using to launch Transmission, but doesn't yet seem to affect the issue at hand.
• Failed to add a torrent overnight, but hard to tell why because of insufficient logging in the daemon
• The server you're running Transmission on likes to disconnect you, so you'd rather not run Transmission in the foreground in a terminal.

The answer to that last bullet point is to run Transmission in the program called "screen" which can survive disconnects. After a disconnect, you can log back in and reattach to the still-running screen.

[blackjancsi]

Replying to charles:

Comment #7's "WEIRD STUFF" is an unrelated red herring. The daemon gives 409 the first time around to make the client re-request with the X-Transmission-Session-Id header, which is required to prevent ​CSRF exploits of our HTTP/JSON RPC. So that log is perfectly OK, and unrelated to the watch directory.

Right, but what about the two others? (401-s, there is one of them that I did not flag below) Those seem to be lagging two hours behind the actual time (right now there are two again in the debug of transmission-remote, both from 12:43:57 GMT (in the morning, it was 08:16:32 GMT, then a bit later around 8:42, so unfortunately, no easy periodicity spotted).

[blackjancsi]

The answer to that last bullet point is to run Transmission in the program called "screen" which can survive disconnects. After a disconnect, you can log back in and reattach to the still-running screen.

OK, I'll install it, run it in the foreground and see what happens (I am not sure I can do it tonight, hope this is not a problem).

[charles]

Comment #7's 401 is also unrelated. That's because you're running with username/password enabled, and that's the standard mechanism for http servers to request authentication.

[charles]

any news?

[wishyou]

Okay, I think I've figured this one out. First of all I had problems with some file permissions being reset, but getting that out of the way got me further.

I've been poking around in the source (branch 1.61) and running things in foreground with screen, and it seems that the way this is implemented with readdir(), it is highly dependent of the last-modified timestamp. This means that any operation preserving timestamps on torrent files, like mv or dropping them in over samba, will not work unless you manage to create and move the torrent within 10 seconds (or less depending on where in the watch the loop cycle you happened to start).

touch-ing the files in-place after a "fail" starts them right away, which makes the explanation very plausible. I'd say this is more of a weakness with the readdir() implementation than a bug, but now that I'm aware of it its easy to work around. Perhaps an improved solution with a done folder could be considered.

Wish

[blackjancsi]

Replying to charles:

any news?

Sorry, not enough time. I installed screen, but being no guru, I still try to understand how it works. Will get back in about a week with my info. On the other hand, if wishyou's info is enough, then I might not... :-)

[wishyou]

Replying to blackjancsi:

Replying to charles:

any news?

Sorry, not enough time. I installed screen, but being no guru, I still try to understand how it works. Will get back in about a week with my info. On the other hand, if wishyou's info is enough, then I might not... :-)

I suggest you check your file timestamps too, to see if my explanation is plausible for your problems as well. If they are, a more robust pickup-routine may be called for, especially for platforms where one has no shell access or no way of touch-ing the modified-timestamp of smb-copied/moved files without special tools.

Wish

[charles]

the issue reported by wishyou is fixed now in r8628.

automated tarballs of new builds can be found at ​http://build.transmissionbt.com/job/trunk-linux-inc/