manual DHT bootstrapping

[User294]

In some cases the only way to go is to manually bootstrap DHT by supplying some remote peer's UDP ip:port to start up DHT operations manually (especially vital when tracker down). Some other torrent clients allow to do such trick.

Real-world example from my recent experience: I wanted to share (completely legal) movie (about Netscape's story and how it became Firefox) since people had troubles downloading it via file hosting services (they're real PITA!). I decided torrent could do this job better if used properly.

To make things better and faster I wanted to set up several Linux boxes with fast connections to help me in seeding with their horsepower.

That's where I got in stuck with Transmission (and other clients as well). The problem was dumb and simple. Most of popular trackers I tried to use simply do NOT allow connections coming from datacenters for some reasons. I guess, there is some IP filters in use or so (maybe because of massive attacks of trackers by anti-pirate companies conducted from datacenters, etc, hell knows).

So at the end of day I had powerful hosts with fast channels to seed stuff. But peers were NOT aware they're here. Because hosts can not contact trackers (I tried several most popular trackers and got tired with this stuf). Since there is no trackers, there is no peers. And since this is FIRST RUN, there is also no DHT... and no ways to start it up since trackers do not accept connections and so, no peers to bootstrap DHT from. Can you see? An infinite loop.

After all fruitless attempts I have resorted to another torrent client which allowed to supply IP and port of DHT-enabled peer to get DHT started (this process is known as manual bootstrap or so).

Then I pointed clients at boxes IP:port of my machine at home (where things were fine in terms of talking to trackers so there were peers and hence DHT has been running as well). Then DHT has started and machine started to upload like a mad. It has been victory! Free content was finally able to reach people at PROPER speeds. I repeated this several times and turned my machines in datacenters I'm using anyway into a nice swarm which allowed to seed files to many peers quickly. It looked like a clear victory.

The only downside was that transmission-daemon seems to fit better for such tasks. And it is sad I was not able to use it due to such VERY DUMB issue like lack of communications to trackers for some unknown reason (I really have no idea why trackers are banning machines in datacenters without any offence from my side).

So, it could be great if there is an option to bootstrap DHT by manually entering IP:port of alive peer. As far as I know, DHT library used haves proper functions to implement this so it's probably just matter of few small changes in UI and command-line parsers. But this can sometimes save a day!

Yet another thing which can relax this problem is that you can supply some "predefined" dht.dat with Transmission's package itself so it does not have to start from scratch. Practice has shown that if we store some 100-200 DHT peers for bootstrap purposes, some of them are usually valid even after SEVERAL MONTHS. So even half-year-old dht.dat can give fair chances to bootstrap DHT.

Or there can be option to download this dht.dat file via specified URL, etc.

[jch]

I think this is a rather exotic task, and I'm not sure it's a good idea to clutter the Transmission UI with it.

Note, however, that copying the dht.dat from another instance of transmission is *not* a good idea, unless you remove the original. The dht.dat contains the node's id, which must be globally unique.

In case somebody decides to implement this, the right way is to dht_ping the given IP address, say three times at one-second intervals.

--Juliusz

[charles]

jch: thanks for weighing in on this. I'm not overly fond of this idea either.

[jch]

I'm renaming and reopening this report, since I need to think about it some more.

The point you're making is that the current support for bootstrapping the DHT is insufficient in some cases. As far as I am aware, there are five techniques for bootstrapping the DHT, only two of which are implemented in Transmission:

1. BitTorrent? protocol extension: if a BitTorrent? peer announces support for the DHT in the initial handshake (using the PORT message), we use that peer for bootstrapping.

2. opportunistic bootstrapping: whenever we receive a DHT request, we use the sending node for bootstrapping.

3. Extension to the .torrent file format: a .torrent file may contain a number of addresses of DHT peers, which we don't currently use.

4. Hard-wired node names (such as router.utorrent.com), which are resolved using the DNS; we don't currently hard-wire any such names.

5. Manual bootstrapping, which is what you suggest.

Now usually Transmission will use method (1) for bootstrapping -- since there is only one global DHT, meeting any one peer that already participates in the DHT is enough; so a user that is firewalled away from TPB only needs to start downloading a torrent from a different tracker in order to bootstrap the DHT. It would appear that this is not enough in your case.

A possible solution would be to implement method (3) as a fall-back; this needs to be done carefully, in order to avoid overloading the bootstrap nodes (i.e. it should only be used if normal bootstrapping failed after 20 minutes or so); the same applies to (4). Method (5), as I noted above, is not a good idea, as it complicates the UI for what is a rather exotic task (and the few people who need this feature are welcome to perform manual surgery on their dht.dat).

All of this needs more thought, especially so since I want our implementation of the DHT to be significantly less aggressive than the one in uTorrent (let alone the one in rtorrent).

--Juliusz

[User294]

As for me I need manual bootstrapping via manual IP:port entry.

I had run into the following scenario:

I wanted to seed good movie about Mozilla (under Creative Commons license) to make it's downloads easier. People who created movie have lacked proper hosting which can handle such download and they used file downloading hostings like RapidShare?. Should I mention that downloading multi-part archives from such hostings is a real PITA? That's why I decided that once I had such PITA and managed to download film (after two days of f...g with file sharing hostings) I have to relax this PITA for others.

Yet I had too slow upload on my primary internet channel. It was clearly not enough to seed uploads like this and provide people with a proper speeds which CC-licensed movie deserves. But I had several machines located in data centers, used for my own purposes. They have a very cool channels so they can help to ignite seeding. They have limited bandwidth (limit is big but it exists and probably can be exceeded if HTTP server will upload stuff instead). But that's ok: once swarm getting big, they can stop seeding and swarm have to live on it's own so that's not a problem.

I had run into very stupid problem though. Trackers are not happy about IPs from datacenters (maybe because of anti-P2P bastards using them as well). I tried 3 or 4 trackers and none of them allowed these "seedboxes" to connect. That was stupid but I has been in stuck.

Furthermore, it has turned out that all publicly available trackers I tried have intermittent issues with allowing initial seeder to connect them, hence I was not able to create stable swarm with complete parts set at all. This sucked a way too much. F%&#k you, trackers. I'm really hate trackers after all this headache!

Then I have to resort to DHT for these seedboxes (and everyone who is not using DHT is really stupid looser. Yep, REALLY stupid). The only problem is that DHT lacks any peers. Once no trackers accepting "seedboxes", they also lack any known peers. And it's obvious I do not had dht.dat at this point at seedbox side (and to make things even worse, initial seeder was not a Transmission client). And no, editing dht.dat is not an option. This is A WAY TOO HARD. It requires format knowledge and Transmission lacks verbose output on DHT operations so I can't even quickly check if DHT started properly. To add more "fun", initial seeder had dynamic IP. Are you kidding about editing dht.dat, right? Entering IP:port is boring but way to go. Hacking binary file to do the same is a bit unfriendly and tricky.

It has turned out that I HAD NO ANY SUITABLE BOOTSTRAP METHODS AT ALL. Yes, this looked like dead end. So, "seedboxes" were here, ready to upload. But peers were not aware of them (since trackers were not available and I failed to ignite DHT).

Solution? I used yet another torrent client! Which allows me to bootstrap with IP:port and I knew IP:Port of initial seeder. It haves too weak channel but it had ignited DHT on my "seedboxes" like a charm via IP:Port (yes, it is!!!). And then "seedboxes" entered the "game" and it has been awesome (and if some looser did not found these seedboxes due to ignorance of DHT and had sucking download speeds at this phase, that's their own problems).

Bottom line: please implement as many bootstrap methods as possible. Including ability to bootstrap from specific IP and port. And no, DHT should not rely on trackers, ever. Look, they were unavailable for my machines and I had a ton of headache. That really sucked too much. DHT have to be INDEPENDENT from trackers, ESPECIALLY bootstrapping. Otherwise once tracker down you can be loser regardless of DHT formally supported by client. So, IP:Port bootstrap is rarely needed. But it can save a day! Just as it has happened for me. So, now ppl can download nice movie about Mozilla under Creative Commons license quickly, in convenient way and at good speed.

So, all these headaches were worth of it. The only drawback is that I wanted to use Transmission on "seedboxes" (since it's lightweight and haves nice web UI) but I failed to do so due to inability to bootstrap DHT since trackers are refusing to allow my IPs so I lacked DHT peers as well.

P.S. I was evil enough to set up severity as "Major" because this stupid issue has caused mission failure and forced me to choose yet another torrent client while I really prefer transmission. Sorry for any inconveniences caused.

[livings124]

This is not a major issue.

[jch]

The following patch implements manual bootstrapping as a fallback.

Just add a file ~/.config/transmission/dht.bootstrap that says something like

router.bittorrent.com 6881
router.utorrent.com 6881

and Transmission will bootstrap from those nodes if normal bootstrap fails.

Charles, please commit this, so I can make a version of #2576 that builds on that.

--Juliusz

[titer]

Now we also have dht.transmissionbt.com (port 6881) set up for the "4. Hard-wired node names" bootstrapping technique

[jch]

Replying to titer:

Now we also have dht.transmissionbt.com (port 6881) set up

Yep. Patch in the works, I'll submit it shortly.

Titer, I love you.

--Juliusz

P.S. Assuming this declaration has put you in a good mood -- please do darcs pull again. I've just fixed a serious bug.

[jch]

Closing this, since I'm working on a verson of #2576 that subsumes it (in a smarter manner).

Charles, let me know whether you want to get 1.80 out now, in which case you should reopen this bug, or whether I have some time to work on a version of #2576 that can be considered stable.

--Juliusz

[charles]

Juliusz: wrt 1.80, my plan is to have a beta out by the end of the year. IMO the changes in 1.80 are large enough that we'll need a couple of beta releases before we can release with a degree of confidence.

[jch]

Fixed in r9549.

--Juliusz

[User294]

Btw thanks for these fixes. Actually my life now much more easy when it comes to using transmission in various environments.