Opened 11 years ago

Closed 10 years ago

Last modified 6 years ago

#4016 closed Enhancement (wontfix)

Blocklists loaded in Transmission should be used to filter DHT communication.

Reported by: x190 Owned by:
Priority: Normal Milestone: None Set
Component: Transmission Version: 2.20
Severity: Normal Keywords: blocklist, DHT
Cc: jch@…

Description

If we are blocking P2P communication for blocklisted peers we should not be communicating with them using PEX or DHT.

Attachments (5)

Archive.zip (85.2 KB) - added by x190 11 years ago.
Archive2.zip (12.8 KB) - added by x190 11 years ago.
blocklist.diff (3.4 KB) - added by jordan 11 years ago.
Archive_DHT_udp.zip (127.2 KB) - added by x190 11 years ago.
dht-blocklist.diff (904 bytes) - added by jordan 10 years ago.

Download all attachments as: .zip

Change History (75)

comment:1 Changed 11 years ago by jordan

I don't understand what you mean about PEX in this ticket. How are we communicating with a blocklisted peer via PEX?

Changed 11 years ago by x190

comment:3 Changed 11 years ago by x190

2011-02-11 22:55:24 -0700 net.c:452 [Error] Transmission: Couldn't connect socket 149 to 24.142.xx.115, port 15908 (errno 61 - Connection refused)[PG2 block]

This address is in a loaded blocklist but connections are repeatedly attempted. There are other examples. Currently, I am using only PEX and cached peers. If I disable PEX the connection attempts to that address cease. If I re-enable PEX the connection attempts resume. I don't pretend to understand the technicalities of PEX communication, so as regards this unblocked connection attempt I assumed it is related to PEX communication for the stated reasons. In any case these connection attempts should not be made.

In summation whether this involves "PEX communication" or PEX-flagged peers, (EDIT: or cached peers for that matter) connection attempts should not be made to peers in a loaded blocklist for any reason.

Last edited 11 years ago by x190 (previous) (diff)

comment:4 follow-up: Changed 11 years ago by jordan

x190, thanks for clarifying that. A few more questions, now that I understand better what you're asking:

  • IIRC you're doing your own builds in xcode, is that correct? Would it be possible for you to set a breakpoint for that line in net.c and provide a backtrace of where it's being called from? My best guess is the call chain will look like makeNewPeerConnections() -> initiateCandidateConnections() -> initiateConnection() -> tr_peerIoNewOutgoing() -> tr_netOpenPeerSocket(), but I'd like to be sure.
  • Could you provide a screenshot of the preferences dialog tab that covers blocklists? I'd like to look at the blocklist you're using, make sure it's enabled, and see how many rules it has.
  • Would it be possible for you to do this in a stock version of 2.20, rather than your patched one? The line numbers don't match up and I don't know whether or not ijuxda has calls to tr_netOpenPeerSocket() anywhere else in the code. More to the point, I don't want unnecessary complications floating around when trying to track down this issue :)

Thanks!

comment:5 in reply to: ↑ 4 Changed 11 years ago by x190

Replying to jordan:

x190, thanks for clarifying that. A few more questions, now that I understand better what you're asking:

  • IIRC you're doing your own builds in xcode, is that correct? Would it be possible for you to set a breakpoint for that line in net.c and provide a backtrace of where it's being called from? My best guess is the call chain will look like makeNewPeerConnections() -> initiateCandidateConnections() -> initiateConnection() -> tr_peerIoNewOutgoing() -> tr_netOpenPeerSocket(), but I'd like to be sure.

I'll try but I'm unsure of the details. I remember seeing that breakpoint thing but how do I initiate a backtrace?

  • Could you provide a screenshot of the preferences dialog tab that covers blocklists? I'd like to look at the blocklist you're using, make sure it's enabled, and see how many rules it has.

From Message Log. EDIT: Just so you have no doubt, I updated these lists and double-checked the entries before posting.

2011-02-11 23:59:07 -0700 blocklist.c:114 [Info] Transmission: Blocklist "badpeers.bin" contains 21427 entries
2011-02-11 23:59:07 -0700 blocklist.c:114 [Info] Transmission: Blocklist "blocklist.bin" contains 288030 entries [P2P]
2011-02-11 23:59:07 -0700 blocklist.c:114 [Info] Transmission: Blocklist "bogon.bin" contains 6653 entries
2011-02-11 23:59:07 -0700 blocklist.c:114 [Info] Transmission: Blocklist "dcha_faker.bin" contains 40 entries
2011-02-11 23:59:07 -0700 blocklist.c:114 [Info] Transmission: Blocklist "dcha_hacker.bin" contains 113 entries
2011-02-11 23:59:07 -0700 blocklist.c:114 [Info] Transmission: Blocklist "dshield.bin" contains 120 entries
2011-02-11 23:59:07 -0700 blocklist.c:114 [Info] Transmission: Blocklist "hijacked.bin" contains 312 entries
2011-02-11 23:59:07 -0700 blocklist.c:114 [Info] Transmission: Blocklist "iana-multicast.bin" contains 1 entries
2011-02-11 23:59:07 -0700 blocklist.c:114 [Info] Transmission: Blocklist "iana-private.bin" contains 13 entries
2011-02-11 23:59:07 -0700 blocklist.c:114 [Info] Transmission: Blocklist "iana-reserved.bin" contains 46 entries
2011-02-11 23:59:07 -0700 blocklist.c:114 [Info] Transmission: Blocklist "level1.bin" contains 224923 entries
2011-02-11 23:59:07 -0700 blocklist.c:114 [Info] Transmission: Blocklist "level2.bin" contains 80128 entries
2011-02-11 23:59:07 -0700 blocklist.c:114 [Info] Transmission: Blocklist "level3.bin" contains 17155 entries
2011-02-11 23:59:07 -0700 blocklist.c:114 [Info] Transmission: Blocklist "MyBlocklist.bin" contains 8 entries
2011-02-11 23:59:07 -0700 blocklist.c:114 [Info] Transmission: Blocklist "spider.bin" contains 875 entries
2011-02-11 23:59:07 -0700 blocklist.c:114 [Info] Transmission: Blocklist "spyware.bin" contains 2689 entries
  • Would it be possible for you to do this in a stock version of 2.20, rather than your patched one? The line numbers don't match up and I don't know whether or not ijuxda has calls to tr_netOpenPeerSocket() anywhere else in the code. More to the point, I don't want unnecessary complications floating around when trying to track down this issue :)

Well, give me the pointers mentioned above re: xcode and I'll try to do that first and then maybe we can use 'find' to find "calls to tr_netOpenPeerSocket()".

Last edited 11 years ago by x190 (previous) (diff)

Changed 11 years ago by x190

comment:6 Changed 11 years ago by jordan

Well as we've discussed I don't run OS X, so I can't give you firsthand xcode advice.

In the section "Adding File-Line Breakpoints" on the page http://developer.apple.com/library/mac/#documentation/DeveloperTools/Conceptual/XcodeDebugging/200-Managing_Program_Execution/program_execution.html, it says you can "Select the line at which you want the breakpoint to be triggered and choose Run > Manage Breakpoints > Add Breakpoint at Current Line."

Also these links have some advice on running the xcode debugger smoothly:

Good luck :)

comment:7 follow-up: Changed 11 years ago by jordan

2011-02-11 22:55:24 -0700 net.c:452 [Error] Transmission: Couldn't connect socket 149 to 24.142.xx.115, port 15908 (errno 61 - Connection refused)[PG2 block]

I can't tell whether this address is in your blocklist or not. level1 blocks 1/128th of 24.142.xx.yy but you've redacted part of the IP address. Also, you've got about a dozen different blocklists that I don't have access to.

To make things simpler, could you show an example of Transmission trying to talk to an address that's listed in one of your blocklists, and also show the relevant section of the blocklist?

Thanks!

comment:8 in reply to: ↑ 7 Changed 11 years ago by x190

Replying to jordan:

2011-02-11 22:55:24 -0700 net.c:452 [Error] Transmission: Couldn't connect socket 149 to 24.142.xx.115, port 15908 (errno 61 - Connection refused)[PG2 block]

I can't tell whether this address is in your blocklist or not. level1 blocks 1/128th of 24.142.xx.yy but you've redacted part of the IP address. Also, you've got about a dozen different blocklists that I don't have access to.

To make things simpler, could you show an example of Transmission trying to talk to an address that's listed in one of your blocklists, and also show the relevant section of the blocklist?

Thanks!

Type iblocklist into a browser address bar! OR trust me that I double and triple checked that address was loaded.

2011-02-12 13:10:59 -0700 net.c:452 [Error] Transmission: Couldn't connect socket 61 to 24.142.33.115, port 62559 (errno 61 - Connection refused)

From Level 3 Blocklist:
Intelligent Communications, Inc:24.142.0.0-24.142.95.255

I did have that gdb breakpoint thing going last night (all night grrrr...!). SIGPIPES nostop and all but what's the bt command? I used bt and bt +/- n but that only gives thread 0. Anyhow, if you take a close look at that crash I included in Archive2 the crash thread probably looks a lot like or same as a proper gdb bt.

Last edited 11 years ago by x190 (previous) (diff)

comment:9 follow-up: Changed 11 years ago by jordan

It looks a little more complicated than that. Here is the 24.142.x.x block in level3:

Intelligent Communications, Inc:24.142.0.0-24.142.95.255
San Bruno Municipal Cable TV:24.142.5.0-24.142.5.255
San Bruno Municipal Cable TV:24.142.75.0-24.142.75.255
Intelligent Communications, Inc:24.142.128.0-24.142.255.255
San Bruno Municipal Cable TV:24.142.152.0-24.142.154.255
San Bruno Municipal Cable TV:24.142.170.0-24.142.170.255
San Bruno Municipal Cable TV:24.142.183.0-24.142.191.255

The last six lines are all subsets of the first one. This could be the problem, since Transmission assumes the rules are sorted in ascending order and don't overlap, so that we can do a binary search on them.

Is this common practice for blocklists? Or is it a bug in the blocklist that needs to be addressed upstream?

comment:10 follow-up: Changed 11 years ago by jordan

Looks like there are several instances of it in level3.

x190, please exit transmission, then delete the .bin files from your blocklists folder, then rebuild with the attached blocklist.diff patch against r11882 or higher. Does it resolve the problem?

Last edited 11 years ago by jordan (previous) (diff)

Changed 11 years ago by jordan

comment:11 in reply to: ↑ 9 Changed 11 years ago by x190

Replying to jordan:

It looks a little more complicated than that. Here is the 24.142.x.x block in level3:

Intelligent Communications, Inc:24.142.0.0-24.142.95.255
San Bruno Municipal Cable TV:24.142.5.0-24.142.5.255
San Bruno Municipal Cable TV:24.142.75.0-24.142.75.255
Intelligent Communications, Inc:24.142.128.0-24.142.255.255
San Bruno Municipal Cable TV:24.142.152.0-24.142.154.255
San Bruno Municipal Cable TV:24.142.170.0-24.142.170.255
San Bruno Municipal Cable TV:24.142.183.0-24.142.191.255

The last six lines are all subsets of the first one. This could be the problem, since Transmission assumes the rules are sorted in ascending order and don't overlap, so that we can do a binary search on them.

Is this common practice for blocklists? Or is it a bug in the blocklist that needs to be addressed upstream?

This is a standard bluetack numerically sorted blocklist. iBlocklist gets its level1/2/3 lists from bluetack. PG2 has no issues with this format and until now we all believed T didn't either...:)

comment:12 in reply to: ↑ 10 Changed 11 years ago by x190

Replying to jordan:

Looks like there are several instances of it in level3.

x190, please exit transmission, then delete the .bin files from your blocklists folder, then rebuild with the attached blocklist.diff patch against r11882 or higher. Does it resolve the problem?

Will do ASAP.

Changed 11 years ago by x190

comment:13 Changed 11 years ago by x190

In initial testing qsort seems to be very promising in regards to the PEX problem, however, we still need to address the DHT udp traffic problem. It is well known that the bad guys use DHT for nefarious purposes and we will not connect using TCP, so we should not communicate with blocklisted peers over the DHT network.

comment:14 follow-up: Changed 11 years ago by jordan

x190, just to confirm, is TCP blocklisting working after r11888?

Last edited 11 years ago by jordan (previous) (diff)

comment:15 in reply to: ↑ 14 Changed 11 years ago by x190

Replying to jordan:

x190, just to confirm, is TCP blocklisting working after r11888?

I am unable to attest to what happens in regard to peers acquired by DHT due to my hesitancy to use that feature because of the issue raised in this ticket, however, peers acquired from trackers, cache, and PEX are being filtered properly now in my testing. Thank you for fixing the blocklist sort code. r11889

comment:16 Changed 11 years ago by jordan

  • Keywords PEX removed
  • Summary changed from Blocklists loaded in Transmission should be used to filter PEX and DHT communication. to Blocklists loaded in Transmission should be used to filter DHT communication.

Changed 10 years ago by jordan

comment:17 follow-up: Changed 10 years ago by jordan

x190, could you give dht-blocklist.diff a try?

comment:18 in reply to: ↑ 17 Changed 10 years ago by x190

Replying to jordan:

x190, could you give dht-blocklist.diff a try?

Tested with r12540. It's definitely a patch I would use as it appears to be 99.5% effective. The reason for the occasional failure for addresses that are known to be in a loaded blocklist might be because you are only testing for incoming messages (right?) while the application I am using to test against is able to block outgoing packets as well.

I'm having second thoughts about the effectiveness rating. The ladies can't be just a little bit pregnant, right? Some of the IPs I still have to block even for outgoing are pretty nasty customers.

Last edited 10 years ago by x190 (previous) (diff)

comment:19 Changed 10 years ago by jordan

  • Milestone changed from None Set to 2.33
  • Resolution set to fixed
  • Status changed from new to closed
  • Type changed from Bug to Enhancement

dht-blocklist.diff applied to trunk by r12544.

Closing ticket as fixed because this completes what's possible in libtransmission. If you want to block the outgoing packets too, you'll need to talk to jch about the DHT module.

comment:20 follow-up: Changed 10 years ago by Username

  • Resolution fixed deleted
  • Status changed from closed to reopened

I would reopen this ticket because:

From what I understand, DHT is all about asking known nodes about lists of other, "more suitable" nodes. So even if you'll discard all incoming packets from banned nodes, you still may receive banned nodes in reply lists as "more suitable nodes". Then these nodes could be used by DHT and outbound traffic could be generated to banned nodes, making your ban attempts rather futile. Since I found no code which deals with this issue, I believe mentioned issue was not fixed. Granted that there is lots of strange nodes doing crappy things ranging from trying to hijack certain hashes and direct most traffic to self up to conducting various attacks (flood, malformed packets, etc).

comment:21 Changed 10 years ago by jordan

  • Resolution set to fixed
  • Status changed from reopened to closed

If I understand you correctly, I already covered this in comment:19.

comment:22 in reply to: ↑ 20 Changed 10 years ago by x190

Replying to Username:

I would reopen this ticket because:

From what I understand, DHT is all about asking known nodes about lists of other, "more suitable" nodes. So even if you'll discard all incoming packets from banned nodes, you still may receive banned nodes in reply lists as "more suitable nodes". Then these nodes could be used by DHT and outbound traffic could be generated to banned nodes, making your ban attempts rather futile. Since I found no code which deals with this issue, I believe mentioned issue was not fixed. Granted that there is lots of strange nodes doing crappy things ranging from trying to hijack certain hashes and direct most traffic to self up to conducting various attacks (flood, malformed packets, etc).

Thanks for showing an interest in this ticket. IMO, a couple of very important improvements have been made to the way DHT works in the last couple of days and it would complete that effort if this issue was addressed. I intend to craft a ticket directed to jch to try to get him to show an interest in filtering outgoing communication. We might have to offer him a weekend at Club Med in order to catch his attention though. :)

comment:23 Changed 10 years ago by jch

Please see my comments on #4371.

-- jch

comment:24 Changed 10 years ago by jordan

  • Milestone changed from 2.33 to 2.40
  • Resolution fixed deleted
  • Status changed from closed to reopened
  • Summary changed from Blocklists loaded in Transmission should be used to filter DHT communication. to DHT and blocklists are incompatible

15:48:55 <+jch> BentMyWookie?: https://trac.transmissionbt.com/ticket/4371#comment:5

15:49:12 <+jch> I wonder why you still bother CC-ing me.

17:09:49 <@jordan> jch: he thought (as did I) that you'd be interested in tickets related to DHT

17:09:57 <@jordan> jch: https://trac.transmissionbt.com/ticket/4371#comment:6

17:10:44 <+jch> Yes.

17:10:54 <+jch> I am interested in tickets related to DHT.

17:11:15 <+jch> I took my time to explain why applying the blocklist to the DHT is a bad idea.

17:11:33 <+jch> ...only to discover you had already made the change to apply the blocklist.

17:14:35 <@jordan> my question is that /if/ transmission is going to support blocklists, it doesn't make sense to support it except in certain places.

17:15:10 <+jch> jordan: and my point is that the Kademlia algorithm supposes transitive connectivity.

17:15:23 <+jch> I suggest disabling the DHT when there is a blocklist.

17:15:52 <@jordan> jch: that's a good idea. or even better, the reverse

17:16:08 <@jordan> disabling blocklists when DHT is enabled

17:16:26 <@jordan> DHT is more important, so it should take precedence

17:17:11 <+jch> jordan: you'll get flak for that (diregarding users' blocklists).

17:17:24 <+jch> But yeah, I fully agree that blocklists are snake oil.

17:19:44 <@jordan> jch: thanks, I'm going to implement your suggestion

17:20:14 <@jordan> jch: do you mind if I quote this discussion in the ticket to document the rationale?

17:20:20 <+jch> jordan: no, go ahead.

Last edited 10 years ago by jordan (previous) (diff)

comment:25 Changed 10 years ago by jordan

17:47:10 < CIA-73> jordan * r12575 libtransmission/tr-udp.c: (trunk libT) "DHT and blocklists are incompatible" -- revert r12544.

comment:26 follow-up: Changed 10 years ago by x190

"jordan> jch: that's a good idea. or even better, the reverse jordan> disabling blocklists when DHT is enabled DHT is more important, so it should take precedence "

Don't even think about it! That idea is right up there with the one about being unable to uninitialize DHT during a session.

"jch> But yeah, I fully agree that blocklists are snake oil."

Of course Mr. jch, if you want to block undesirables from your network you use iptables, which is not an option for most of us.

Blocklists are an excellent security tool in the same way as locking your vehicle and dwelling and their use should be enhanced. Going forward, I hope Transmission's users' needs will take precedence over various developers' mindsets.

comment:27 in reply to: ↑ 26 Changed 10 years ago by jch

Blocklists are an excellent security tool in the same way as locking your vehicle and dwelling and their use should be enhanced.

Nobody is taking blocklists away from you.

The point is, I don't see a good way to get the DHT to work at the same time as a blocklist. So you get to choose: blocklist or DHT, but not both.

If you see a good way to get blocklists to work with the DHT without endangering the DHT, I'll be glad to review your patches.

Of course Mr. jch, if you want to block undesirables from your network you use iptables, which is not an option for most of us.

Heh. I run an open wifi network, which I share with my neighbours, and my firewall is to prevent the people inside from harming the Internet, not the other way around.

-- jch

comment:28 follow-ups: Changed 10 years ago by x190

" Heh. I run an open wifi network, which I share with my neighbours, and my firewall is to prevent the people inside from harming the Internet, not the other way around. -- jch"

So if you are running your Hekate application and someone sends you a continuous stream of corrupt packets or if you connect to an IP known to be used by malicious hackers, malware peddlers, bad peers, and other assorted unsavory types, you just sit there and say "Give me your best shot!"? Somehow, I think that you too would be reaching for the "Get Lost" button in short order.

"Nobody is taking blocklists away from you."

Hopefully no, not even when using DHT. Totally, unnecessary. But did you have to scare Jordan so badly that he momentarily couldn't think straight? :)

"The point is, I don't see a good way to get the DHT to work at the same time as a blocklist. So you get to choose: blocklist or DHT, but not both.

If you see a good way to get blocklists to work with the DHT without endangering the DHT, I'll be glad to review your patches."

I think you're being overly protective of your kademilian brainchild. Dropping a tiny percentage of total potential communication should not break such a robust network. I believe Jordan's patch for libtransmission/tr-udp.c, was an excellent first step and perhaps you could reconsider and allow at least that protection to stand.

comment:29 in reply to: ↑ 28 Changed 10 years ago by jch

Replying to x190:

if you connect to an IP known to be used by malicious hackers, malware peddlers, bad peers,

...drug dealers, paedophiles, and free software developers.

-- jch

comment:30 in reply to: ↑ 28 Changed 10 years ago by jch

Replying to x190:

I think you're being overly protective of your kademilian brainchild. Dropping a tiny percentage of total potential communication should not break such a robust network.

If people only blacklisted one or two IPs that actually cause trouble, it wouldn't be a problem. If Transmission automatically blacklisted IPs that it has noticed are causing problems, it wouldn't be a problem.

The trouble is with people downloading random third party blocklists on the web, and importing them into Transmission. Such blocklists typically block a fair part of the IPv4 address space (the more the better), which would cause inefficiency in Kademlia.

To give you an idea of the efforts we're already doing to work around buggy nodes (including ones that apply a blocklist to the DHT), a Kademlia search is supposed to search for just n=8 nodes. All actual implementations search for a larger number of nodes m (in Transmission, m=12), in the hope that at least n among the m will behave according to spec. Searching for 12 nodes instead of just 8 takes time and uses up extra network traffic, but with the current state of the DHT, there's just no way around it.

-- jch

comment:31 follow-ups: Changed 10 years ago by x190

jch>I'll be glad to review your patches.

Here's one.

static void
dht_bootstrap(void *closure)
          ... 
            port = ntohs(port);
            if( !tr_sessionIsAddressBlocked( cl->session, &addr ) ) {
                tr_dhtAddNode(cl->session, &addr, port, 1);
                tr_ninf( "DHT", "Node added in dhtAddNode" );
            } 
            else if( tr_sessionIsAddressBlocked( cl->session, &addr ) ) {
                tr_ninf( "DHT", "Node blocked in AddNode" );
            }
        }

Actually, got that tr_ninf message to fire. And Kademilia survived without a scratch. So proud of self! :D (...waiting now for the big put down. :lol:)

On a somewhat more serious note, "int tr_dhtAnnounce(tr_torrent *tor, int af, tr_bool announce)" might be one of the more important areas to filter as it generates the most hits. How does one extract a filterable address type out of that code block?

Here's the blow-by-blow with 20 loaded blocklists.


2011-07-23 05:22:46 -0600 tr-dht.c:173 [Info] DHT: Node blocked in AddNode

2011-07-23 05:22:41 -0600 tr-dht.c:170 [Info] DHT: Node added in dhtAddNode
2011-07-23 05:22:43 -0600 tr-dht.c:170 [Info] DHT: Node added in dhtAddNode
2011-07-23 05:22:49 -0600 tr-dht.c:170 [Info] DHT: Node added in dhtAddNode
2011-07-23 05:22:51 -0600 tr-dht.c:170 [Info] DHT: Node added in dhtAddNode
2011-07-23 05:22:52 -0600 tr-dht.c:170 [Info] DHT: Node added in dhtAddNode
2011-07-23 05:22:55 -0600 tr-dht.c:170 [Info] DHT: Node added in dhtAddNode
2011-07-23 05:22:58 -0600 tr-dht.c:170 [Info] DHT: Node added in dhtAddNode
2011-07-23 05:23:01 -0600 tr-dht.c:170 [Info] DHT: Node added in dhtAddNode
2011-07-23 05:23:13 -0600 tr-dht.c:170 [Info] DHT: Node added in dhtAddNode
2011-07-23 05:23:29 -0600 tr-dht.c:170 [Info] DHT: Node added in dhtAddNode
2011-07-23 05:23:48 -0600 tr-dht.c:170 [Info] DHT: Node added in dhtAddNode
2011-07-23 05:23:59 -0600 tr-dht.c:170 [Info] DHT: Node added in dhtAddNode
2011-07-23 05:24:21 -0600 tr-dht.c:170 [Info] DHT: Node added in dhtAddNode
2011-07-23 05:24:44 -0600 tr-dht.c:170 [Info] DHT: Node added in dhtAddNode
2011-07-23 05:25:00 -0600 tr-dht.c:170 [Info] DHT: Node added in dhtAddNode
2011-07-23 05:25:21 -0600 tr-dht.c:170 [Info] DHT: Node added in dhtAddNode
2011-07-23 05:25:29 -0600 tr-dht.c:170 [Info] DHT: Node added in dhtAddNode
2011-07-23 05:25:37 -0600 tr-dht.c:170 [Info] DHT: Node added in dhtAddNode
2011-07-23 05:25:47 -0600 tr-dht.c:170 [Info] DHT: Node added in dhtAddNode
2011-07-23 05:26:03 -0600 tr-dht.c:170 [Info] DHT: Node added in dhtAddNode
2011-07-23 05:26:17 -0600 tr-dht.c:170 [Info] DHT: Node added in dhtAddNode
2011-07-23 05:26:35 -0600 tr-dht.c:170 [Info] DHT: Node added in dhtAddNode
2011-07-23 05:26:50 -0600 tr-dht.c:170 [Info] DHT: Node added in dhtAddNode
2011-07-23 05:27:00 -0600 tr-dht.c:170 [Info] DHT: Node added in dhtAddNode
2011-07-23 05:27:14 -0600 tr-dht.c:170 [Info] DHT: Node added in dhtAddNode
2011-07-23 05:27:29 -0600 tr-dht.c:170 [Info] DHT: Node added in dhtAddNode
2011-07-23 05:27:40 -0600 tr-dht.c:170 [Info] DHT: Node added in dhtAddNode
2011-07-23 05:28:01 -0600 tr-dht.c:170 [Info] DHT: Node added in dhtAddNode
2011-07-23 05:28:16 -0600 tr-dht.c:170 [Info] DHT: Node added in dhtAddNode
2011-07-23 05:28:34 -0600 tr-dht.c:170 [Info] DHT: Node added in dhtAddNode
2011-07-23 05:28:54 -0600 tr-dht.c:170 [Info] DHT: Node added in dhtAddNode
2011-07-23 05:29:10 -0600 tr-dht.c:170 [Info] DHT: Node added in dhtAddNode
2011-07-23 05:29:18 -0600 tr-dht.c:170 [Info] DHT: Node added in dhtAddNode
2011-07-23 05:29:30 -0600 tr-dht.c:170 [Info] DHT: Node added in dhtAddNode
2011-07-23 05:29:47 -0600 tr-dht.c:170 [Info] DHT: Node added in dhtAddNode
2011-07-23 05:30:03 -0600 tr-dht.c:170 [Info] DHT: Node added in dhtAddNode
2011-07-23 05:30:12 -0600 tr-dht.c:170 [Info] DHT: Node added in dhtAddNode
2011-07-23 05:30:35 -0600 tr-dht.c:170 [Info] DHT: Node added in dhtAddNode
2011-07-23 05:30:45 -0600 tr-dht.c:170 [Info] DHT: Node added in dhtAddNode
2011-07-23 05:31:27 -0600 tr-dht.c:170 [Info] DHT: Node added in dhtAddNode
2011-07-23 05:31:44 -0600 tr-dht.c:170 [Info] DHT: Node added in dhtAddNode
2011-07-23 05:32:00 -0600 tr-dht.c:170 [Info] DHT: Node added in dhtAddNode
2011-07-23 05:32:23 -0600 tr-dht.c:170 [Info] DHT: Node added in dhtAddNode
2011-07-23 05:32:33 -0600 tr-dht.c:170 [Info] DHT: Node added in dhtAddNode
2011-07-23 05:32:48 -0600 tr-dht.c:170 [Info] DHT: Node added in dhtAddNode
2011-07-23 05:33:08 -0600 tr-dht.c:170 [Info] DHT: Node added in dhtAddNode
2011-07-23 05:33:24 -0600 tr-dht.c:170 [Info] DHT: Node added in dhtAddNode
2011-07-23 05:33:37 -0600 tr-dht.c:170 [Info] DHT: Node added in dhtAddNode
2011-07-23 05:33:54 -0600 tr-dht.c:170 [Info] DHT: Node added in dhtAddNode
2011-07-23 05:34:15 -0600 tr-dht.c:170 [Info] DHT: Node added in dhtAddNode
2011-07-23 05:34:33 -0600 tr-dht.c:170 [Info] DHT: Node added in dhtAddNode
+++

How is one dropped bad node in dozens of good ones going to break the lady? I'm actually seeing excellent node building times (150 nodes in 20 min. & all incoming firewalled).

And, for added incentive, T now uses 40% less real memory! :)

2175 Transmission sl 1.8 5 28.9 MB Intel (64 bit) 28.56

Last edited 10 years ago by x190 (previous) (diff)

comment:32 in reply to: ↑ 31 ; follow-up: Changed 10 years ago by jordan

Replying to x190:

And, for added incentive, T now uses 40% less real memory! :)

That doesn't make sense. Why do you think dropping 1 node saved 40% of Transmission's memory use?

comment:33 in reply to: ↑ 32 Changed 10 years ago by x190

Replying to jordan:

Replying to x190:

And, for added incentive, T now uses 40% less real memory! :)

That doesn't make sense. Why do you think dropping 1 node saved 40% of Transmission's memory use?

Sheesh, I really don't know! Maybe, it's a side effect of blocking some of the characters listed in comment:29. :)

comment:34 follow-up: Changed 10 years ago by Username

The point is, I don't see a good way to get the DHT to work at the same time as a blocklist.

So you get to choose: blocklist or DHT, but not both.

Why the hell on the Earth I have to make such a weird choice? There are plenty of P2P apps capable of filtering both "dht" and "tcp client" without making silly choices. So this indicates there is no real technical reasons to make this choice. Not to mention that I can foresee bugs like "dht does not works?!!!111". That would be WEIRDEST bug fix I ever seen.

Usually if I'm loading blocklist, I would expect absolutely no traffic to those blocked nodes but no other functionality loss. DHT is a robust thing and inability to exchange traffic with some certain nodes would not actually affect things too much - it will count as if they went offline which is fairly common case anyway.

Sure, I can use iptables, but loading large lists is not very straightforward operation, you will need ipset extension (else firewall performance would be awful) which has only been merged into recent mainline kernels (you have to patch older kernels, yikes) and, heck, there are OSes who do not deliver such a decent firewalling options at all. On other hand, p2p app can easily deal with it once and for all OSes in more convenient manner. Lots of P2P apps are doing so and I doubt that refusal to communicate with corrupt node is worse than do it, then spend time (spreading corrupted nodes on the way) wasting lots of time to get own buckets working (since P2P flooders often do not bother with fully correct protocol implementation and rather abusing it to mass-inject corrupt neighbours by always offering them as "closest nodes", regardless of anything).

comment:35 in reply to: ↑ 31 Changed 10 years ago by jch

Replying to x190:

jch>I'll be glad to review your patches.

Here's one.

static void
dht_bootstrap(void *closure)
          ... 
            port = ntohs(port);
            if( !tr_sessionIsAddressBlocked( cl->session, &addr ) ) {
                tr_dhtAddNode(cl->session, &addr, port, 1);
                tr_ninf( "DHT", "Node added in dhtAddNode" );
            } 
            else if( tr_sessionIsAddressBlocked( cl->session, &addr ) ) {
                tr_ninf( "DHT", "Node blocked in AddNode" );
            }
        }

Thank you very much for your contribution. Unfortunately, your patch will not prevent communication with blacklisted nodes.

If you look carefully at what this patch does, it merely prevents bootstrapping from blacklisted nodes. This will slow down bootstrapping of the DHT somewhat, but will not prevent us from learning about those nodes from other nodes after we've finished bootstrapping at communicating with them at that point.

Jordan proposed a different patch, which simply prevents receiving any traffic from blacklisted nodes. This patch is wrong too, since it doesn't prevent us from sending traffic to those nodes. This would lead to the situation in which we send requests to blacklisted nodes but ignore their replies, hence resend the request, ad nauseam.

The obvious fix -- preventing both sending and receiving to/from blacklisted nodes -- is also wrong, since it won't prevent getting blacklisted nodes into our routing tables, which will cause us to attempt sending traffic to them.

A better solution would be to hook the blacklist at a lower level, so that we never get blacklisted nodes into our routing tables. However, this would lead to the situation in which the set of live nodes is different from the point of view of different nodes, which is not something the Kademlia algorithm is designed to handle.

Looking forward to your explanation of why I'm wrong,

-- jch

comment:36 Changed 10 years ago by jch

Folks, may I please request that you keep this discussion on a strictly technical level? I am really not interested in getting into a shouting match with either of you.

-- jch

comment:37 Changed 10 years ago by Username

This would lead to the situation in which we send requests to blacklisted nodes but ignore their replies, hence resend the request, ad nauseam

Hmm... agree, this is a bad idea. Extra traffic which is both parasitic and reveals us to bad guys. Proper solution to block both in and out traffic (and maybe filter nodes on adding? Or it will self-correct due to inability to send traffic?).

Also I haven't got idea why x190 wants to filter out bootstrap phase and only it. Maybe he got a really nasty node in his dht.dat which sends nasty neighbours, gets most of buckets filled with malicious nodes and then they're bombarding his client with some garbage? In any case such approach as banning peers on bootstrap seems to be futile.

comment:38 in reply to: ↑ 34 Changed 10 years ago by jordan

Replying to Username:

There are plenty of P2P apps capable of filtering both "dht" and "tcp client" without making silly choices.

Could you name a few BitTorrent clients that use blocklists to block DHT traffic? I'm not aware of any.

Proper solution to block both in and out traffic

That's an improvement over blocking incoming traffic (which was reverted in r12575) but doesn't address jch's concern about different nodes having different sets of live nodes.

Last edited 10 years ago by jordan (previous) (diff)

comment:39 follow-up: Changed 10 years ago by Username

Could you name a few BitTorrent? clients that use blocklists to block DHT traffic? I'm not aware of any.

I can name at least aMule and eMule from what I learned well. These are not torrent, but they do not fundamentally different in their idea: there is UDP-based DHT to find the things and "legacy TCP client" from pre-DHT age to transfer data. So overall idea is quite the same. And btw, they did a really decent job on improving their DHT and counter-measuring vs most annoying attacks. Well, ed2k TCP client, file transfer style and priorities are weird (legacy stuff of ed2k). However, their DHT implementation is quite impressive and uses certain interesting ideas to protect itself from bad guys.

To name the few ideas I found reasonable in their DHT:
1) They limit number of known nodes per /24 subnet to just a few. Excessive nodes from same subnets are discarded ("banned", if you prefer). This makes certain attacks much harder, i.e. attacker is no longer able to allocate bunch of IPs in a single fast network campus/datacenter and mass-spoof zounds of malicious nodes at affordable price with low efforts. This makes large-scale attacks on DHT much harder because attacker have to allocate dozens of non-clustered IPs over whole world and this is about to be expensive in terms of efforts and money. On the downside, this may affect test or private deployments in small nets, but in this case some custom actions required anyway, so not a really big deal.
2) When they searching for key, they track outcoming DHT requests and would only accept DHT reply packets with extra nodes only if they asked for. This prevents bad guys from easy injecting of arbitrary nodes onto buckets. The downside is memory consumed by tracking data, but in normal scenario client should not have much of pending requests.
3) If I remember well, they also temp-ban nodes sending too much of unrecognised or malformed packets. This seems to be a good idea: communicating with malicious or broken clients does not leads to good results anyway. Just a waste of resources.
4) They publishing all the way, not to just to few ("k") of "most suitable nodes" found at the end of iterative process. That is it: key is published to each and every node encountered during search for that key (those nodes who believes it's really way too far could elect to forward key into better node they know and do not store "far" key themself). This trick does not allows "focused" attacks when attackers are trying to knock-down closest peers by overloading them (btw if you're "lucky" and have hash close to attacked one, you may face serious amounts of traffic and overall load). This also makes it harder to completely jam target key with fake "closest nodes" because key tolerance is rather wide and attackers should setup unreasonable number of nodes over whole keyspace to overtake key anyhow reliably, which is further complicated by 1). From what I know it also could reduce average number of iterations ("hops") since searches tend to go the same ways, hence early hit on "not so close" node becomes possible.

but doesn't address jch's concern about different nodes having different sets of live nodes.

I do not see any real reasons for these concerns: in a real world, traffic could (and would) fail to travel one direction or another due to routing issues, excessively smart filters/firewalls, outages, etc. So in real world peers could have somewhat different views due to network conditions. Furthermore, torrent clients are running different DHT implementations and it's hard to expect they all operate in exactly same way. Actually, DHT could even survive complete split into 2 pieces (when routing to large part of network lost) and would transparently merge into one DHT again when routing restores. This structure appears to be robust and flexible. It's hard to completely broke it, at very most you can degrade it's performance a bit. Jch is absolutely right that he cares about this. However, corrupt nodes do not care about this and it's a big question what would degrade network performance more: communicating with malicious nodes who would try to do as much damage as possible to you and DHT as whole or erroneously banning few good nodes together with harmful attack nodes. Latter equals to usual disconnection or IP routing failure and is not something that DHT can't handle. The real issue could happen if everyone would turn on some silly blocklist which cluelessly bans half of the planet. However you have blocklist option off by default. So you could expect only some people to use blocklist and only very few of them would use overzealous blocklists banning noticeable amounts of useful nodes. As for me it not looks like serious harm, while aggressive nodes could do a reasonable damage (after all they pursue this goal for some reasons).

P.s.: it would be nice if your client provides a better debug logs (including blocklist hits, malformed packets, etc).

P.p.s: sure, I can grind in other clients sources as well, if you really want me to. However, I see no reasons why people should stick self to "torrent clients only". Good (or bad) ideas could be encountered in all kinds of software.

comment:40 Changed 10 years ago by x190

Why make everything such an unproductive battle? Transmission already has DHT blocklist support in a release version. I don't hear people crying in the streets, or in the forum either, that their DHT no longer functions. If problems arise in the future, they can be appropriately dealt with. We're not doing a manned Mars shot here!

Good: "Jordan proposed [EDIT: ''implemented''] a different patch, which simply prevents receiving any traffic from blacklisted nodes."

Needs patch: "A better [EDIT: ''important complementary''] solution would be to hook the blacklist at a lower level, so that we never get blacklisted nodes into our routing tables."

comment:41 follow-up: Changed 10 years ago by jordan

but doesn't address jch's concern about different nodes having different sets of live nodes.

I do not see any real reasons for these concerns

jch could answer this much better than I can, but http://srhea.net/papers/ntr-worlds05.pdf discusses some of the issues.

Transmission already has DHT blocklist support in a release version. I don't hear people crying in the streets, or in the forum either, that their DHT no longer functions.

Conversely, this ticket has been open for five months and I only see two people pushing for applying blocklists to DHTs.

comment:42 in reply to: ↑ 41 Changed 10 years ago by jch

Replying to jordan:

http://srhea.net/papers/ntr-worlds05.pdf discusses some of the issues.

This paper is about Chord, not about Kademlia. Kademlia is notoriously more difficult to analyse than Chord, and I don't know of any published analyses of Kademlia's behaviour in the presence of non-transitive communication. I'd be glad to hear of any papers on the subject.

-- jch

comment:43 in reply to: ↑ 39 ; follow-up: Changed 10 years ago by jch

Replying to Username:

I'm not sure how that relates to this ticket, but here we go.

I can name at least aMule and eMule from what I learned well.

I am not aware of a written description of eMule's "Kad" protocol. I'd be cautious of relying too much on forum postings.

1) They limit number of known nodes per /24 subnet to just a few.

That's certainly a good idea, assuming that they limit that to the larger buckets. (It's wrong to do that in the two or three smallest buckets.)

2) When they searching for key, they track outcoming DHT requests and would only accept DHT reply packets with extra nodes only if they asked for.

Azureus does that too. It's possible to implement it without extra memory by encoding a cryptographic nonce within the request id. We're currently encoding different extra data in the request id, though.

I'm not sure what attacks this is supposed to prevent, since our "prefer oldest" policy should be resistant to the attack you doutline.

3) If I remember well, they also temp-ban nodes sending too much of unrecognised or malformed packets.

The former is not a good idea, since it prevents extensions to the protocol (it would have prevented us from doing µTP). The latter we already do.

4) They publishing all the way, not to just to few ("k") of "most suitable nodes" found at the end of iterative process.

I hope you misunderstood, because that's completely bogus.

I do not see any real reasons for these concerns: in a real world, traffic could (and would) fail to travel one direction or another due to routing issues, excessively smart filters/firewalls, outages, etc. So in real world peers could have somewhat different views due to network conditions.

Sure. 8-redundant Kademlia, which is what we implement, is able to resist to a moderate amount of such breakage, with reduced performance. Increasing the amount of breakage does not seem like a good idea, since it decreases the performance further. The point here is that it doesn't just decrease performance for you -- it decreases performance and increases load for everyone. Hence my stand -- if you're not willing to play by the rules, you're not allowed to join the game.

Perhaps I'm overly cautious. Perhaps introducing a few million nodes that break the rules will not have a drastic effect on performance. I'd be glad to be convinced of that.

-- jch

Last edited 10 years ago by jch (previous) (diff)

comment:44 Changed 10 years ago by x190

If someone would produce a patch we could get on with the testing. :)

comment:45 Changed 10 years ago by x190

Some additional perspective:

•Transmission 2.33 has been on the market for 5 days now with "* Apply blocklists towards DHT communication" in place.

As of this post I'm getting as many as 17 nodes reporting back from each announce. DHT appears healthy to me. (firewalled, 165 nodes).

comment:46 Changed 10 years ago by jch

Please have a look at r12586 and r12587.

-- jch

comment:47 follow-up: Changed 10 years ago by x190

Sorry, I'm totally confused. :-)

Testing on r12595 from .dmg, as apparently livings124 broke xcode on 10.6. How do I get it to use my blocklists?

/* This must be provided by the user. */
int dht_blacklisted(const struct sockaddr *sa, int salen);

I have no idea what that means. What must I provide? How do I get that function to use my loaded blocklists?

int
dht_blacklisted(const struct sockaddr *sa UNUSED, int salen UNUSED)
{
    return 0;
}

The following is the function from session.h that the rest of the code uses.

bool         tr_sessionIsAddressBlocked( const tr_session        * session,
                                         const struct tr_address * addr );

comment:48 in reply to: ↑ 47 ; follow-up: Changed 10 years ago by jch

Replying to x190:

Sorry, I'm totally confused. :-)

Am I surprised.

/* This must be provided by the user. */
int dht_blacklisted(const struct sockaddr *sa, int salen);

I have no idea what that means.

you're only supposed to use this functionality if you know what you are doing. With this patch, I've made adding blocklist functionality very easy, but no further instructions will be given -- if you're not able to work it out for yourself, you're not the intended audience.

Hint: read the comment.

-- jch

Last edited 10 years ago by jch (previous) (diff)

comment:49 in reply to: ↑ 48 ; follow-up: Changed 10 years ago by x190

Replying to jch:

Replying to x190:

Hint: read the comment.

-- jch

This one? "/* This must be provided by the user. */" Okay, I'll get to work directly on that, err I mean this.

My main concern at this point is that Transmission's development team be open and honest with their users when security related features are added then removed or when networking behavior is radically altered.

Torrent client users and all internet users have always had options for blocklist use. Iptables, Moblock, Peerguardian, and Norton Firewall are but a few of them. I think it was great that Transmission attempted to extend that capability within the application.

Last edited 10 years ago by x190 (previous) (diff)

comment:50 in reply to: ↑ 49 Changed 10 years ago by jch

Replying to x190:

This one?

No, the one in r12587.

-- jch

comment:51 Changed 10 years ago by x190

Networking tools should be designed to work well in the real world, because if they are not, then it will soon be the tools that are upgraded or redesigned, as the real world will not long be held back by the inadequacies of its technology.

"We choose to go to the Moon..."

---JFK

Last edited 10 years ago by x190 (previous) (diff)

comment:52 follow-up: Changed 10 years ago by jordan

  • Milestone changed from 2.40 to None Set
  • Resolution set to invalid
  • Status changed from reopened to closed

Judging from those comments, I don't think there are any more action items to do in this ticket. Transmission now has hooks for people who want to explore the direction.

comment:53 in reply to: ↑ 52 Changed 10 years ago by x190

Replying to jordan:

Transmission now has hooks for people who want to explore the direction.

Done, done, and done! Patch available, if needed (for a small donation, of course). :-)

DHT is functioning very well without unwanted bad nodes.

2011-08-07 05:35:14 -0600 tr-dht.c:674 [Debug] Transmission: Node blocked in dht_blacklisted. 192.168.1.1:17009
2011-08-07 05:35:14 -0600 tr-dht.c:674 [Debug] Transmission: Node blocked in dht_blacklisted. 213.10.10.74:7236
2011-08-07 05:35:14 -0600 tr-dht.c:674 [Debug] Transmission: Node blocked in dht_blacklisted. 80.240.214.228:51413

Getting 174 good nodes in a normal amount of time.

EDIT: Please note the blocking of 192.168.1.1:17009. This guy kept at it, using several different ports. By blocking this type of questionable behavior, it seems to me that the DHT network might actually be strengthened.

Could someone please review my code (see forum "Coding Questions")? I would appreciate this ticket being changed back to it's original premise and re-opened for further consideration.

Last edited 10 years ago by x190 (previous) (diff)

comment:54 Changed 10 years ago by x190

Why is it not a good idea to block the following type of activity in DHT?

2011-08-21 00:16:46 -0600 tr-dht.c:652 [Debug] DHT: Node blocked in dht_blacklisted. 203.82.95.74:16955
2011-08-21 00:16:46 -0600 tr-dht.c:652 [Debug] DHT: Node blocked in dht_blacklisted. 203.82.95.74:21545
2011-08-21 00:16:46 -0600 tr-dht.c:652 [Debug] DHT: Node blocked in dht_blacklisted. 203.82.95.74:14739
2011-08-21 00:16:46 -0600 tr-dht.c:652 [Debug] DHT: Node blocked in dht_blacklisted. 203.82.95.74:31501
2011-08-21 00:16:46 -0600 tr-dht.c:652 [Debug] DHT: Node blocked in dht_blacklisted. 203.82.95.74:29121
2011-08-21 00:16:46 -0600 tr-dht.c:652 [Debug] DHT: Node blocked in dht_blacklisted. 203.82.95.74:5989
2011-08-21 00:16:46 -0600 tr-dht.c:652 [Debug] DHT: Node blocked in dht_blacklisted. 203.82.95.74:17780
2011-08-21 00:16:46 -0600 tr-dht.c:652 [Debug] DHT: Node blocked in dht_blacklisted. 203.82.95.74:17063
2011-08-21 00:16:48 -0600 tr-dht.c:652 [Debug] DHT: Node blocked in dht_blacklisted. 203.82.95.74:9527
2011-08-21 00:16:48 -0600 tr-dht.c:652 [Debug] DHT: Node blocked in dht_blacklisted. 203.82.95.74:1286
2011-08-21 00:17:02 -0600 tr-dht.c:652 [Debug] DHT: Node blocked in dht_blacklisted. 203.82.95.74:1286
2011-08-21 00:17:02 -0600 tr-dht.c:652 [Debug] DHT: Node blocked in dht_blacklisted. 203.82.95.74:7313
2011-08-21 00:17:02 -0600 tr-dht.c:652 [Debug] DHT: Node blocked in dht_blacklisted. 203.82.95.74:3845
2011-08-21 00:17:02 -0600 tr-dht.c:652 [Debug] DHT: Node blocked in dht_blacklisted. 203.82.95.74:12617
2011-08-21 00:17:02 -0600 tr-dht.c:652 [Debug] DHT: Node blocked in dht_blacklisted. 203.82.95.74:20983
2011-08-21 00:17:02 -0600 tr-dht.c:652 [Debug] DHT: Node blocked in dht_blacklisted. 203.82.95.74:12287
2011-08-21 00:17:02 -0600 tr-dht.c:652 [Debug] DHT: Node blocked in dht_blacklisted. 203.82.95.74:15079
2011-08-21 00:17:18 -0600 tr-dht.c:652 [Debug] DHT: Node blocked in dht_blacklisted. 203.82.95.74:15079
2011-08-21 00:17:35 -0600 tr-dht.c:652 [Debug] DHT: Node blocked in dht_blacklisted. 203.82.95.74:15079
2011-08-21 00:18:30 -0600 tr-dht.c:652 [Debug] DHT: Node blocked in dht_blacklisted. 203.82.95.74:23417
2011-08-21 00:18:55 -0600 tr-dht.c:652 [Debug] DHT: Node blocked in dht_blacklisted. 203.82.95.74:5963
2011-08-21 00:18:57 -0600 tr-dht.c:652 [Debug] DHT: Node blocked in dht_blacklisted. 203.82.95.74:11018
2011-08-21 00:23:42 -0600 tr-dht.c:652 [Debug] DHT: Node blocked in dht_blacklisted. 203.82.94.127:2842
2011-08-21 00:24:58 -0600 tr-dht.c:652 [Debug] DHT: Node blocked in dht_blacklisted. 203.82.94.127:12201
2011-08-21 00:24:58 -0600 tr-dht.c:652 [Debug] DHT: Node blocked in dht_blacklisted. 203.82.94.127:7787
2011-08-21 00:25:49 -0600 tr-dht.c:652 [Debug] DHT: Node blocked in dht_blacklisted. 203.82.94.127:7787
2011-08-21 00:26:10 -0600 tr-dht.c:652 [Debug] DHT: Node blocked in dht_blacklisted. 203.82.94.127:7787
2011-08-21 00:26:27 -0600 tr-dht.c:652 [Debug] DHT: Node blocked in dht_blacklisted. 203.82.94.127:7787
2011-08-21 00:44:18 -0600 tr-dht.c:652 [Debug] DHT: Node blocked in dht_blacklisted. 203.82.95.74:17780
2011-08-21 00:44:18 -0600 tr-dht.c:652 [Debug] DHT: Node blocked in dht_blacklisted. 203.82.95.74:5989
2011-08-21 00:44:18 -0600 tr-dht.c:652 [Debug] DHT: Node blocked in dht_blacklisted. 203.82.95.74:17063
2011-08-21 00:44:18 -0600 tr-dht.c:652 [Debug] DHT: Node blocked in dht_blacklisted. 203.82.95.74:20131
2011-08-21 00:44:18 -0600 tr-dht.c:652 [Debug] DHT: Node blocked in dht_blacklisted. 203.82.95.74:15079
2011-08-21 00:44:18 -0600 tr-dht.c:652 [Debug] DHT: Node blocked in dht_blacklisted. 203.82.95.74:7313
2011-08-21 00:44:18 -0600 tr-dht.c:652 [Debug] DHT: Node blocked in dht_blacklisted. 203.82.95.74:3845
2011-08-21 00:44:18 -0600 tr-dht.c:652 [Debug] DHT: Node blocked in dht_blacklisted. 203.82.95.74:12617
2011-08-21 00:44:18 -0600 tr-dht.c:652 [Debug] DHT: Node blocked in dht_blacklisted. 203.82.95.74:20983
2011-08-21 00:44:18 -0600 tr-dht.c:652 [Debug] DHT: Node blocked in dht_blacklisted. 203.82.95.74:12287
2011-08-21 00:44:18 -0600 tr-dht.c:652 [Debug] DHT: Node blocked in dht_blacklisted. 203.82.95.74:27745
2011-08-21 00:44:18 -0600 tr-dht.c:652 [Debug] DHT: Node blocked in dht_blacklisted. 203.82.95.74:16698
2011-08-21 00:44:37 -0600 tr-dht.c:652 [Debug] DHT: Node blocked in dht_blacklisted. 203.82.95.74:16698
2011-08-21 00:44:55 -0600 tr-dht.c:652 [Debug] DHT: Node blocked in dht_blacklisted. 203.82.95.74:16698
2011-08-21 00:45:15 -0600 tr-dht.c:652 [Debug] DHT: Node blocked in dht_blacklisted. 203.82.95.74:16698
2011-08-21 00:45:16 -0600 tr-dht.c:652 [Debug] DHT: Node blocked in dht_blacklisted. 203.82.95.74:31501
2011-08-21 00:45:16 -0600 tr-dht.c:652 [Debug] DHT: Node blocked in dht_blacklisted. 203.82.95.74:21545
2011-08-21 00:45:16 -0600 tr-dht.c:652 [Debug] DHT: Node blocked in dht_blacklisted. 203.82.95.74:14739
2011-08-21 00:45:39 -0600 tr-dht.c:652 [Debug] DHT: Node blocked in dht_blacklisted. 203.82.95.74:23417
2011-08-21 00:45:39 -0600 tr-dht.c:652 [Debug] DHT: Node blocked in dht_blacklisted. 203.82.95.74:16955
2011-08-21 00:45:40 -0600 tr-dht.c:652 [Debug] DHT: Node blocked in dht_blacklisted. 203.82.95.74:26667
2011-08-21 00:45:40 -0600 tr-dht.c:652 [Debug] DHT: Node blocked in dht_blacklisted. 203.82.95.74:4457
2011-08-21 00:52:13 -0600 tr-dht.c:652 [Debug] DHT: Node blocked in dht_blacklisted. 203.82.94.127:12201
2011-08-21 00:52:32 -0600 tr-dht.c:652 [Debug] DHT: Node blocked in dht_blacklisted. 203.82.94.127:12201
2011-08-21 00:52:51 -0600 tr-dht.c:652 [Debug] DHT: Node blocked in dht_blacklisted. 203.82.94.127:12201
2011-08-21 00:53:13 -0600 tr-dht.c:652 [Debug] DHT: Node blocked in dht_blacklisted. 203.82.94.127:12201
2011-08-21 00:53:13 -0600 tr-dht.c:652 [Debug] DHT: Node blocked in dht_blacklisted. 203.82.94.127:8536
2011-08-21 01:09:58 -0600 tr-dht.c:652 [Debug] DHT: Node blocked in dht_blacklisted. 203.82.95.74:18953
2011-08-21 01:09:58 -0600 tr-dht.c:652 [Debug] DHT: Node blocked in dht_blacklisted. 203.82.95.74:17962
2011-08-21 01:09:59 -0600 tr-dht.c:652 [Debug] DHT: Node blocked in dht_blacklisted. 203.82.95.74:11933
2011-08-21 01:09:59 -0600 tr-dht.c:652 [Debug] DHT: Node blocked in dht_blacklisted. 203.82.95.74:27883
2011-08-21 01:10:01 -0600 tr-dht.c:652 [Debug] DHT: Node blocked in dht_blacklisted. 203.82.95.74:20818
2011-08-21 01:10:21 -0600 tr-dht.c:652 [Debug] DHT: Node blocked in dht_blacklisted. 203.82.95.74:20818
2011-08-21 01:10:21 -0600 tr-dht.c:652 [Debug] DHT: Node blocked in dht_blacklisted. 203.82.95.74:27745
2011-08-21 01:10:21 -0600 tr-dht.c:652 [Debug] DHT: Node blocked in dht_blacklisted. 203.82.95.74:7313
2011-08-21 01:10:21 -0600 tr-dht.c:652 [Debug] DHT: Node blocked in dht_blacklisted. 203.82.95.74:3845
2011-08-21 01:10:21 -0600 tr-dht.c:652 [Debug] DHT: Node blocked in dht_blacklisted. 203.82.95.74:20983
2011-08-21 01:10:21 -0600 tr-dht.c:652 [Debug] DHT: Node blocked in dht_blacklisted. 203.82.95.74:12287
2011-08-21 01:10:21 -0600 tr-dht.c:652 [Debug] DHT: Node blocked in dht_blacklisted. 203.82.95.74:6482
2011-08-21 01:10:21 -0600 tr-dht.c:652 [Debug] DHT: Node blocked in dht_blacklisted. 203.82.95.74:15079
2011-08-21 01:10:42 -0600 tr-dht.c:652 [Debug] DHT: Node blocked in dht_blacklisted. 203.82.95.74:15079
2011-08-21 01:10:59 -0600 tr-dht.c:652 [Debug] DHT: Node blocked in dht_blacklisted. 203.82.95.74:15079
2011-08-21 01:11:19 -0600 tr-dht.c:652 [Debug] DHT: Node blocked in dht_blacklisted. 203.82.95.74:20131
2011-08-21 01:11:19 -0600 tr-dht.c:652 [Debug] DHT: Node blocked in dht_blacklisted. 203.82.95.74:5989
2011-08-21 01:11:19 -0600 tr-dht.c:652 [Debug] DHT: Node blocked in dht_blacklisted. 203.82.95.74:17063
2011-08-21 01:11:19 -0600 tr-dht.c:652 [Debug] DHT: Node blocked in dht_blacklisted. 203.82.95.74:2939
2011-08-21 01:11:55 -0600 tr-dht.c:652 [Debug] DHT: Node blocked in dht_blacklisted. 203.82.95.74:14739
2011-08-21 01:11:55 -0600 tr-dht.c:652 [Debug] DHT: Node blocked in dht_blacklisted. 203.82.95.74:3668
2011-08-21 01:11:55 -0600 tr-dht.c:652 [Debug] DHT: Node blocked in dht_blacklisted. 203.82.95.74:21545
2011-08-21 01:11:55 -0600 tr-dht.c:652 [Debug] DHT: Node blocked in dht_blacklisted. 203.82.95.74:10681
2011-08-21 01:36:50 -0600 tr-dht.c:652 [Debug] DHT: Node blocked in dht_blacklisted. 203.82.95.74:16737
2011-08-21 01:37:11 -0600 tr-dht.c:652 [Debug] DHT: Node blocked in dht_blacklisted. 203.82.95.74:16737
2011-08-21 01:37:11 -0600 tr-dht.c:652 [Debug] DHT: Node blocked in dht_blacklisted. 203.82.95.74:27745
2011-08-21 01:37:11 -0600 tr-dht.c:652 [Debug] DHT: Node blocked in dht_blacklisted. 203.82.95.74:7313
2011-08-21 01:37:11 -0600 tr-dht.c:652 [Debug] DHT: Node blocked in dht_blacklisted. 203.82.95.74:3845
2011-08-21 01:37:11 -0600 tr-dht.c:652 [Debug] DHT: Node blocked in dht_blacklisted. 203.82.95.74:20983
2011-08-21 01:37:11 -0600 tr-dht.c:652 [Debug] DHT: Node blocked in dht_blacklisted. 203.82.95.74:15079
2011-08-21 01:37:11 -0600 tr-dht.c:652 [Debug] DHT: Node blocked in dht_blacklisted. 203.82.95.74:6482
2011-08-21 01:37:11 -0600 tr-dht.c:652 [Debug] DHT: Node blocked in dht_blacklisted. 203.82.95.74:18503
2011-08-21 01:37:11 -0600 tr-dht.c:652 [Debug] DHT: Node blocked in dht_blacklisted. 203.82.95.74:24642
2011-08-21 01:37:11 -0600 tr-dht.c:652 [Debug] DHT: Node blocked in dht_blacklisted. 203.82.95.74:2446
2011-08-21 01:37:32 -0600 tr-dht.c:652 [Debug] DHT: Node blocked in dht_blacklisted. 203.82.95.74:2446
2011-08-21 01:37:48 -0600 tr-dht.c:652 [Debug] DHT: Node blocked in dht_blacklisted. 203.82.95.74:2446
2011-08-21 01:38:20 -0600 tr-dht.c:652 [Debug] DHT: Node blocked in dht_blacklisted. 203.82.95.74:21545
2011-08-21 01:38:20 -0600 tr-dht.c:652 [Debug] DHT: Node blocked in dht_blacklisted. 203.82.95.74:3668
2011-08-21 01:38:20 -0600 tr-dht.c:652 [Debug] DHT: Node blocked in dht_blacklisted. 203.82.95.74:10681
2011-08-21 01:38:20 -0600 tr-dht.c:652 [Debug] DHT: Node blocked in dht_blacklisted. 203.82.95.74:9346
2011-08-21 01:38:40 -0600 tr-dht.c:652 [Debug] DHT: Node blocked in dht_blacklisted. 203.82.95.74:12787
2011-08-21 02:03:43 -0600 tr-dht.c:652 [Debug] DHT: Node blocked in dht_blacklisted. 203.82.95.74:6482
2011-08-21 02:03:43 -0600 tr-dht.c:652 [Debug] DHT: Node blocked in dht_blacklisted. 203.82.95.74:7313
2011-08-21 02:03:43 -0600 tr-dht.c:652 [Debug] DHT: Node blocked in dht_blacklisted. 203.82.95.74:3845
2011-08-21 02:03:43 -0600 tr-dht.c:652 [Debug] DHT: Node blocked in dht_blacklisted. 203.82.95.74:20983
2011-08-21 02:03:43 -0600 tr-dht.c:652 [Debug] DHT: Node blocked in dht_blacklisted. 203.82.95.74:15079
2011-08-21 02:03:43 -0600 tr-dht.c:652 [Debug] DHT: Node blocked in dht_blacklisted. 203.82.95.74:18503
2011-08-21 02:03:43 -0600 tr-dht.c:652 [Debug] DHT: Node blocked in dht_blacklisted. 203.82.95.74:24642
2011-08-21 02:03:43 -0600 tr-dht.c:652 [Debug] DHT: Node blocked in dht_blacklisted. 203.82.95.74:2154
2011-08-21 02:04:00 -0600 tr-dht.c:652 [Debug] DHT: Node blocked in dht_blacklisted. 203.82.95.74:2154
2011-08-21 02:04:19 -0600 tr-dht.c:652 [Debug] DHT: Node blocked in dht_blacklisted. 203.82.95.74:2154
2011-08-21 02:04:20 -0600 tr-dht.c:652 [Debug] DHT: Node blocked in dht_blacklisted. 203.82.95.74:9346
2011-08-21 02:04:20 -0600 tr-dht.c:652 [Debug] DHT: Node blocked in dht_blacklisted. 203.82.95.74:3668
2011-08-21 02:04:20 -0600 tr-dht.c:652 [Debug] DHT: Node blocked in dht_blacklisted. 203.82.95.74:10681
2011-08-21 02:04:20 -0600 tr-dht.c:652 [Debug] DHT: Node blocked in dht_blacklisted. 203.82.95.74:4802
2011-08-21 02:04:43 -0600 tr-dht.c:652 [Debug] DHT: Node blocked in dht_blacklisted. 203.82.95.74:4802
2011-08-21 02:04:45 -0600 tr-dht.c:652 [Debug] DHT: Node blocked in dht_blacklisted. 203.82.95.74:16643
2011-08-21 02:05:18 -0600 tr-dht.c:652 [Debug] DHT: Node blocked in dht_blacklisted. 203.82.95.74:6164
2011-08-21 02:05:18 -0600 tr-dht.c:652 [Debug] DHT: Node blocked in dht_blacklisted. 203.82.95.74:14232
2011-08-21 02:07:41 -0600 tr-dht.c:652 [Debug] DHT: Node blocked in dht_blacklisted. 203.82.94.127:7482
2011-08-21 02:07:42 -0600 tr-dht.c:652 [Debug] DHT: Node blocked in dht_blacklisted. 203.82.94.127:26098
2011-08-21 02:07:42 -0600 tr-dht.c:652 [Debug] DHT: Node blocked in dht_blacklisted. 203.82.94.127:21764
2011-08-21 02:07:44 -0600 tr-dht.c:652 [Debug] DHT: Node blocked in dht_blacklisted. 203.82.94.127:21764
2011-08-21 02:09:44 -0600 tr-dht.c:652 [Debug] DHT: Node blocked in dht_blacklisted. 203.82.94.127:21764
2011-08-21 02:10:01 -0600 tr-dht.c:652 [Debug] DHT: Node blocked in dht_blacklisted. 203.82.94.127:21764
2011-08-21 02:29:31 -0600 tr-dht.c:652 [Debug] DHT: Node blocked in dht_blacklisted. 203.82.95.74:13481
2011-08-21 02:29:48 -0600 tr-dht.c:652 [Debug] DHT: Node blocked in dht_blacklisted. 203.82.95.74:13481
2011-08-21 02:29:49 -0600 tr-dht.c:652 [Debug] DHT: Node blocked in dht_blacklisted. 203.82.95.74:24642
2011-08-21 02:29:49 -0600 tr-dht.c:652 [Debug] DHT: Node blocked in dht_blacklisted. 203.82.95.74:7313
2011-08-21 02:29:49 -0600 tr-dht.c:652 [Debug] DHT: Node blocked in dht_blacklisted. 203.82.95.74:3845
2011-08-21 02:29:49 -0600 tr-dht.c:652 [Debug] DHT: Node blocked in dht_blacklisted. 203.82.95.74:20983
2011-08-21 02:29:49 -0600 tr-dht.c:652 [Debug] DHT: Node blocked in dht_blacklisted. 203.82.95.74:15079
2011-08-21 02:29:49 -0600 tr-dht.c:652 [Debug] DHT: Node blocked in dht_blacklisted. 203.82.95.74:2154
2011-08-21 02:29:49 -0600 tr-dht.c:652 [Debug] DHT: Node blocked in dht_blacklisted. 203.82.95.74:17875
2011-08-21 02:29:49 -0600 tr-dht.c:652 [Debug] DHT: Node blocked in dht_blacklisted. 203.82.95.74:9954
2011-08-21 02:30:07 -0600 tr-dht.c:652 [Debug] DHT: Node blocked in dht_blacklisted. 203.82.95.74:9954
2011-08-21 02:30:25 -0600 tr-dht.c:652 [Debug] DHT: Node blocked in dht_blacklisted. 203.82.95.74:9954
2011-08-21 02:30:46 -0600 tr-dht.c:652 [Debug] DHT: Node blocked in dht_blacklisted. 203.82.95.74:4929
2011-08-21 02:30:47 -0600 tr-dht.c:652 [Debug] DHT: Node blocked in dht_blacklisted. 203.82.95.74:24758
2011-08-21 02:30:47 -0600 tr-dht.c:652 [Debug] DHT: Node blocked in dht_blacklisted. 203.82.95.74:14232
2011-08-21 02:30:47 -0600 tr-dht.c:652 [Debug] DHT: Node blocked in dht_blacklisted. 203.82.95.74:8620
2011-08-21 02:30:47 -0600 tr-dht.c:652 [Debug] DHT: Node blocked in dht_blacklisted. 203.82.95.74:10681
2011-08-21 02:30:47 -0600 tr-dht.c:652 [Debug] DHT: Node blocked in dht_blacklisted. 203.82.95.74:3668
2011-08-21 02:30:47 -0600 tr-dht.c:652 [Debug] DHT: Node blocked in dht_blacklisted. 203.82.95.74:27385
2011-08-21 02:32:09 -0600 tr-dht.c:652 [Debug] DHT: Node blocked in dht_blacklisted. 203.82.95.74:16643
2011-08-21 02:32:09 -0600 tr-dht.c:652 [Debug] DHT: Node blocked in dht_blacklisted. 203.82.95.74:28889
2011-08-21 02:32:09 -0600 tr-dht.c:652 [Debug] DHT: Node blocked in dht_blacklisted. 203.82.95.74:11202
2011-08-21 02:35:17 -0600 tr-dht.c:652 [Debug] DHT: Node blocked in dht_blacklisted. 203.82.94.127:20954
2011-08-21 02:37:03 -0600 tr-dht.c:652 [Debug] DHT: Node blocked in dht_blacklisted. 203.82.94.127:20954
2011-08-21 02:38:12 -0600 tr-dht.c:652 [Debug] DHT: Node blocked in dht_blacklisted. 203.82.94.127:20954
2011-08-21 02:38:12 -0600 tr-dht.c:652 [Debug] DHT: Node blocked in dht_blacklisted. 203.82.94.127:3932
2011-08-21 02:38:30 -0600 tr-dht.c:652 [Debug] DHT: Node blocked in dht_blacklisted. 203.82.94.127:3932
2011-08-21 02:54:58 -0600 tr-dht.c:652 [Debug] DHT: Node blocked in dht_blacklisted. 203.82.95.74:28767
2011-08-21 02:54:58 -0600 tr-dht.c:652 [Debug] DHT: Node blocked in dht_blacklisted. 203.82.95.74:14616

In addition, I'm seeing several iana-private addresses using rotating ports and also several node blocks with address/port blank.

I don't wish to upset anyone, but I'm going to re-open this ticket so that comment:53 and comment:54 can be evaluated. Also, I would appreciate the code posted in the forum under "Coding Questions" being reviewed. It appears to work flawlessly. Thanks!:-)

comment:55 Changed 10 years ago by x190

  • Resolution invalid deleted
  • Status changed from closed to reopened
  • Summary changed from DHT and blocklists are incompatible to DHT and blocklists are incompatible ( or perhaps they are compatible? )

comment:56 Changed 10 years ago by livings124

  • Cc jch@… added

cc'ing jch

comment:57 in reply to: ↑ 43 ; follow-up: Changed 10 years ago by Username3

Replying to jch:

Replying to Username:

Ugh, sorry, I forgot my password and even has failed to remember email I used during registration (shame on me). And I even did something wrong when trying to register as username2, so it has fails to log in. So I will use nickname Username3 instead at this point :(.

I can name at least aMule and eMule from what I learned well.

I am not aware of a written description of eMule's "Kad" protocol. I'd be cautious of relying too much on forum postings.

Well, I'm not sure if they have up to date documents about this. I used source and verbose debug logs some time ago to get an idea how it works. Basically, it's still Kademlia under the hood. Though it has been somewhat improved to reduce numbers of hops and hardened against various attacks. But overall idea of Kademlia still remains the same. So I believe many things and findings would apply to other Kademlia implementations as well. Just because all Kademlia flavors have more or less the same idea under the hood.

Why I refer to aMule/eMule?

Well, first because they understand DHT, they implemented DHT ages ago, and improved it a lot.

Second is because they have a reasonably good logging/debugging facility which is capable of logging protocol things I may want to see, it's output could be adjusted to see only messages I actually want to, and packet handlers are placed in quite obvious places, have a reasonably simple logic, do not use cool libs and therefore it's easy to see what's going on. So, a serious packet-level protocol tracing and debugging was much easier for me on aMule than it would be with, say, Transmission.

Transmission, unfortunately, not seems to offer any adequate logging/tracing facilities so I can't examine things at protocol packets level in real time in convenient ways and it not looks like if it is really easy to add such things either, since protocol messages are handled in various places, there are some tricky hacks to deal with several kinds of packets on one socket, and at the end, DHT (which handles DHT part of UDP protocol) turns out to be a completely separate lib, so even if LibT would provide any suitable facility to do all kinds of logging including hardcore verbose debug, there is no easy and universal way to add a packet level tracer/logger into DHT anyway with a reasonable amount of efforts. Correct me if I'm wrong, though.

To be honest I would be happy to spent some time and effort tracing protocol implementations and logic, analyzing results and possibly finding some bugs or ways to improve things, etc. However, at this point, T looks hostile to this kind of activity, unfortunately. So even if I wish to refer to T, I would face serious troubles on this way. So at the moment I'm able to do only minimal and occasional observations on DHT with Transmission.

1) They limit number of known nodes per /24 subnet to just a few.

That's certainly a good idea, assuming that they limit that to the larger buckets.

(It's wrong to do that in the two or three smallest buckets.)

There is no clue to keep such nodes in any buckets, actually. Such nodes are often extremely hostile and broken and would only spread their attacking friends in all replies to any request. No matter what you ask them, they return their friends (who only replies with such attacking nodes as well). Those seems to try to poison Kademlia using team play of all fake nodes to improve efficiency of attack. So as for me, limit of nodes per subnet should occur without any exception to effectively stop such kinds of species. Else you're risking to accumulate such bastards in your buckets if they'll happen to hit a "proper" part of keyspace. On other hand, a bit worse network efficiency due to offering "a bit less optimal" nodes with another IPs seems to be much smaller issue than to spread hostile nodes over network. I do not see reasons to make any exceptions, especially taking into account that DHT IDs are not really related to IPs, so idea of excepting someone from sanity checks is strange and very unlikely to do anything good but will relax pressure on evildoers and allow them to try to craft IDs in specific ways to inject self to chosen "nodes of interests", for example. On other hand, sanity checking makes their efforts fruitless while only doing a minimal harm to correct DHT operations, uncomparable with harm from bunch of attacking nodes spreading self in broken bastardized ways.

From what I seen, recent *Mule implementations may only elect to uplift this restriction when they detect they're running in a small networks with only a few nodes. This is to make it possible to run a private DHT over LANs, allowing people to do smaller scale private DHT deployments. In this case nodes are expected to be from same subnets for sure, and uplifting of limits is reasonable, but T's DHT not seems to even consider such usecases anyway.

2) When they searching for key, they track outcoming DHT requests and would only accept DHT reply packets with extra nodes only if they asked for.

Azureus does that too. It's possible to implement it without extra memory by encoding a cryptographic nonce within the request id. We're currently encoding different extra data in the request id, though.

Whoa, really :) though wouldn't it somewhat increase CPU load? Anyway, this countermeasure supposed to prevent hostile nodes from aggressively injecting into "proper" nodes in efficient ways, so improved network operation probably outweighs anyway.

I'm not sure what attacks this is supposed to prevent, since our "prefer oldest" policy should be resistant to the attack you doutline.

Wrong. Imagine this:

  • We've just started and are filling our buckets.
  • We've got evlidoing node during this process.
  • We asked evildoer for more nodes.
  • He replies us with even more attacking nodes.
  • We may insert them into our buckets
  • We will start replying with evildoing nodes to other nodes (not to mention we'll use broken nodes for own searches and will receive broken nodes in their replies, wasting time on dead end searches).
  • Then, "prefer oldest" policy further secures these nodes in our buckets. It's not wrong on it's own, but granted that most of attacking nodes are placed in some campus nets or datacenters capable of providing required amounts of traffic, those nodes are fairly persistent and we probably going to prefer them, save them before sessions, etc. So this could even play against us, actually.

From what I understand, the purpose of "prefer oldest" policy is to

  • Relax pressure on overloaded nodes so if node is jammed by DHT traffic, Kademlia would automatically correct this by removing overloaded node from some buckets as "dead", reducing traffic to that node and reducing load on that node, shifting it to other nodes instead.
  • Direct more traffic to stable nodes who can cope with load, hence improving overall network reliability and speed. Obviously, powerful server on a gigabit channel can handle more traffic than old Pentium II notebook using 2.5G wireless. Kademlia uses this fact to improve, but when someone starts a well coordinated attack, this may help to attackers to divert traffic to attacking nodes as long as they can cope with such load.

Result: poison spreads and "proper" nodes would help malicious ones to poison network. Poisoned nodes could even be preferred over others due to stability (average server in datacenter is more stable than random ADSL/3G/wi-fi/wi-max/etc broadband user on dynamic IP). So, "prefer oldest" actually may play a bad joke. I guess that's why mentioned countermeasure techniques were invented to limit the harm.

3) If I remember well, they also temp-ban nodes sending too much of unrecognised or malformed packets.

The former is not a good idea, since it prevents extensions to the protocol (it would have prevented us from doing µTP). The latter we already do.

Well, they only ban those who sends really incorrect packets. Some unrecognized structures are okay as long as they do not try to violate protocol logic, overflow buffers by giving incorrect values, aggressively flood with packets, etc.

4) They publishing all the way, not to just to few ("k") of "most suitable nodes" found at the end of iterative process.

I hope you misunderstood, because that's completely bogus.

Maybe. But from what I understood, it has been discovered that if you'll search for some key, it's no matter what is your node, the search is likely to go through the same "ways" for all nodes and you're likely to deal with quite the same nodes along the way, so caching key on those nodes could save some hops if you're lucky without doing much harm. And let's admit, k of best nodes is not a really fixed thing. Network is dynamic thing. Some of nodes may become offline and some could become overloaded so "not so close" nodes would take responsibility for same key, that's how Kademlia designed. So in fact there could be more or less than k nodes keeping the same key anyway. So as for me, this behavior does not brutally violates nature of Kademlia and only makes some intentional extra publishing to make it harder to jam k of best nodes and maybe even save some hops on searches. Hence, as for me it appears as way to improve operations in some aspects. Though I guess this technique should be implemented network wide, not client wide. I.e. it only makes real sense when most nodes will play using same ruleset (*Mule can afford this since they are keeping over 90% of DHT). As for me it's hard to evaluate mixed network. Granted that only small fraction of nodes would play these rules, I do not see any real gain from doing so compared to efforts and unevaluated consequences.

etc. So in real world peers could have somewhat different views due to network conditions.

Sure. 8-redundant Kademlia, which is what we implement, is able to resist to a moderate amount of such breakage, with reduced performance.

But well, nodes who conduct attacks in a well-coordinated groups, running extremely broken implementations of DHT, the only purpose of which is to spread poison with maximum efficiency, are doing breakage either and unlike good nodes, they seems to be optimized for purposes of making as many harm as possible by running broken clients whose sole purpose it to divert maximum amount of traffic to broken nodes and spread this poison even further (which is predictable and on attacker's place I would do something like this for sure). That's where blacklists and sanity checks could actually improve things rather than make them worse. Proper nodes can't suffer large amount of breakage due to these actions. But all those guys running 1 000 000 fake nodes on very few servers located in campus/datacenter would feel really unhappy since such a small change to client makes them wasting their money with a very little effect and single blocklist entry completely knocks them off (at least for particular client using this list). Sure, in theory it's possible to ban innocent nodes as well. In real world, blocklists are usually not banning too many of good IPs most of time or they'll have bad reputation. After all, banning good IPs running torrent clients would also affect download speeds, making people unhappy.

Increasing the amount of breakage does not seem like a good idea, since it decreases

the performance further.

The problem is that I have reasons to believe that those who attacks networks on a purpose are doing damage in much larger amounts and they're much more efficient in doing it. Because it's their purpose to cause damage and they seems to do it very well compared to anything else. They do not have to care and can implement absolutely bastardized techniques to achieve their purposes. Sometimes you have to choose from two evil things and blacklists and countermeasures seems to be less evil than packs of corrupt nodes conducting coordinated attacks.

The point here is that it doesn't just decrease performance for you -- it decreases performance and increases load for everyone.

Sure. But P2P flooders are doing quite the same and their methods are usually far worse than just not responding to someone's query. As for me, it's better ban 'em rather than help 'em to divert traffic to corrupt nodes.

Hence my stand -- if you're not willing to play by the rules, you're not allowed to join the game.

That's exactly why I may want blacklist in DHT. I want ability to knock off evildoers who are unwilling to play this game using proper ruleset and rather pollute network and degrade it. Helping them do so by including them into my buckets is stupid and destructive. As I would spent some of my resources spreading corrupt nodes. It's surely not what I want. And blocklist more or less fits into inherent properties of DHTs. Wrecked nodes who only attacks network like poison and does not behave correctly are not. In ideal word you're right. In real one, we have to choose from two evil options. As for me, I would prefer to stick to "less evil" one and in my opinion, countermeasures and blocklists are less evil than harmful nodes. And I sure that just not respond to suspicious peer is better than include it into my buckets and spread it to the world, making network poisoning easier for evildoers since they used me to amplify their attack so I'm wasting my resources to spread their corrupt nodes.

Perhaps I'm overly cautious.

That's actually good. The only thing is that you have to evaluate all options and you may want to understand that in real world, networks could be a somewhat hostile environment. So you may want ability to defend self and keep network running well. Look, we still need police who would "blocklist" some "bad nodes" even if in ideal world we should not need them at all.

Perhaps introducing a few million nodes that break the rules will not have a drastic effect on performance. I'd be glad to be convinced of that.

At least, based on my knowledge of DHT I can imagine how to conduct destructive behavior in DHT with a maximum efficiency: use cancer-like mode - coordinated attack by group of nodes spreading themselves in all replies and never giving "good" nodes in replies at all. Do not bother with 100% correct client: if you have to damage, who cares?! Only process requests necessary to remain in buckets (so others are spreading your infection further, sharing their resources for your evil task) and always answer that there are only your nodes are existing in the world (actually that's a final purpose of DHT takeover, isn't it?). Now switching back into "good guy" mode... It looks like if some nodes are doing exactly something like this. If you have diverted lots of traffic to yourself, you may affect DHT in a noticeable ways, definitely, because many nodes will contact you and noticeable amount of requests will be processed by you and you're who controls the replies. And you can't completely forbid such nodes to enter game. But you can raise the bar and allow people to "call police" and "jail 'em" if they feel it's required.

comment:58 Changed 10 years ago by x190

Thank you for your well-reasoned support Username 3. I'm including the following link as it illustrates the fact that blocklist users expect and assume that DHT communication will be filtered. https://forum.transmissionbt.com/viewtopic.php?f=4&t=12071

comment:59 follow-ups: Changed 10 years ago by livings124

Strange that someone with the name "Username3" has a mailinator email address (as do "Username" and "Username2"), and wrote a massive message arguing a point only you care about x190.

Last edited 10 years ago by livings124 (previous) (diff)

comment:60 in reply to: ↑ 59 ; follow-up: Changed 10 years ago by x190

Replying to livings124:

Strange that someone with the name "Username3" has a mailinator email address (as do "Username" and "Username2"), and wrote a massive message arguing a point only you care about x190.

Never heard of mailinator before but it sounds like an excellent idea for a security aware or non-spam loving person. Username 3 is Username -- see the beginning of his post. He is just replying to jch's comments about his original post. As far as who cares about the issue, I think blocklist users assume DHT traffic is already filtered (which it currently is for incoming messages). See comment:58 link.

Has anyone checked the code in the forum ("Coding Questions") for technical correctness?

comment:61 in reply to: ↑ 60 ; follow-up: Changed 10 years ago by jordan

Replying to x190:

Has anyone checked the code in the forum ("Coding Questions") for technical correctness?

Technical correctness doesn't enter into it until after jch and Username have settled the fundamentals. If the idea is bad, it doesn't matter whether or not the code is good.

comment:62 in reply to: ↑ 61 ; follow-up: Changed 10 years ago by x190

Replying to jordan:

Replying to x190:

Has anyone checked the code in the forum ("Coding Questions") for technical correctness?

Technical correctness doesn't enter into it until after jch and Username have settled the fundamentals. If the idea is bad, it doesn't matter whether or not the code is good.

In a free society, you too have a vote. jch has already stated that he knows of no studies having been done regarding the effect of using blocklists in this context. He did however provide the "hook", albeit with caveats. In addition, his code already recognizes the need to blacklist certain nodes, hence the value "DHT_MAX_BLACKLISTED" which has always been in dht.c.

My question was why would it not be a good idea to additionally be able to block the type of questionable or outright wrong node behavior outlined in comment:53 and comment:54?

As far as the code goes, it was a moment of personal triumph to see it working so well after putting such a major effort into producing it. It may well be something unrepeatable for me, being an old retired postie with no formal programming training. :-)

comment:63 in reply to: ↑ 57 Changed 10 years ago by jch

I don't usually respond to sock puppets, but just this once, I'll make an exception.

Replying to Username3:

Why I refer to aMule/eMule?

[...]

Second is because they have a reasonably good logging/debugging facility which is capable of logging protocol things I may want to see, [...]

Transmission, unfortunately, not seems to offer any adequate logging/tracing facilities [...]

As usual, x190, you have no idea what you're speaking about. Had you so much as glanced at the code, you'd have noticed that my DHT has a very detailed logging facility, which is triggered by the code at line 278 in tr-dht.c.

--jch

comment:64 in reply to: ↑ 62 ; follow-up: Changed 10 years ago by jch

  • Resolution set to wontfix
  • Status changed from reopened to closed

Replying to x190:

In a free society, you too have a vote.

I vote that you f*ck off.

-- jch

comment:65 Changed 10 years ago by x190

Your confusion and crudeness speak for themselves.

comment:66 in reply to: ↑ 59 Changed 10 years ago by Username3

Strange that someone with the name "Username3" has a mailinator email address (as do "Username" and "Username2"), and wrote a massive message arguing a point only you care about x190.

That's because spamers have learned me some lessons by overflowing my mailboxes. Now their job will be a bit harder - if they'll discover this mailbox, they'll flood /dev/null, actually. Which is funny enough, I believe.

And I honestly believe that bug tracker is more than enough to communicate all things related to bugs, features and somesuch, so it should not be a major issue. As for Username, I've forgot my password and failed to remember mailbox I used during registration, hence failed to reset password (you should see logs with a dozen and half of failed password failures and then reset attempts, trying to guess mailinator mailbox I used). My fault, sure. I've cleared saved passwords in browser and was not able to remember password myself. Furthermore, looks like if I was not able to remember email I entered (except that it's mailinator). So it looks like if I locked myself out. Silly, sure.

So I was not able to recover my account. Shame but true. Then I attempted to register as Username2. Unfortunately something went wrong with trac and it has misbehaved during registration. So while user name has been allocated, I was never able to log in with username2 name at all (password fails + reset does not delivers email, I do not know what's wrong but it does not works). So I have re-registered as Username3 to show I'm still same person. Do you honestly believe that if I wanted to fool you, I would openly register 3 very similar names? This assumption is strange enough, isn't it? So I'm finding your reaction weird, actually. Maybe you should be a little less paranoid and more attentive to details? I guess even such a nuts like x190 should guess that registering 3 similar names to disguise self is a really bad idea :)

And well, as for me, it would be still interesting to read Jch answer on my post where I attempted to explain my views on DHTs and blocklists. In fact, DHTs are my topic of interests. So it has been quite interesting discussion for me, though I understand it's time consuming.

P.S. Dear x190, you're somewhat silly, so while such a funny disguise of my person could be funny, it's not as funny as it should be, granted that you behave in silly ways. Can I give you small hints, at least? Please, get one simple idea: dev's are working on code and you're not paying them salary, right? So, you can propose features or bugs. But that's up to dev's how to deal with them. Since that's their time and efforts. You can't order them what to do (this would violate their basic freedoms). You want to have a vote? Vote on what? Vote on what devs should be FORCED to do? Lol! Then I should be able to vote that x190 should give me $ 1 000 000 right now, isn't it? That's where your logic fails in a stupid way. The problem? Only one: I do not want to be associated with such a laughable stupidity. At least I'm able to understand that dev's are not my personal slaves. So if I dislike something, I can either argue my point of view in hope devs will agree it, or if this fails ... uh, I have to shut up and patch code myself. Calling vote on forcing devs to do something is futile, to say the least.

comment:67 Changed 10 years ago by x190

Deleted, as I really don't believe in making tickets " go AWOL". :)

Last edited 10 years ago by x190 (previous) (diff)

comment:68 Changed 10 years ago by x190

Ticket Summary:

  • This ticket was opened 7 months ago to address a blocklist sorting problem that was fixed by jordan in r11888. See comment:15.
  • At that point the ticket then morphed into one directly related to DHT, the summary now being "Blocklists loaded in Transmission should be used to filter DHT communication". This is a summary change to which I fully agreed but made no further comment. Please see comment:16 from 6 months ago.
  • 7 weeks ago jordan produced the "dht-blocklist.diff" patch which I tested as requested and although initially reasonably impressed with it, I soon realized it lacked support for outgoing connections which led to comment:19.
  • A lot has transpired in the last 7 weeks, some good, some not so good. On the plus side, we now have a way to add full DHT blocklist support from within Transmission. If this upsets anyone, it should not; remember, bittorrenters have always had access to, and do use, third party blocking software, so this is not a new concept, but simply ensures that the DHT implementation handles it properly.
  • When I recently re-opened this ticket, I was simply hoping to get some input on the fact that I was now able to block the surprising number of iana-private and other invalid addresses from unnecessarily filling the node list.
Last edited 6 years ago by x190 (previous) (diff)

comment:69 follow-up: Changed 10 years ago by x190

  • Summary changed from DHT and blocklists are incompatible ( or perhaps they are compatible? ) to Blocklists loaded in Transmission should be used to filter DHT communication.

Summary changed to reflect comment:16.

Last edited 10 years ago by x190 (previous) (diff)

comment:70 in reply to: ↑ 69 Changed 6 years ago by cfpp2p

If it's true it looks like x190 had excellent foresight.

https://torrentfreak.com/thousands-of-spies-are-watching-trackerless-torrents-151004/

comment:71 in reply to: ↑ 64 Changed 6 years ago by x190

Replying to jch:

Replying to x190:

In a free society, you too have a vote.

I vote that you f*ck off.

-- jch

Hey jch, how the f*ck are you, you old f*rt? :)

A patch is available, if wanted.

Last edited 6 years ago by x190 (previous) (diff)
Note: See TracTickets for help on using tickets.