RPC Protocol Specification
This document explains how to remotely, or locally, connect to a Transmission session programatically using the rpc protocol.
If you're looking to control a Transmission session non-programatically, you can either use the transmission-remote cli, or the Web Interface.
Versions before 1.30 used to connect to a Transmission session by writing directly into a socket file. Many not-so-up to date articles on setting up Transmission with a Web Interface (Clutch) may still refer to this technique.