[Ethereum] Signing a transaction: What is actually signed? [noob question]

I never found any proper documentation about the Recovery ID but I did talk with somebody on Reddit and they gave me my answer:

id = y1 & 1; // Where (x1,y1) = k x G;
if (s > curve.n / 2) id = id ^ 1; // Invert id if s of signature is over half the n

I had to modify the mbedtls library to pass back the Recovery ID but when I did I could generate transactions that Geth accepted 100% of the time.

The long explanation:

During signing, a point is generated (X, Y) called R and a number called S. R's X goes on to become r and S becomes s. In order to generate the Recovery ID you take the one's bit from Y. If S is bigger than half the curve's N parameter you invert that bit. That bit is the Recovery ID. Ethereum goes on to manipulate it to indicate compressed or uncompressed addresses as well as indicate what chain the transaction was signed for (so the transaction can't be replayed on another Ethereum chain that the private key might be present on). These modifications to the Recovery ID become v.

There's also a super rare chance that you need to set the second bit of the recovery id meaning the recovery id could in theory be 0, 1, 2, or 3. But there's a 0.000000000000000000000000000000000000373% of needing to set the second bit according to a question on Bitcoin.SE.

The transaction ID calculation is correct. For the signing, the arguments are still RLP encoded (the dummy "v" for EIP-155 is a bit special, see below).

Take this live transaction as an example. It has the following parameters.

nonce = 0 [RLP: 0x80]
gasPrice = 50000000000 wei (0x0BA43B7400) [RLP: 85 0BA43B7400]
gasLimit = 21000 (0x5208) [RLP: 82 5208]
to = 0x7917bc33eea648809c285607579c9919fb864f8f [RLP: 94 7917bc33eea648809c285607579c9919fb864f8f]
value = 1050000000000000 wei (0x03BAF82D03A000) [RLP: 87 03BAF82D03A000]
data = <empty> [RLP: 80]
v = 018080 (this is a place-holder value before signing, see EIP-155)

Then the input parameters for the hash is the RLP of the concatenation:

EB80850BA43B7400825208947917bc33eea648809c285607579c9919fb864f8f8703BAF82D03A00080018080

And the hash value for the signing is:

python3
>>> from Crypto.Hash import keccak
>>> keccak_hash=keccak.new(digest_bits=256)
>>> txn=bytearray.fromhex('EB80850BA43B7400825208947917bc33eea648809c285607579c9919fb864f8f8703BAF82D03A00080018080')
>>> keccak_hash.update(bytes(txn))
<Crypto.Hash.keccak.Keccak_Hash object at 0x10fb6e2e8>
>>> print(keccak_hash.hexdigest())

which gives the result

a4060d01d4add248db470b4121616cbe5b2015daf328809000ec9a1d0954d649

For the transaction ID, do Keccak hash on the final raw transaction bytes (RLP encoded as you mentioned, which are also readable from Etherscan) and it will give the same result as shown on Etherscan.