[Ethereum] Difference between Account Extended Private Key and BIP32 Extended Private Key

Some of the links in the comments helped me get the answers to my questions. I have collected all that information combined with my own (recently acquired) knowledge of cryptography from the Coursera Cryptography-I class (for the answer to question number 3).

A1. The private key is never (or should never be) saved unencrypted on disk. It is generated from the info in the keystore file when the user enters the password.

A2. The public key can be generated from the private key. However, it is needed by other parties (that do not have the private key) to verify signatures by this account. These parties would derive it from a signature, as explained here: Get public key of any ethereum account Then, the signer's address is the last 20 bytes of the Keccak-256 hash of the public key. Presumably, that should match the from: address of the transaction.

A3. Here is a (somewhat long) answer about the structure and meaning of the private key's JSON representation:

I got a lot of this information from the wiki page: https://github.com/ethereum/wiki/wiki/Web3-Secret-Storage-Definition and added some of my own to make it more comprehensive.

All the information to decrypt the private key is in the crypto element. The cipher element identifies the symmetric encryption scheme used to encrypt the private key. In the given example, the scheme is aes-128-ctr, the Advanced Encryption Standard (aes) block cipher with a key length of 128 bits in the counter (ctr) mode of operation. Aside: The other mode of operation is cipher block chaining (CBC). cipherparams contains the values of any parameters used in the encryption. For AES-128-CTR, there is only one parameter, iv (Initialization Vector). ciphertext is the encrypted private key.

To decrypt the private key's ciphertext, we need the above values. We also need the symmetric encryption key, which was used to encrypt the private key. The encryption key is not shown in plain text either. It needs to be derived using the specified key derivation function (kdf) using the values of the parameters that the specified KDF needs (kdfparams). The KDF in the given example is pbkdf2 - a password-based KDF. The parameters for pbkdf2 are the password (to be entered by the user), a pseudo-random hashing function (prf) (in example, it is HMAC-SHA256 hmac-sha256), a salt value (salt), an iteration count (c) and the derived key length (dklen) in bytes. The key is derived by taking the concatenation of the password and salt values, and applying the hashing function c times. This is done because passwords selected by humans do not have sufficient entropy. To defend against that, a random salt value is used and also a slow hash function. (The number of iterations is to make the hash function slower. It does not make the key derivation any more secure (as some believe). It only makes it take longer so as to make brute-force attacks more difficult.)

After deriving the encryption key, it is verified using the MAC value (mac). The second block of 128 bits (16 bytes) is concatenated with the ciphertext value. The Keccak-256 hash of the resulting byte array should be equal to the mac value.

If the encryption key passes verification, then it is used to decrypt the ciphertext, using the decryption function of the specified cipher with the parameter values given in cipherparams. The result of this decryption is the plain text private key.

A4. I realized later that a new account should be created only on a node controlled by oneself. That is the reason that the personal API is enabled by default on IPC, but not on RPC, to prevent a new account creation request being accepted by a remote node.

According to this github issue page it turns out the answer is the private key is not generated as part of the node structure but instead the result of HMACing the first 32 bytes of the the entropy. An example below

import hdkey from 'ethereumjs-wallet/hdkey';
import ethwallet from 'ethereumjs-wallet';
import ethtx from 'ethereumjs-tx';
import bip39 from 'bip39';

const mnemonic = bip39.generateMnemonic();
console.log(`Mnemonic: ${mnemonic}`);
const root = hdkey.fromMasterSeed(bip39.mnemonicToSeed(mnemonic));
const derivedNode = root.derivePath("m/44'/60'/0'/0");
const address = this.generateAddress(derivedNode);
console.log(`Private Key: ${root._hdkey.privateKey.toString('hex')}`);

The actual private key generation occurs in hdkey