The keyfile follows the Web3 Secret Storage Definition.
I believe the parts of the keyfile that are not necessary for account recovers are the address and mac value. Technically the version, cipher, and kdf are not required since you could guess from the small set of possible choices for these.
Private key recovery
The password you provide is used to derive an encryption key by passing it through a key derivation function or KDF.
Once the encryption key has been derived the mac value can be used to verify the key (this step is optional)
The encryption key can then be used to decrypt the ciphertext which will return the private key.
Some of the links in the comments helped me get the answers to my questions. I have collected all that information combined with my own (recently acquired) knowledge of cryptography from the Coursera Cryptography-I class (for the answer to question number 3).
A1. The private key is never (or should never be) saved unencrypted on disk. It is generated from the info in the keystore file when the user enters the password.
A2. The public key can be generated from the private key. However, it is needed by other parties (that do not have the private key) to verify signatures by this account. These parties would derive it from a signature, as explained here: Get public key of any ethereum account Then, the signer's address is the last 20 bytes of the Keccak-256 hash of the public key. Presumably, that should match the from: address of the transaction.
A3. Here is a (somewhat long) answer about the structure and meaning of the private key's JSON representation:
I got a lot of this information from the wiki page: https://github.com/ethereum/wiki/wiki/Web3-Secret-Storage-Definition and added some of my own to make it more comprehensive.
All the information to decrypt the private key is in the crypto element. The cipher element identifies the symmetric encryption scheme used to encrypt the private key. In the given example, the scheme is aes-128-ctr, the Advanced Encryption Standard (aes) block cipher with a key length of 128 bits in the counter (ctr) mode of operation. Aside: The other mode of operation is cipher block chaining (CBC). cipherparams contains the values of any parameters used in the encryption. For AES-128-CTR, there is only one parameter, iv (Initialization Vector). ciphertext is the encrypted private key.
To decrypt the private key's ciphertext, we need the above values. We also need the symmetric encryption key, which was used to encrypt the private key. The encryption key is not shown in plain text either. It needs to be derived using the specified key derivation function (kdf) using the values of the parameters that the specified KDF needs (kdfparams). The KDF in the given example is pbkdf2 - a password-based KDF. The parameters for pbkdf2 are the password (to be entered by the user), a pseudo-random hashing function (prf) (in example, it is HMAC-SHA256 hmac-sha256), a salt value (salt), an iteration count (c) and the derived key length (dklen) in bytes. The key is derived by taking the concatenation of the password and salt values, and applying the hashing function c times. This is done because passwords selected by humans do not have sufficient entropy. To defend against that, a random salt value is used and also a slow hash function. (The number of iterations is to make the hash function slower. It does not make the key derivation any more secure (as some believe). It only makes it take longer so as to make brute-force attacks more difficult.)
After deriving the encryption key, it is verified using the MAC value (mac). The second block of 128 bits (16 bytes) is concatenated with the ciphertext value. The Keccak-256 hash of the resulting byte array should be equal to the mac value.
If the encryption key passes verification, then it is used to decrypt the ciphertext, using the decryption function of the specified cipher with the parameter values given in cipherparams. The result of this decryption is the plain text private key.
A4. I realized later that a new account should be created only on a node controlled by oneself. That is the reason that the personal API is enabled by default on IPC, but not on RPC, to prevent a new account creation request being accepted by a remote node.
Best Answer
The UTC file has the private key encrypted with your password.
To access the private key from the file I've used keythereum. Then you can use elliptic to derive the public key.