I know that the yellow paper does not specify how transactions are to be ordered in a block and this is up to the miner to decide. But I am interested in how this is practically handled (I assume not calling some rand function to get ordering). This answer to a related question suggests that gas price plays a role. How does, e.g. geth order transactions during the mining process?
[Ethereum] the default ordering of transactions during mining
blocksminingtransactions
Related Solutions
You've misunderstood the steps. But close!
Consensus is easily misunderstood.
The fundamental problem first solved by Bitcoin was how to reach eventual consensus among nodes that do not know or necessarily trust each other. Add money to the environment and it becomes an adversarial environment. This is in no way like the situation in a typical cluster of computers that do know each other and do trust each other.
We get a hint about how to solve it from transaction logs employed by databases. If one has a replayable log with all the inputs in the right order, then one can reconstruct a database. It's not important to have "the state" if you have the inputs that created that state.
There is a non-obvious challenge to deal with. Given physics, it is not possible for all nodes to learn about all transactions at the same time or even in the same order. It will (probably) never be possible to make everything as fast or faster than everything else.
In a trustful environment, that could be resolved with accurate, trustworthy timestamps that would help everyone enumerate the "correct" transaction order. In an adversarial environment with no authority, that doesn't work because no one's clock is considered more trustworthy than anyone else's.
So, how to order the transactions?
The mining process collapses that ambiguity. The "winning" miner earns the privilege of setting a de facto network transaction order, for one block. Importantly, this is not even an attempt to resolve the temporal order of transactions. In fact, gasPrice, is a way to queue jump. One can probably incentivize a miner to include a transaction sooner by attaching a high bid. Conversely, one can save on transaction fees by being patient.
Nothing happens anywhere until nodes receive news of transactions included in blocks. Transactions in blocks are well-ordered. The blocks themselves are well-ordered. So, the chain of blocks is a well-ordered set of all transactions that have happened.
Each full node, not just the miners, processes transactions completely, reaching their own conclusions about state changes. This is very much like replaying a transaction log because everyone agrees on the inputs and the order of those inputs. The functions are deterministic (strict requirement) so there can be no disagreement between well-functioning nodes at the same block height.
Blocks have time-stamps but transactions don't. Block time is the minimal resolution, and all that can be said is that all transactions in a block were mined at that block time, in the order of inclusion. Each transaction executed in the context left by the previous transaction.
Miners have non-trivial privilege. Working together they can censor transactions. They can play with timestamps and transaction order, if there is selfish benefit doing so. This can be important in contract design when factors such as "deadline" exist.
In any case, the consensus resolves transaction order. Nodes figure out the state for themselves.
Hope it helps.
p.s. There is a lot more going on in mining but I wanted to focus on your question.
I have read that PoW is about finding a nonce such that: hashOf(Block Data + Previous Block Data + Nonce) == A Hash With Desired Leading Zeros
I suppose you mean hashOf(Block Data + Previous Block Hash + Nonce)
.
And that once a miner has found this nonce, it can broadcast the nonce to the network where each miner can validate that it is the correct nonce by using the same function above.
No. Once a miner has found a nonce leading to a valid hash according to the current difficulty criteria, it should broadcast the block. Ethereum has a slightly more complicated and more memory oriented proof of work than what is usually used as an example, you can read more about it here but we can stick to the basic example.
But from my statement above I feel like if a Miner doesn't have a standard mechanism for ordering and including transactions that complies with at least 51% of the other miners then they will never be able to mine a block.
Correct me if I'm wrong but it seems that you think of the consensus in PoW as miners generally agreeing on what transactions should be contained in a block. To do so they would indeed need to have a shared rule about which transactions to include, and in what order. However, they would also need to all have the same knowledge of pending transactions. That assumption cannot hold due to network delays / message drops / etc...
So it's not about having the right transactions, or any transaction for that matter, a block without transactions is perfectly valid (not including the coinbase tx) as you can see for Bitcoin or Ethereum.
What the consensus is really about is twofold :
- What is the mainchain (longest chain rule for Bitcoin, GHOST for Ethereum)
- Valid block content (hash and transactions)
What we call Proof of Work, is in fact known as Nakamoto Consensus (PoW + chain selection rule).
So when a miner aim to create a new block, it selects some transactions (or none) while maximizing the fees for their personal gain. Those transactions may only be known by this miner and the participants that created them, after all, each transaction includes a signature and the actions that it does can be checked against the current state. The miner then works out the proof of work by varying the block nonce and, if successful, broadcast it to the network.
Other participants will eventually receive that block and basically checks for the following :
- Hash is valid and matches difficulty criteria.
- All tx are valid (no one spends more than it owns / tx nonce are in increasing order if one account has several tx included in the block)
It doesn't matter that participants learn the existence of some previously unknown transactions when the block is received, transactions are signed and their validity can be checked. Plus, a transaction doesn't really "exist" before being included in a block, in the sense that it has no effect whatsoever prior to that inclusion.
If that new block extends the main chain, the state transition is applied and miners will use that block as parent block for their next block creation attempt. And you have a new consensual view of the current state of the blockchain that never required a network wide view of the pending transactions.
I hope that answers your question.
Best Answer
Edit: 23 June 2017 - Added Parity details (see below)
Geth
For the vanilla Geth implementation, the
commitNewWork()
function in worker.go orders in the following way:i.e. Sorts by gas price and nonce value (and by owner account - see below).
There are two other approaches in the code which are both commented out, but may give a clue as to previous ideas (or give examples to miners who want to use their own implementation). They are:
SortByPriceAndNonce()
is defined in transaction.go:Once sorted, the transactions are further screened in
commitTransactions()
to remove any with "low gas".Parity
For Parity things are a little easier: there's a CLI option. This will, at least, allow ordering to be changed in some basic ways.