Summary

It is limited by your computer's memory.

From my basic testing in What happens when a transaction nonce is too high?, I crashed geth with 400 x 64 transactions with a data payload of 4,500 bytes when running in 4 Gb RAM.

I am assuming that your question is asking what is the maximum number of transactions that can be stored in the transaction pool. If you are asking what is the maximum size of a single transaction, the answer is at Is there a limit for transaction size?.

Details

There is a pool for pending transactions and a queue for transactions that are not ready to be processed.

The data structure representing the transaction pool in geth can be seen at go-ethereum - tx_pool.go, lines 54 to 77:

// TxPool contains all currently known transactions. Transactions
// enter the pool when they are received from the network or submitted
// locally. They exit the pool when they are included in the blockchain.
//
// The pool separates processable transactions (which can be applied to the
// current state) and future transactions. Transactions move between those
// two states over time as they are received and processed.
type TxPool struct {
    config       *ChainConfig
    currentState stateFn // The state function which will allow us to do some pre checks
    pendingState *state.ManagedState
    gasLimit     func() *big.Int // The current gas limit function callback
    minGasPrice  *big.Int
    eventMux     *event.TypeMux
    events       event.Subscription
    localTx      *txSet
    mu           sync.RWMutex
    pending      map[common.Hash]*types.Transaction // processable transactions
    queue        map[common.Address]map[common.Hash]*types.Transaction

    wg sync.WaitGroup // for shutdown sync

    homestead bool
}

From stackoverflow.com - Maximum number of elements in map, the maximum number of elements that can be stored in a Go map data structure is 2147483647 for 32-bit architecture and 9223372036854775807 for 64-bit architecture.

Pending Transaction Pool

Pending transactions are stored in TxPool.pending that is a map(transaction hash -> pointer to transaction). In go-ethereum - tx_pool.go, there is no limit check to the number of transaction when adding transactions to this data structure.

The number of transactions in the pending transaction pool is effectively limited by memory.

Transaction Queue

Queued transactions are stored in TxPool.queue that is a map(from address -> map(transaction hash -> pointer to transaction)). In go-ethereum - tx_pool.go, there is a check that the number of transactions from the same From address does not exceed maxQueued = 64.

The number of transactions in the transaction queue is effectively limited by memory, with the limitation that there can only be a maximum of 64 transactions for transactions with the same From address.

What is the difference between a pending transaction and a queued transaction?

Pending transactions are transactions that are ready to be processed and included in the block.

Queued transactions are transactions where the transaction nonce is not in sequence. The transaction nonce is an incrementing number for each transaction with the same From address.

For example:

• Transaction from account 0xaaaa...aaaa with nonce 0 has been included in the blockchain.
• Transaction from account 0xaaaa...aaaa with nonce 1 has been included in the blockchain.
• 10 transactions from account 0xaaaa...aaaa with nonces 3, 4, 5, 6, 7, 8, 9, 10, 11, 12 as sent to an Ethereum node. These are placed in the transaction queue as the transaction from account 0xaaaa...aaaa with nonce 2 has not been seen by the Ethereum node.
• Once the transaction from account 0xaaaa...aaaa with nonce 2 is added to the transaction pool, the 10 transactions with nonces 3, 4, 5, 6, 7, 8, 9, 10, 11 and 12 will be moved from the queue to the pending transaction pool and all 11 transactions are ready to be processed and inserted into into the blockchain (provided there is enough gas).

the result is ok. At first, i send 110 txs, the result of txpool.status

{
    pending: 110,
    queued: 0
}

and then start mined,all of transactions were mined. As follows so the tx_pool doesn't drop any transactions. In go-ethereum source files,tx_pool.go file deal with transaction logic , some transactions will be removed if tx_pool is full that pool size over 4096+1024,the source as follows:

// If the transaction pool is full, discard underpriced transactions
if uint64(len(pool.all)) >= pool.config.GlobalSlots+pool.config.GlobalQueue {
    // If the new transaction is underpriced, don't accept it
    if pool.priced.Underpriced(tx, pool.locals) {
        log.Trace("Discarding underpriced transaction", "hash", hash, "price", tx.GasPrice())
        underpricedTxCounter.Inc(1)
        return false, ErrUnderpriced
    }
    // New transaction is better than our worse ones, make room for it
    drop := pool.priced.Discard(len(pool.all)-int(pool.config.GlobalSlots+pool.config.GlobalQueue-1), pool.locals)
    for _, tx := range drop {
        log.Trace("Discarding freshly underpriced transaction", "hash", tx.Hash(), "price", tx.GasPrice())
        underpricedTxCounter.Inc(1)
        pool.removeTx(tx.Hash())
    }
}

so there is no overflow.