We talked about random initiative in our group a few weeks ago. The DM mostly cited the "I do X because the boss can't act until you are done" kind of thing as a reason.

There are, however, a number of drawbacks. Here's what we could see:

It makes readied actions a big gamble

This is, in my opinion, the biggest sin of importing randomized initiative into 5e D&D.

Suppose I act after the monster in a round, and I want to ready an action to shoot an arrow at it as soon as it sticks its head out of full cover.

Unfortunately, the next turn I happen to end up higher on the initiative than the monster. My turn comes up again before my readied action has even had a chance to go off. I have, effectively, lost an entire turn.

This happens about 25% of the time with your proposed system. That may not sound like a lot, but given that the penalty is completely losing a turn, it's still kind of bad.

Why do I find readied actions so important? Because they are a huge enabler for teamwork and improvisation. If you can't ready actions effectively, you're a lot more likely to get locked into an "I attack it until it falls over" mindset.

It also makes ranged characters/monsters more powerful.

It messes with reactions

A superset of the above problem. Suppose I'm a counterspell-happy wizard fighting a spellcaster. Each turn, I use my counterspell to interrupt their spellcasting. This works fine with a static initiative.

But in a random initiative system, I would sometimes get back-to-back turns ("losing" a reaction), and sometimes have turns where the monster went twice in a row (causing me to be unable to counterspell).

This issue can be mitigated by having reactions (and some durations) reset at the beginning of a round, rather than the beginning of a player's turn. That's a little less intuitive, though.

It reduces, but does not eliminate initiative-order metagaming

Randomizing the initiative does eliminate a lot of player knowledge about the ordering of turns. But it doesn't eliminate it entirely.

You will still run into instances where two PCs have not yet gone, but the monsters have. In this case, the players know just as surely that they are clear to proceed as they would with a pre-set initiative.

It discourages pre-planning

It's one thing to say "okay, I'm up in two turns. I doubt much will change, so I'll start figuring out what I want to do."

It's another thing to say "okay, some unknown number of people are going before me so... I'll start planning ahead?"

It's also a lot easier to go "Bob just went, time to start thinking about my turn."

It adds per-round overhead

This one doesn't apply much to you, since it will generally not take long to gather up the tokens, put them in the bag, and give it a good shake.

Still, randomized initiative does usually take a similar amount of time to establishing a pre-combat initiative order. Except it takes that amount of time every round.

In your system, you could probably save even more time by just doing the draws at the start of combat, setting them in order on the table, and looping through that.

It makes hard combats "swingy"

A challenging monster (like a dragon) getting two consecutive turns can be pretty back-breaking (the worst case scenario being a player taking two breath weapons and dying before they have a chance to react). In the other direction, multiple PCs getting double turns can make a fight dramatically easier, potentially spoiling a boss fight.

Boosting initiative is a relevant "thing" in this edition

There aren't a ton of classes who are destroyed by eliminating initiative modifiers, but there are a few. Assassination rogues are the ones most hurt by this change. Also, as noted elsewhere, barbarians, the alert feat, and any heavy or medium armor wearers who put points into dexterity for the initiative bonus.

It's also worth noting that classes that rely on buffs (such as Druids) will tend to want to be able to invest in a higher initiative, although they do not always do so in practice.

Summary

All in all, none of this is game breaking. I don't think that I would use this system, as generating a turn order doesn't take our group very long (just a quick pass around the table, asking what each person's numbers are).

Still, if your table is running into issues, go for it. Just allow people to rebuild characters after implementing it.

From the SRD:

Initiative

Initiative determines the order of turns during combat. When combat starts, every participant makes a Dexterity check to determine their place in the initiative order. The GM makes one roll for an entire group of identical creatures, so each member of the group acts at the same time.

The GM ranks the combatants in order from the one with the highest Dexterity check total to the one with the lowest. This is the order (called the initiative order) in which they act during each round. The initiative order remains the same from round to round.

If a tie occurs, the GM decides the order among tied GM-controlled creatures, and the players decide the order among their tied characters. The GM can decide the order if the tie is between a monster and a player character. Optionally, the GM can have the tied characters and monsters each roll a d20 to determine the order, highest roll going first.

Nowhere does it state that the GM must reveal what he rolled for monsters, as such it can be left up to GM discretion.

Also note that it doesn't state that the players need to reveal their initiative roll to other players. They only need to let the GM know so he can rank them in order.

I've played in games where players initiative checks are secret because we don't know what order people are going to go in. It can make the first round of combat more interesting as people have to make choices without complete information.