Why didn't arcade games keep scaling? Because the problems of making a game stand out scaled in a different direction than the benefits arcade cabinets provide.
Decades ago, as mentioned in the question, a game could stand out by simply having 3D graphics, or doing other things that required hardware that could not practically be included in consumer home gaming devices. As technology improved, that became less and less the case, and now there is essentially nothing you can make a video game do that commercial hardware can't handle. Nicer hardware might get you more polygons or better lighting effects or even better physics simulations, but you can't get the same qualitative difference as 3D graphics in a world of otherwise 2D games.
Once we reached that point, the main challenges of developing impressive new games shifted to things like innovating on mechanics, making game worlds bigger and deeper, improving 3D model fidelity, and others. These things do not necessarily demand more of the hardware, but they require much more development resources.
Considering those shifts, it is often not worthwhile to make an arcade machine, in terms of both user experience and profit.
Let's say you wanted to try the "expensive hardware" strategy of building an arcade cabinet with today's technology. You get the best graphics money can buy: a 4K monitor and 4 RTX 2080 TI connected with SLI. You get the most powerful processor on the market: an Intel Core i9-10980XE. You get all the memory you can cram into a PC motherboard: about 64GB. And you fill it out with more storage than you could possibly fill with one game. This adds up to several thousand dollars worth of hardware, and you can do real-time ray tracing in 4K, which is pretty cool.
Now you need to make a game that actually uses that hardware. You're going to need designers, modelers, artists, animators, programmers, and others to spend a lot of time making a game with the fidelity to take advantage of that hardware. And in the end, this is all still commercial gaming hardware, so you can provide a lower resolution graphics option and the game will run on people's computers at home. And once you sell the game to people directly, most people probably would not consider it worthwhile to travel somewhere to pay more money to play the same game with marginally better graphics than what they can get in the comfort of their own home. Plus, if people buy your game to play at home, you don't have to handle selling hardware.
I recently went to a Dave & Busters, which is probably one of the best places to see modern arcade games. Almost every game there distinguished itself not by being more visually impressive, but by having an input device that you would rarely see on consumer gaming setups, like a steering wheel and pedals, or a gun, or a dance pad, or a single big button. For the most part, they were designed to be easy to understand, and quick enough to finish that other people could get a turn.
Best Answer
First, we must understand a fundamental: Technology has improved and miniaturized a great deal since the arcade machines of the 70's and 80's. The smartphones that we hold in our hands today are more powerful than some of the best supercomputers of that era.
While arcade tables and their innards may vary depending on the company that produced them (and the exact year of production), I'll be looking specifically at one called "Space Zap", as I found a blog where someone opened up an old unit.
The first thing that modern arcade tables have over their classic counterparts are the monitors. The classics use CRT displays (if they were old enough, they were the kind that used replaceable vacuum tubes) which were bulky, and would take up quite a bit of room behind the display.
Next, let's look at the CPU of this Space Zap unit. The blog says it's a Z80 chip, the same sort that powered the Tandy/RadioShack TRS-80. It runs at about 1.77MHz. There are also two ram boards at 8KB each (for a total of 16) There is also a board that contains the game, that is the same dimensions as the CPU board. This board contains roughly 12KB of data (might be a little more, was looking at the file size of the ZIP'd MAME ROM file)
Those main boards are sandwiched together in a case that looks roughly the size of a 2-slice toaster. With the panel that deals with counting coins (and the box below for collecting them), the wires to connect everything and supply power, and leaving enough room for venting so the whole thing doesn't overheat. There's not too much room left.
By comparison, a modern-built cocktail arcade can use an LCD display that could be no more than an inch-and-a-half thick (or less) freeing up all that room underneath.
And underneath, you don't need all that much room! You could probably fit any modern PC in there; with hard-drive, video card, and everything. You'd probably be able to run the latest AAA games on it. But if all you're running is a an emulator, you need very little more than a Raspberry Pi which has at least 395 times the processing power (700MHz - 1.2GHz), and at least 16-thousand times the RAM (256MB - 1GB) of our starting Space Zap table. All in a package that is under 3.5in by 2.5in.
To store all 5000 game ROMs, you'd just need to connect a MicroSD card. even if we go with a cheap 16GB card, at 5000 games, that gives us 3.2MB per game. That's over 200 times the space our Space Zap game needs (granted, depending on the eras included in the collection, not all are going to be as small as this game).
(NOTE: I am not saying the machines you are looking into are using the setup I propose here. I am just offering comparison to illustrate just how much the underlying technology has evolved. What used to be stored on a board around the size of a DVD case, now fits into a tiny fraction of a chip that's about the size of a dime.)