Replacing Doorbell Transformer

Once you remove the cover of that box with the switch and everything else, you should see three wires coming into the top of the box from the transformer. These are usually white (neutral), black (hot), and green (ground). Take a picture of all the wire connections and make sure that everything gets connected the same when you wire in the new transformer. There will be additional wires in this box that are connected, and these connections all need to remain the same.

The transformer will most likely be physically attached to the top of the box with a conduit nut (inset of picture).

These can be hard to remove and install in a small box, but the basic method is to place a flat headed screw driver on one of the raised portions of the nut and tap it with a mallet to spin the nut in the correct direction. The new transformer needs to be tight. Loose connections lead to loose or damaged wires, and that leads to bigger issues.

When you reattach the low voltage wires to the transformer (the red and white in your pictures), it doesn't matter which screws the wires go to because it is alternating current (there is not a positive side and negative side like with a DC power adapter).

Any "modern" doorbell transformer will be 16VAC, which is what the Ring wants. You are correct that this 10VAC transformer is not powerful enough, regardless of how you wire the Ring. Note that if you plan on using your current chime, a higher voltage transformer might not work. It could damage the chime (immediately, or in the long run). Plan on replacing or removing the chime as well.

Do not replace the junction box. It is a standard junction box and there is nothing wrong with it.

Backstab connections fail more often than transformers

Look at the receptacle. See how the wires are connected? They're "back wired" with a backstab style connection. That is only allowed for 14 AWG solid wire, the receptacle's labeling plainly says

PUSH IN #14 CU SOLID WIRE FOR 15A BRANCH CIRCUIT ONLY.

CU = copper. It is then repeated in French.

Looking at the transformer's ground wire it definitely stranded wire that has been tinned with solder to keep it from getting a case of the frizzies. The ground wire has been horked onto the already-occupied ground screw in an impressive display of bad workmanship.

Equally bad workmanship is obvious in the attaching of the transformer wires to backstabs. The wires are stranded and certainly not 14 AWG. This is a violation of NEC 110.3 which requires the instructions and labeling of the receptacle be followed (14 AWG solid only).

Since backstabs have a spectacular failure rate (open circuit) even when they are used to spec, we should expect this to be a failed backstab.

So, I would remove the transformer wires from the backstab (if it's possible to do this while also facepalming), and move each wire to the appropriate screw terminal. I would also pause to inspect the wire for any arcing or galling, because that would explain the mystery.

As far as actually removing the wires, there should be a little slot into which you can jab a tiny screwdriver, and sort of pry away from the wire and that should reduce the extraction force somewhat. Otherwise feel free to just use an excessive amount of force. Worst case the wire will break off inside the backstab and you'll have to strip off another 1/4" to attach it to the screw.

When removing #14 wires, just twist the receptacle back and forth about 60 degrees repeatedly while using serious force. Don't cut; wire length is precious in most cases.

While you're at it, fix the ground

The ground installation was acceptable until this transformer came along. Now it is one too many wires on that ground screw. Move the "U" loop on the receptacle screw so you have about 2" of extra length sticking out (instead of the 1/2" now sticking out). Wire-nut the transformer ground wire to that.