Electrical – wire a doorbell transformer in-wall this way

Once you remove the cover of that box with the switch and everything else, you should see three wires coming into the top of the box from the transformer. These are usually white (neutral), black (hot), and green (ground). Take a picture of all the wire connections and make sure that everything gets connected the same when you wire in the new transformer. There will be additional wires in this box that are connected, and these connections all need to remain the same.

The transformer will most likely be physically attached to the top of the box with a conduit nut (inset of picture).

These can be hard to remove and install in a small box, but the basic method is to place a flat headed screw driver on one of the raised portions of the nut and tap it with a mallet to spin the nut in the correct direction. The new transformer needs to be tight. Loose connections lead to loose or damaged wires, and that leads to bigger issues.

When you reattach the low voltage wires to the transformer (the red and white in your pictures), it doesn't matter which screws the wires go to because it is alternating current (there is not a positive side and negative side like with a DC power adapter).

Any "modern" doorbell transformer will be 16VAC, which is what the Ring wants. You are correct that this 10VAC transformer is not powerful enough, regardless of how you wire the Ring. Note that if you plan on using your current chime, a higher voltage transformer might not work. It could damage the chime (immediately, or in the long run). Plan on replacing or removing the chime as well.

The old transformer has identical wire colors for hot and neutral. The new one differentiates them.
To connect the new transformer, connect the black wire from the new one to the same black wire where your old transformer was connected with the wire nut. Connect both the green and white wires to the same spot where the black and green are connected together from your old transformer.
In your case, this is a main panel, which is why it is acceptable to connect the neutrals and grounds together (though it is not the best way to do it, but you might as well stay consistent with how your panel is wired). This question/answer should give you some more background on this.

Also note that if your new transformer looks similar to the old one, you should put the threaded part where the 120V wires come through through the knockout in the side of the panel, and use a nut to attach it there. The way the old transformer is attached is not how it should be, those wires should not be passing along the side of the panel and through an unprotected knockout like that.