Well, first of all...you probably didn't have an actual 277volts. What you had is a cheap meter. I have seen all kinds of whacky voltage readings from the $5 harbor freight meters I have laying around.
The only place one sees something other than 120 or 240 volts in a "normal" residential setting is the very rare "3 phase power". The vast majority of homes do not have any 3 phase power available. It has to be run to the house specifically as an unusual setup. When this exists in a home it would add "208V" to the list of possible voltages. Again, very rare and probably not at your house.
The only place you would see a real 277V is in a big factory. It is normally used for HUGE machines such as big welders and bigger shop equipment than you have probably seen before. Normal DIY shop equipment runs on 120V. Big shop equipment runs on 240V. HUGE shop equipment requires 3 phase connection with the 208V "wye 3 phase" mentioned above. It is extremely rare in a house. The 277V delta 3 phase would never be seen in a house, and only rarely in any commercial building you have been in. It is typical only in large factories.
Now as for what's probably wrong...you probably have a loose connection or a broken wire on one side of your 220 connection. This would give you 0v between the two hots, but still 120 from 1 hot to neutral.
As for how to fix it? Hire an electrician. Could you fix it yourself? maybe. Could you burn your house down? maybe. Could you sleep at night after messing with it yourself? only just maybe. Hire a professional for anything that is a combination of dangerous and slightly outside your comfort zone.
It doesn't sound like the #14 and #12 grounding wires being connected is the issue, to me.
EDIT:
NEC 250.148 (C) Metal Boxes. A connection shall be made between the
one or more equipment grounding conductors and a metal box by means of
a grounding screw that shall be used for no other purpose, equipment
listed for grounding, or a listed grounding device.
The OP uploaded a picture showing that the #12 is an independent circuit, in new 12/2 w/ground NM cable. So what you actually have is grounding via the BX armor, presuming that the BX tightly clamped in both this box and the service panel, and grounding through the #12 wire in the 12/2 circuit, and the two should be bonded together to the metal box.
The ground path through the BX is likely to have higher resistance than the ground path through that nice clean #12 copper. The metal box has to be grounded no matter what. I say leave all the grounds bonded together and to the box, and make sure the BX clamp is clean and tight at the junction box and back at the panel. If you're really concerned about the grounding through the BX sheath, try to run another lone ground wire back to the panel, or pull new NM cable to replace the BX.
Also, you would get a much tighter connection between the armored cable and the box using a better clamp than the one built-in to that old box, something like one of the following (that red plastic bushing protects the wires from sharp edges, so you never energize your grounding system accidentally):
END EDIT
All of the grounds in your house are ultimately bonded together, anyway, to zero the voltage differential all over your house (Neutrals are different! The neutrals are only bonded to the grounding bus in the main panel, nowhere else, and neutrals can only be shared by two circuits under specific conditions).
You wouldn't want a large circuit using a too-skinny conductor for grounding, though.
It sounds to me as if the original circuit is possibly BX armored cable, and the flexible conduit itself is the only grounding conductor going back from this junction box to the panel.
This not up to current code if the flexible metal conduit is the only ground path--there can be continuity problems through the flexible armor. BX was actually demarketed for a while, then came back as armored cable with a metal bonding strip that makes contact with the flexible conduit all along it's length to ensure a good ground path.
However; if you can somehow run a #12 bare or green-insulated wire back from that junction box to the panel (route it up through the attic?) and bond it to the grounding bus inside the panel, as well as to both the #12 and #14 wires, and the BX armor in the junction box by using the right metal clamps and bonding the wire to the metal box, you'd be perfectly fine.
Having said all that, I guess somebody could argue that some grounding through the BX armor is better than no grounding. But if you're in there messing around, you want to do it right and bring it up to code.
Best Answer
Since your 100 amp, 240 volt service consists of 100 amp, 120 volt branches, you can have 100 amps of load on each of the two branches. Note that the AC has two 120 volt breakers with the handles connected together. That uses both of the 120 volt branches to make a 30 amp, 240 volt breaker. As shown the box has breakers totaling 75 amps on one branch and 80 amps on the other. If you add a 60 amp, 240 volt breaker you will have 135 amps on one side and 140 amps on the other.
You can have breaker capacity in excess of the main circuit breaker rating based on the assumption that many of the load devices will only be used for short intervals of time during a day. However if the server uses 48 amps all of the time, it would not be difficult to imagine a day when the AC is running a lot, the refrigerator is cycling on occasionally, you stick something in the microwave and your wife is vacuuming.
It may not be out of the question, but it is somewhat doubtful. The service is already nut up to today's code. Today, a residence should have two 20 amp circuits dedicated to kitchen counter outlets and one 20 amp circuit dedicated to bathroom outlets. A microwave and garbage disposer often have their own circuits.
The box itself is likely capable of handling 200 amps at 240 volts, but you may have service entry wiring only rated for 100 amps. An electrician may come and quote a price at no charge. You can not get a good answer without that, but there is likely a way to get it done. Note that 48 amps at 240 volts is a lot of power. That is like having the oven and all the burners going at one time in an electric stove.
You don't need the Hubbell 363R6W. Use a standard electric range outlet.