Leave the subpanel breaker as-is
For the barn subpanel, leave its breaker at 100A (or whatever the factory provides). You don't need a breaker there, you need a shut-off switch. Using a "main-breaker panel" is just cheaper and neater than having a totally separate shut-off switch... nobody cares if the shutoff switch says "100A" on the handle :)
Breakers protect wires and equipment, and the 40A breaker in the house panel more than suffices to protect the 100A rated panel. If your logic is "oh, I want the breaker to trip locally first", timing breakers like that just doesn't work. It's always the far one that blows.
(On the other hand if you used large enough wire to justify a 150A supply breaker, then the 100A breaker would protect the 100A rated panel. You might reasonably do that if the 150A feeder was going onward to supply other subpanels at other barns).
Consider running bigger pipe
It's easy to get in a hurry to pinch pennies, but there are two places not to do that. First is on pipe. Price it all ways, but the cost increment to larger conduit is often not that much, and buys you a lot of flexibility. (also consider the "aluminum factor").
Consider aluminum wire
Ironically, one place people fail to pinch pennies is going with copper wire as a default, or worse, as a moral panic. There's a lot of FUD running around about aluminum, and that had some truth when it comes to small branch circuit wiring (the wires to your outlets and lights). Look closer, the bad actor was actually copper lugs/terminals that did not play nice with aluminum wire. Aluminum lugs play well with copper though -- which is why your subpanel's main lugs will be aluminum. Aluminum remains the wire of choice for feeder (though only the power company is still allowed to use the scandalized AA-1350 alloy, you'd use the new AA-8000 alloy).
At 40A circuit size (#8 copper or #6 aluminum), there still isn't much of a cost motivation to go aluminum. Starting the next size up, #6Cu or #4Al, aluminum makes increasing sense. So if you are thinking of a size bump for voltage drop, price aluminum - especially for such a long distance.
On feeders, copper is not an upgrade. Go with Al if it makes cost sense.
You may not need to bump for voltage drop
For a certain circuit size (40A), there is a minimum wire size you must use (8 AWG) or the wire will overheat. This is good for any length of wire, though at excessive lengths, the resistance of the wire stacks up, and you start running into serious voltage drop. The voltage drop is proportional to current actually being drawn in that moment - lights alone won't provoke much drop, but start the table saw and you'll get a bunch.
Voltage drop can be computed, and there are voltage drop calculators on the web to make that easy. The classic blunder, made even by many experts here, is to calculate voltage drop based on the number written on the handle of the circuit breaker (40A). That is wrong. Voltage drop should be computed based on your actual, expected loads, while thinking about how sensitive they actually are to voltage drop.
For instance if you have a delicate data center and a huge air conditioner motor on the same feeder, you'd need to get out the sharp pencil and really run the numbers. For LED lighting (again with auto-ranging power supplies) and a "12A" table saw, maybe not so much.
That said, if you were using 80% of circuit capacity (32A) most of us would proscribe a wire size bump at about 110 feet, so in your case, 2-1/2 bumps (5 wire sizes) to #3 Cu (really #1 Al, unless you own a copper mine). Needless to say, your conduit pipe needs to be large enough to accommodate that wire, back to my advice to upsize conduit.
On one hand, you won't be loading this circuit to 32A. On the other hand it is still quite a distance. It's leeeegal to use #8 Cu wire and just lump the voltage drop, but I think I would price #4 Al (=#6Cu) just the same. Cheaper to invest now than have to tear out and re-pull such a long run.
Also, 120V circuits suffer twice as badly from voltage drop. So if you will mainly have one load, and it's 120V, think about additional bumps still. Make sure you are saying "120V" in the voltage drop calculator. For instance a 120V table saw will have four times the voltage drop of a 240V saw of same power. If there's also a dust collector, and it's on the same leg, that will make things worse still. If it's on the opposite leg, that will help things somewhat, by moving some of the return current to the other leg.
If all else fails: transformers and 480V
One thing I love to do in these cases is get a couple of appropriately sized transformers (10KVA in your case) and connect them tail to tail, 480V or 600V primary to primary. You backfeed the house transformer with plain 240V (no neutral). It sends 480V or 600V down the 2 wires to the barn, for 1/4 to 1/6 the transmission losses for the same power (1/16 to 1/36 for 120V loads). The second transformer drops it and splits it to 120/240V.
The transformers make it a separately derived service, which means the local panel here becomes a main panel which must have neutral and ground bonded. The ground rod becomes your one ground. You don't need to pull a ground or neutral from the house, 2 wires does the trick. The wires can be smaller - in your case 12 AWG would suffice.
This is wizard-tier stuff, and you'd want to very carefully design it so there's basically no chance of anyone getting near the 480/600 primary voltage. That stuff bites very hard.
Misc.
You actually need two ground rods, unless you do a complex test to confirm the efficacy of the one ground rod.
As you know, you need a ground wire also. The ground wire can be smaller than the conductors, but take note -- Its size must be based on the conductor size not the circuit ampacity. Don't go "A 40A circuit can use a #10 ground wire", that is wrong - if you upsize wire for voltage drop, you must upsize the ground wire also; a table from conductor size to ground size tells you what you need.
If you're even thinking of using an alternate power source in the barn, such as generator or solar, install a Siemens main-lug subpanel with the "shutoff switch" breaker in the #1 position. That will let you use their $30 factory-spec generator interlock later. FYI, a generator at the barn cannot backfeed the house unless you put a second set of feeders in the pipe.
You can have up to 3 circuits in the pipe before you have to start derating wires. If you have upsized wires for voltage drop, you can go to 4 circuits. Since it runs to an outbuilding, each circuit would need to have a reason that it couldn't just be supplied out of the subpanel, e.g. Switched circuit for lights or pump, gen backfeed, etc.
Never forget the dust collector. When there's a table saw, there's a dust collector. Don't forget to include it in the figures. As mentioned, putting it on the opposite 120V pole will actually ease voltage drop, by splitting the load across all 3 wires instead of just 2.
Best Answer
You most likely have a bad connection of the breaker to the sub-panel bus bar or a loose terminal somewhere. It could POSSIBLY be a bad pole on the 50A breaker, but that's fairly rare in comparison. You MUST address this immediately by the way, because it's a fire waiting to happen.