Electrical – is it safe to run a 300 watt 220v motor (european) on a 20 amp breaker (american)

The output appears to be ungrounded or center grounded. It could be a transformer intended for outdoor tools in UK. If there is also a safety ground 3 wire plug, it could be coupling some of the harmonic current from the switch mode active power supply (which should be flowing equally on hot and neutral) on the safety ground. That would be unbalanced current and trip the GFCI. The smaller transformer could be wired with normal grounding or it may have a plastic case. Adding filtering to the mains input could fix the GFCI issue.

40 watts at 12 vac is about 4 amp which leaves plenty of headroom on the 12ga cable (which is normally rated for 20 amps (with heat loss and safety factors included). But what if your wire gets cut? Shorted? the worst case max current that a 300 watt transformer can put out (if it really is a 300 watt not a 310 watt or 315 watt) is 25 amps which is flirting with the danger limit of the cable. Soi I would either put a 15 or 20 amp fuse on the secondary side of the transformer, use 10ga wire, or put a 2amp fuse on the primary side of the transformer. 2 amp fuses could be a trick to find, 10 ga is expensive, but a 15 or 20 amp automotive fuse will work just fine considering it is only 12 v. And since you are pulling less than 5 amps a smaller fuse would be fine.