You said:
I disconnect the red, black, and the neutral from the generator to the house neutral and the transfer switch leads and I read 120 volts from the black feed to the generator neutral and 120 volts from the red feed to the generator neutral and 240 volts between the red and black generator leads as would be expected but when I measure the black generator lead to the house neutral I get 0 volts and from the red generator lead to the house neutral I get 240 volts and the red generator lead is the same one popping the 30 amp generator circuit
If the black generator lead to the house neutral is 0 volts then someone has connected them together somewhere. That is why the red lead to the house neutral is 240 volts.
It sounds like something is seriously mis-wired here. Take apart what you have and post some pictures or better yet trace every wire and draw a diagram of what you have and post that with the pictures.
Heck No. Checklists of switches to throw in sequence are absolutely unacceptable. I know this is "new to you", but this is well-hashed territory to safety investigators and most people familiar with electrical. It falls in the "facepalm, what were you thinking" category.
This may seem counterintuitive, but for a temporary fix, you permanently modify the wire. You de-energize the entire system, then run around with flashlights physically severing the mains connections, and physically splicing in the generator connection. Like you'll never use it again, but leaving it intact for potential future use.
For instance I would physically remove the main breaker from the main panel and replace it with a (listed) blanking cover, and/or remove and insulation-wrap the mains feeds. Then hard-splice the wires to the garage so they go to the generator instead of the garage panel.
Walk that whole thing, double/triple check it, have the inspector visit if that's a requirement, then light 'er up. That's what I do for temporary work.
For permanent work, obviously, I'd put in a proper interlock. This is not as big a deal as it sounds. I would start by installing a small** sub-panel right next to the main panel. This would have two "main breakers" in positions 1-4, with an interlock between them (rather simple affair). One would feed from the main panel, the other from the genny. Then I would decide which circuits I would ever want to light from the generator, and move those circuits to the subpanel. The original main panel stays mains-only.
src
I gather you want to use the wiring to the garage subpanel in a bidirectional manner, feeding the garage from mains, or feeding the house from genny. This has come up on this forum before, and there's just no way to do that even remotely safely. Bite the bullet and lay parallel cable.
** by my definition, "small" is 42-space. Panel space is dirt cheap and often comes with free bonus breakers, so a net win. Whereas, running out of panel space is a catastrophe.
Best Answer
From whence are they fed? Where will you put your interlock?
I happen to have a superficially (since I know no details of yours, yet) similar setup - mine is fed from one meter that feeds both panels. I dread to think what it would cost to put a transfer switch on that feed line, so each panel has an interlock and generator input. My reasoning is that I want to be able to power any circuit without any unsafe power outage creative wiring, but not all at the same time (that would be a BIG generator and the cost of that is also outrageous.)
Most of the time only one panel (in my setup) will need a generator attached to it, but the input is there on the other panel in case I wanted to power one of its circuits in the event of an outage - to avoid unsafe power outage creative wiring. Unless I got crazy and got two generators, I'd expect to power down the one panel and move the generator cord to the other one, rather than trying to power things on both panels at the same time. The "primary" panel in that sense is the one with the well pump and most lighting circuits on it. Refrigeration will also end up there, by design. Load control is by hand, but it's much nicer to be able to have well water (and turn the fridge and some lights off while getting it) than to haul water in a bucket from a stream.
I'm dubious about trying to feed from one generator into both panels at the same time, but if you have a large enough generator (and enough output breakers switched off) and sufficient hardware it should be possible, it's just not something that I personally looked into in any detail, as I had the thought in mind that I'd land loads I might want on a generator mostly on the one panel, but spend the small amount on an interlock for the other panel as well. I don't know if you'd have to have a sub-panel from the generator input to split to the interlock inputs, or if you could just depend on the generator output breaker at one end and each interlock breaker at the other end, with no more hardware needed in-between. Given how interlocks work, I can't see any unsafe condition arising from two interlocks on the two panels (i.e. you don't appear to be asking about "backfeeding from a sub-panel" which you cannot do.)
Depending what your reasoning (and scale of generator) is, you could possibly move circuits you expect to need to power from a generator all to one panel.