I don't quite follow your description, the standpipe configuration might be part of the problem. The gurgling toilet indicates a venting problem, I don't think it would contribute to overflowing, but it is a problem. As mentioned, a photo or diagram is needed for better assessment. My intent is to describe a proper washer installation and address how the height plays into it.
There are all sorts of variations for pipe configurations, this one is very typical. The washer drain ties into the 4" horizontal pipe with a vertical 4x4x2 LTTY (long turn tee-wye). It can be installed flat to gain some vertical, at the cost of an extra ell that introduces a potential clog point.
From the LTTY is a vertical 2" pipe as short as possible before connecting to a 2" sanitary tee. A 2" vertical pipe continues up to tie into the venting system. The vent could be reduced to 1.5" if need be. The side inlet of the santee takes a 2" horizontal (2% slope actually) trap arm. There is a length limit, but it's rather long, it's best to keep it rather short. The trap arm connects to a 2" trap assembly. The trap inlet accepts a 2" vertical stand pipe.
The stand pipe must be a minimum 18" high, straight up, no bends. You cannot connect a washer discharge directly to a closed drain system. There must be an overflow provision such as an open stand pipe. I prefer to see about a 24" stand pipe, but 18" is a code requirement.
This gives you about a 24-30 inch tall stand pipe/trap assembly on top of the 48" height of the main drain, so a total 6-6.5 feet off the floor. Washers have a limit of how high they can pump discharge water. The installation instructions should tell you this somewhere. If not, you may need to contact the manufacturer. If your washer can pump up to the top of the stand pipe while sitting on the floor, you're all set. If not, you will need a platform that raises the washer to within the allowable discharge height.
If you still get overflowing with a proper standpipe-trap configuration, then the main line remains obstructed somewhere.
It's not going to be that simple. Unless you support the pipe above where you want to put the rubber boot, it's likely going to come crashing down. Depending on what's above, you might be better off cutting the cast iron where you want, and replacing everything above the cut with PVC/ABS. Without knowing more about what's above this, and what else ties into that pipe, there's no way to provide an accurate answer.
I will say that the rubber boot, is likely not the best course of action.
- Remove the galvanized pipe.
- Use a chain break to cut the cast iron a bit lower than you planned.
- Install a Fernco coupling to transition from cast iron to PVC/ABS.
- Finish the plumbing in PVC/ABS, including the tee for the standpipe trap arm.
- Properly support PVC/ABS plumbing.
Make sure the trap arm for the standpipe is not too long for the size of pipe you're using (2" pipe = 8' max. developed length trap arm 1.5" = 6'). Also make sure the drain has the proper slope.
Best Answer
This situation indicates that there is a plumbing drain problem. The easiest thing to try is adding some height to the stand pipe with a coupling and a pipe section. If you can add some detail and maybe a picture we can advise you further.