Oh dear. That's an unfortunate detail. :( I'm afraid DA01 is actually correct, the proper solution is tear out and properly rebuild. The proper approach to this would have been to either stop the barge end (the gable end overhang) several inches above the lower roof so the bottom could be properly enclosed and the lower roof properly flashed. This would not be a good detail architecturally though.
Another approach would be to fully extend the barge end so that it is actually fully framed into the lower roof, then flashing applied around the intersection. The barge end trim could extend a bit over the flashing, stopping short of the lower roof an inch or so. But the main structure is actually physically connected to the lower roof.
Those are the correct options, which involves a lot of remedial work and undoing much of what has been done in this area. If need be, you could do a stop gap treatment that will seal off the gap for now. This is far from ideal, and proper treatment should certainly be done when it comes time to re-roof.
This would involve fitting carefully shaped wood shims into the gaps to give the appearance the barge end had been framed into the lower roof. The outward facing shims could be toenailed in place, but the parts under the soffit would be just a pressure fit or sort of glued in with caulking. There will likely still be small gaps that need to be filled with caulk. Use a paintable caulk so when it's done and painted, it will not appear to be a patch job from any distance.
I don;t like this final option at all, but I understand if you are unwilling to do a proper fix at this point and thus offer this suggestion.
Yeah, that roof's pretty beat. I'd actually go heavily at that seam with Mortar Caulk (its gritty & gray), the whole length of the seam if you don't know where the leak is. But, spraying a hose at the seam & very slowly moving the spray up the seam should give you a good idea of the problem area.
Still caulk the whole seam, but then do under the shingles in the leak zone. Be careful & don't pry them anymore if they're brittle & chunks start snapping off. I caulked a flat roof in the rain 3-years ago & it still hasn't let in a drop.
Best Answer
You're right that nailing so low is a concern. Typically I'd leave at least an inch between the shingles and the channel. If you use color-coordinating step flashing this isn't an aesthetic concern. You can also overlay rubberized window flashing tape on the step flashing to seal nail punctures better.
You don't say where you are, but I'd worry about melting snow leaking through low punctures in the flashing, or flow and splash from heavy rain.