stairs
Because the renovators squeezed a half bathroom in the space beside/under the stairwell, there is an awkward 3-4" dead space between the banister and the stairwell wall (pics below).
I've never quite seen anything like this, so I'm a bit puzzled as to how I could eradicate this space.
It might be helpful to know that both the stairs and the adjacent hallway are already quite narrow (25 in. or 63.5 cm each).
First of all, since it sounds like this could be considered "occupied space", you better check your local building codes and/or with your local inspector to see what is allowed. Most codes have specific requirements for stairs, and in some jurisdictions strange things like spiral stair cases may not be allowed or may require specific approval because of their narrow treads.
I'd definitely look to see if there is any way to get more space at all. For example, what is behind the closet? If you can lose a bit of ceiling height there, you could get a bit more room for the staircase. Likewise, is it acceptable to have the stairs coming out into the room at all? This too, will provide a bit more space. Using this calculator, if you can do 10 rises of 8" (this may be steep though), you need 6'8" of total run space.
Depending on the adjoining space, you might also be able to make use of the room under the stairs.
If you can't get any extra space, you might consider a "ladder staircase" (kind of a hybrid of ladders and stairs):
There are some interesting designs for these, including alternating treads:
And some really funky variations if you want to be really creative:
The answers of the tread then riser SEEM to make sense, but consider this: A properly installed tread will not come off before the lip breaks off. Using just regular Elmer's wood glue, I used to make treads by gluing the edges of 2 2x6 planks together and then planing and cutting them down (after being clamped for 24 hrs). After that, if I took a sledge to it the plank would never break at the glue joint, but somewhere else along the natural grain. So the stair tread flipping off is highly unlikely. What IS likely is that the overhang could be chipped or damaged and the tread may need to be replaced. If you go riser over tread then you will essentially lock the stairs together to where you cannot remove a single tread in the middle somewhere without tearing up a lot of the assembled stairs. Additionally, locking any wood assembly together completely isn't always good for expansion and contraction caused by potential seasonal humidity change. This isn't always a factor in certain natural climates and/or exceptionally well controlled climates, but if it can be, it should be considered.
Best Answer
I think I'd go with part of my original answer, but leave the railing as-is. Take the balusters out for all treads above the new wall. Replace the treads and risers with longer ones that extend all the way to the wall, closing the gap. Replace the balusters. Done deal. I wouldn't even mess with the framing.
You'll have a little bit of wasted tread space along the wall, but it'll be much less odd than a deep gap. The lowest riser would notch over the wall over the new doorway. It looks like a very clean situation from here.
[original answer]
Were it my home I'd return the handrail into the wall and widen the five upper stairs to the wall, filling in as appropriate above the bathroom door casing to the first wide tread.
I can't see what's going on with the soffit/ceiling above the lower hall, but you might also be able to extend the wall above the bathroom door to meet the stairs, and either terminate the railing there or run it past/above the wall, which will either have a ledge at the top or run all the way to the high ceiling.