Filling gap between drywall and stairs

Scotch guard will greatly increase the stiffness and how it holds up to dirt and abuse. I suggest a heavy heavy layer of scotch guard once a year right after it has been shampooed (let it dry of course).

Long-term carpet might not be ideal. You need to try to figure out what is under the carpet. Almost all of the time you can take out the carpet tacking, sand, and the stain/paint the stairs. This is not hard to do, doesn't cost much, and it is a true DIY project.

There are 2 ways to do this, the first way is in essence just cutting the wall at the top of the skirt board and capping it off and setting the handrail on top of the cap that is finishing off the cut wall, sounds a bit crude, but essentially that describes what you will have.

The up side to this is you will not need to do a floor repair, depending on where your wall ends where the newel will be.

The other way to do it is by removing the wall entirely, repairing the floor, removing the treads and risers, and set longer ones with return nosings on either side. and re-support the stairs with a new walls under the existing carriages.

This is the most work, but will give the best result in my opinion.

Jan. 2, 2014 edit

Here is the work I did in 1989 that shows what I did. This one shows the best example of the effect I was trying to explain. I do not have a shot that shows how the handrail goes past the ceiling, but with a little imagination you could picture what the rail would have to do if the stairs were much wider at the bottom.

Edit #2

I did a little calculating to help, if it may