Learn English – Difference between at and to

This is an issue of tense agreement.

"he did" is past tense. "should have done." is also past tense, so it agrees and is proper

"should do" is present tense, so in your second sentence there is a tense mismatch between "he did" and "should do". A proper sentence ending in "do" would require you to change the verb tense (and the meaning), for example "He is doing what he should do."

English is a ridiculous language! Maybe Ram should slap whoever created our tense structures!

Instead of is used when you were given a plan and you want to change it. This phrase stresses more strongly you cannot choose both variants. Imagine you are building a football team with a limited number of 8 players and someone gives you a list of names. If you agree with his selection except for one name, you could say

I would choose Steve instead of Homer.

Rather than connotes a situation where you are asked about something and you like both options and you need to choose your favourite. Even though this also usually means you cannot choose both, rather than feels a bit more formal because it highlights you like one option, rather than dislike the other. Thus, if you use it in a sentence like

I want Betty to do the job, rather than Michelle.

you can lighten the weight of your choice.