Alignment is to a devil what matter is to a creature of the Prime Material.
PHB p. 122
Alignment is an essential part of the nature of celestials and fiends.
A devil does not choose to be lawful evil, and it doesn’t tend toward
lawful evil, but rather it is lawful evil in its essence. If it
somehow ceased to be lawful evil, it would cease to be a devil.
You might want to start your investigation of this question with a quick read on Plato and his philosophy of Ideals. Plato posited that everything we see in this world is just a shadow of a world of "ideals". When we see a dog, we see a lump of flesh attached to some bones and wrapped in fur, more or less like lots of other lumps that we also call "dogs". Plato said that since we can recognize all these peculiar lumps as "dogs", then somewhere there must be an "ideal dog" - a single creature that completely embodies all things doggy.
DMG 57, Outer Planes
When discussing anything to do with deities, the language used must be highly metaphorical. Their actual homes aren't literally places at all, but exemplify the idea that the Outer Planes are realms of thought and spirit. As with the Elemental Planes, one can imagine the perceptible part of the Outer Planes as a border region, while extensive spiritual regions lie beyond ordinary sensory experience.
The DnD multiverse embraces this idea and places these ideals in the Outer Planes. The game no longer has all the summon spells it once did, but when it did, the implied mechanism was that whatever beast was summoned was not any particular beast but the Ideal of that beast. (And, yes, you read that right - the Ideal Dog is not a particular dog, it is all dogs - even those that never existed.)
Of course, to exist on the Prime Material that Ideal had to cloak itself in the stuff of that plane. Those of us made of ordinary earth can not properly perceive or interact with Ideals in their true form. This is why we have the concept of Avatars and why fiends who die on the Prime Material are not truly killed. You can't kill an Ideal through material weapons. If it is possible to truly kill an Ideal, it must be by subverting the very concept that the Ideal represents.
D&D 5e doesn't get very deep into this set of ideas, but there are a few hints here and there. You have stumbled into one of the most direct examples.
The Outer Planes are ideal manifestations of Law, Chaos, Good and Evil. Alignment is the stuff they are made of, as surely as the Prime Material is made of earth, water, air and heat. (DMG p. 59.)
All devils are made of that same material. When a devil visits the Prime Material, he manifests with a material body so he can be compatible with our elements, but this is only a "cloak" over his true nature. If you kill him, you are only rending this "cloak" - you can't touch his true nature with mere material weapons.
If it dies outside of the Nine Hells, a devil disappears in a cloud of
sulfurous smoke or dissolves into a pool of ichor, instantly returning
to its home layer where it reforms at full strength. Devils that die
in the Nine Hells are destroyed forever - a fate that even Asmodeous
fears. (MM p. 67)
Finally, to heart of the question.
If you change a devil's alignment, you are literally changing the stuff of which he is made. Whether his appearance changes is subject to DM's discretion - the books don't say. But in all important ways, he will change into the celestial or fiend appropriate to the new alignment. There is at least one interesting edge case: the Neutral alignment. Perhaps a good follow-up question for this forum would be, "Is there an Ideal manifestation of Neutrality in the Outer Planes?"
A devil is formed/born when (MM 66, top paragraph)
sending his servants to the mortal realm to corrupt the souls from which new devils are spawned
So, answering about Rakshasas and Hell Hounds, they are each a unique fiend rather than being a form of corrupted mortal soul.
On the next page is a list of four main categories of devils, with a subsection called Promotion and Demotion covering how a corrupted soul can get from newly minted minion all the way up to the top levels of the Infernal Hierarchy.
In the stat blocks on MM 70-78, it appears that all Devils have the following features:
- Resistance to: cold; bludgeoning, piercing, and slashing from nonmagical weapons that aren't silvered
- Immunity to fire and poison
- Immunity to the Poisoned condition
- 120' darkvision
- 120' telepathy
- Infernal language
- Magic Resistance
Most also have natural armor and Devil's Sight (for some reason, not Pit Fiends or Erinyes).
Making Your Own Devil
If you're making your own devil then the only real core is Lawful Evil and some kind of corruption of a mortal soul. I see two options to make it interesting.
First: Backstory.
A favored NPC is a literal devil in disguise. The drama arises from the emotional connection PCs have to this NPC. This is basically a literary classic.
In D&D terms, though, you might consider what happens differently when an Elf or Dwarf is corrupted, as opposed to a Human. The books don't seem to say anything on this, but it's totally within DM purview to build the world in this regard.
Second: Custom monster.
DMG 273 details how to create or modify a monster.
My favourite technique for this is covered in the Modifying a Monster section. Just take a different monster and describe it differently. A short step away from that is describing a creature differently and also tweaking small things like energy type on damage. For example, you could just take a Deva, give him red skin and horns, then tweak that Radiant damage to Necrotic. There's a section on Fallen Angels on MM P. 15 if you'd like to commit to this route.
If that's not enough, then use the subsequent sections to create your own. The most important step here is in the opening paragraphs: the concept. If you're sticking with the book's 'corrupted soul' concept, then consider what sins/corruptions might have created the various devils in the book. Then consider what your new creation's sin might have been (backstory again!) and what thematic powers, traits, and quirks might manifest. Add those to the above list of generic devil traits. Additionally, you should consider how this devil fits into the Infernal Hierarchy, including which devils owe it fealty and which it owes fealty to.
Best Answer
In the case of demons, the monster description (MM p. 50) argues against the reverse of falling that the angels are susceptible to.
Angel: pride can lead to a mistake that leads to a fall. (MM p. 15)
Demon: opposite of angel ~ the opposite of pride is humility.
Humility doesn't fit the profile of lacking empathy, mercy, and compassion. From that starting point, the DM needs to establish a motive that outweighs this fundamental characteristic of a demon. Something, some experience, someone, or some event induces a demon to act in an other than selfish and destructive manner.
The DM then must resolve how a creature spawned as an extension of the Abyss (MM p. 50, Spawn of Chaos) breaks free of that influence sufficiently to change its nature. In the case of demons formed from mortal souls (since in 5e souls exist) the memory of being something other than a demon could provide the leverage needed to initiate change. In that case, the specific follow-on effects will vary as with the motive for change.
Two simple and opposed potential outcomes:
The demon enlists allies to return to the abyss and avenge --- something.
(And many points between). Without a story hook on the why of a demon's change, the what of the aftermath of the change is too varied to nail down.
MM p. 7 (RAW boils down to "You can if you want to.")
The only thing stopping the DM is the crafting of an in-world reason that fits the larger narrative of the campaign and the story. The only constraint is the imagination of the DM, not a rule.
What makes for a good change is a decision to create the exception to the general rule of the MM default description for demons - the general case is that a demon isn't motivated to change. Establishing motivation is step zero to this change occurring in, or fitting the narrative of, a campaign.
If lore extends beyond 5e, something from 2e Planescape or FR novels may offer a concrete example.