OK, I don't have time to answer this as I want to. My background is in psychology, and I fell into role playing games when I turned 10 in 1976. So by the time I was in college, understanding where the term Roleplaying game really came from, I understood the critical nature of immersion, how it is the most important ingredient for game success.
And to be clear, the definition of immersion is to "Immerse oneself into the identity and Role of the part one is playing. To respond, as much as possible, as the person one is playing, not as oneself."
And before getting into the smaller details, I will dive right into the fact that the very system/game one chooses has a huge amount to do with the amount of Immersion.
Metagaming is the opposite of immersion. You use both terms, but I need to make that absolute definition from the beginning. This also means rules that encourage metagaming decrease the immersion in a game and therefore, decrease the main ingredient of a roleplaying game. The mechanics are called "Dissociated Mechanics", a term coined by Justin Alexander. This is very worth reading, because it gets into many of the larger picture issues with players being able to use in-game logic to see the world around them, as opposed to the rules forcing dissociation from in-game logic.
Once the players assume that rules are going to determine the content of an encounter or treasure (based on EL, or whatever) instead of what the environment or history of the area dictate, verisimilitude is lost.
Vreeg's Rules of Setting design are also heavily immersion related. My current campaign is 26 or so years old (started in '83). Building verisimilitude is a huge part of this.
Vreeg's first Rule of Setting Design
Make sure the ruleset you are using
matches the setting and game you want
to play, because the setting and game
WILL eventually match the system.
Corollary to Vreeg's First Rule
The proportion of rules given to a
certain dimension of an RPG partially
dictate what kind of game the rules
will create. If 80% of the rulebook
is written about thieves and the
underworld, the game that is meant
for is thieving. If 80% of the
mechanics are based on combat, the
game will revolve around combat.
- Multiply this by 10 if the reward
system is based in the same area as
the preponderance of rules.
2nd Corollary
Character growth is
the greatest reinforcer. The
synthesis of pride in achievement
with improvement in the character
provides over 50% of the
reinforcement in playing the game.
Rules that involve these factors are
the most powerful in the game.
Vreeg’s Second Rule of Setting Design
Consistency is the
Handmaiden of Immersion and
Verisimilitude. Keep good notes, and
spend a little time after every
creation to ‘connect the dots’. If
you create a foodstuff or drink, make
sure you note whether the bars or inns
the players frequent stock it. Is it made
locally, or is it imported? If so,
where from? If locally made, is it
exported?
Vreeg's Third Rule of Setting Design
The World In Motion is critical
for Immersion, so create 'event
chains' that happen at all levels of
design. The players need to feel like
things will happen with or without
them; they need to feel like they can
affect the outcome, but event-chains
need velocity, not just speed.
Vreeg's Fourth Rule of Setting Design
Create motivated events and
NPCs, this will invariably create
motivated PCs. Things are not just
happening, they happen because they
matter to people (NPCs). There is no
need to overact, just make sure that the
settings and event-chains are
motivated and that the PCs feel
this.
Vreeg's Fifth Rule of Setting Design
The Illusion of Preparedness is critical
for immersion; allowing the players to see
where things are improvised or changed
reminds them to think outside the setting,
removing them forcibly from immersion.
Whenever the players can see the hand of the GM - even when the GM needs to change things in their favor -
it removes them from the immersed position.
(Cole, of the RPGsite, gets credit for the term).
Remember that part of immersion is the lack of feeling walls around and rails under the characters. This means that the players should not feel that there are things that their character cannot do solely because of the rules or the GM's mindset. The job of the GM is to enable roleplay, not to inhibit it.
This also means the GM must be as immersed as the players, or more.
Another big-picture thing that may irk some folk who sell stuff is that published settings can hurt immersion. They don't destroy it; but when the players have a lot of knowledge about a setting that their character would not have, this increases the opportunity to use it, consciously or unconsciously. Similarly, if your setting has its own bestiary that the characters learn as they go along, or at least a lot of homebrew tweaks, the players get used to working with the in-house data and not trusting the published sources.
If you have done all of this larger-scope stuff, the smaller scope stuff becomes easier. As a GM with miles on the tires, I find that playing up the level of knowledge my NPCs might have and do not have helps keep the players in the same mindset. Players key heavily off the way the GM plays their NPCs. They won't do the funny voices or the mannerisms if the GM does not, and if the GM is particularly careful about what their NPCs know and don't know, especially verbally, the players emulate this.
Consider having the characters designed so that there is some meaningful reason for them to be the ones selected.
I remember one campaign I played (Planescape) in where some red dragon asked us first-level scrubs to go find her precious stolen dragon egg. My response to that was "You must not want it back very much!"
Contrast another campaign (Eberron), where one of the PCs was of the dragonmarked noble house, and the rest of us were retainers or otherwise friendly/bound to him. He gets sent not because he's "high level" (metagame concern) but because he's an important guy, and we get sent because we're his posse. That's a lot better story.
In other words, if these organizations would hire "professional agents" - why are they not generating characters that are professional agents? Or are in that nation's military? Et cetera? The whole "let people generate anything they want and then it's the GM's job to somehow shoehorn them into the party and the campaign" approach is old and busted; don't do it. See this previous question on forming parties for relevant advice.
Of course suitability doesn't have to be "best for the job," it can mean "disposable," "the only volunteers," "some power bloc wants them to go," etc. In the Pathfinder Society, the organized play society for Pathfinder, all PCs have to pick a faction that is then part of the motivating force (and that works behind the scenes to get its agents included in certain missions...).
At a bare minimum, if this has to be a "hire off the street" kind of thing, have them try to make sure that they are at least objectively hireable. I remember one spacefaring campaign where a player's new character couldn't explain any reason why our space freighter crew should hire him. In fact, he was pretty militant about it - we were looking for reasons to bring him on, since he was a PC... "So, what do you do?"
"Things... That need doing."
"Uh, do you have any specific skills we'd find useful?"
"I have many... Skills."
After about 15 minutes of that we took off and left his dumb ass on the planet. (We actually bounced the player from the group too, as being a muffinhead like that was a common character trait of his.)
Best Answer
Evil? We're SAVING the world!
No one thinks they're the villain. Outside of a few very adolescent power fantasy type cults, nearly every other cult is based in imagining it's doing something for either the greater good or at least the good of it's members, and has rationalized all the things it has to do in that regard.
Sacrifice a baby? The baby was evil & deserved it / by killing the baby we actually set it's soul free / the world would end if we didn't / the parents have wronged us and we're stealing the baby's power, they deserve it / it's not even a real baby anyway, this is all a test by (deity), etc.
Cults run on hope, but are backed by social pressure
All cults promise something. They promise something the general society doesn't offer, and most of the time it's some form of secret power, knowledge or salvation. "The rest of the world will fall like fools because they couldn't see the TRUTH. The truth only WE know. You and I? We'll be like gods, while THEY burn! You are wise to join us!" etc.
It appeals to the desperate and secretly greedy.
Once people are in, eventually more and more commitments are requested, and the way to keep people in is the social pressure. They try to cut people off from their families or bring the whole family into the cult itself and then cut people off from social backing outside of the cult (except when, say, they're using those same connections, like a politician or celebrity). They find out secrets, they threaten and blackmail, but it's always under the guise of having to fulfill their dream, the person being "corrupted by the world", or having to protect their own.
The greater the isolation, the deeper the fanaticism goes, and the more extreme the acts of violence and abuse become because people have become normalized to it and their lines of what's reasonable are further and further astray.
Leaders are charismatic
To sell this line of hope and dreams, and to rationalize all the abuse, the leaders are charismatic. Either they're the form of salvation itself, or they're the one everyone wants to be, or be with. These leaders are very good at playing people against each other, using social bullying, rumors, and lies to keep the cult busy and obeying rather than simply fact checking with each other and realizing they're all getting played.
As much as the sexual abuse might be enacted from the leader, there's also a lot of people who will throw themselves, or offer up their partners, or even children to the leader to win favor. It's pretty much the most horrific end result of transactional thinking applied to salvation.
Layered Membership
The cult you get when you first join is not the cult you get when you're in deep. The frontward facing side of cults focused on recruiting is often no different than many other religions. There may be a few doctrinal differences, but otherwise, it's a fundraising and general recruitment pool.
Those who seem especially dogmatic or eager to throw their energy in, or, those who are willing to buy their way in and meet higher and higher cash requests, will be recruited deeper in.
It's harder to say if the fanaticism is a requirement to push the more violent, abusive and bizarre behaviors, or if, even left with relatively normal, vague commandments, fanatics will just find a way to make it violent and abusive anyway. We can see extremists of any religion take it there no matter what the original values taught were...
Taking it to fantasy
The big difference in fantasy is that religious elements are more immediately "provable". You can do magic or you can't. You can summon spirits, or you can't. This isn't a perfect set up, since, once you HAVE magic operating in your world, the reason WHY magic works and what it means theologically can be faked or rationalized a thousand ways.
A wizard with no ties to any deity or philosophy could easily pretend all their powers come from a deity and form a cult. A trickster cleric could pretend to get his powers from another god or power. etc. A ritual might always fail not because it's bullshit, but because "you haven't earned the god's favor, yet."
The other side of it is because magic is real, depending on how rare/common it is, how much people are desperate for that power or what it might promise, people are extremely motivated and possibly more vulnerable to a cult's offers.
If you have a game where magic cannot raise the dead, maybe cults form around seeking this. If you have a game where magic CAN raise the dead, maybe high end members regularly die and are brought back to "earn piety"!!!
All in all, if you're taking real world cults and putting it in your fantasy, just realize they mostly come down to money, sexual favors, and sometimes political push, such as ethnic supremacy cults. Even if the magic exists to (travel to another world, summon their god, etc.) most of the cults actually won't try to really make that happen, instead using it as a carrot to keep the cult itself wasting their energy while bleeding off on exploitation.
This also means that even jaded and well informed people will probably dismiss most cults as anything other than a minor social evil, outside of their abuse and occasional acts of public violence. So, if a cult actually DOES start tapping into real evil powers, just about everyone will be surprised by it...
(Also this means you will see violent fanatical cults supposedly dedicated to good deities as well, which sets up interesting roleplaying conflicts...)