Are you familiar with the Same Page Tool? It sounds like you had expectations that you tried to convey subtly to the group in-game, but this wasn't overwhelming enough to overturn their existing expectations or the conflicting messages being sent by your campaign kickoff's dominant tropes, so you weren't on the same page.
Getting on the same page is the first step toward your players having their PCs act like real people – you can establish your expectation, and the possibility, that they can ignore the common game-like meta structures that are often taken for granted around a roleplaying game. Things like they are all a party and cooperate and they're aiming for survival of their PCs rather than character embodiment are common assumptions that are often necessary for certain kinds of campaigns, because it is easy and expedient to say that some play possibilities are "off limits" in order for the group to focus their game time on the fruitful voids of the campaign.
What appears to be the problem is that you have tacitly broadened the fruitful void, without notifying the players of this strongly enough to make them set aside their (productive, functional) RPG-playing conventions. On top of that you set up the campaign with a very standard "form party, loot dungeon" structure, which strongly conveys a standard "dungeoncrawl campaign" set of expectations that are the opposite of what you seem to have hoped for.
How to get them play their PCs like real people is then a two-step process: first, clearly grant them the breadth of allowed play possibilities necessary to be able to play them like real people. Second, cultivate a group value of character embodiment.
The second is the hardest part actually, and the how can't be covered here because the barriers to doing that are personal and depend on your players. Given that they haven't even yet become aware of the possibility with the first step, I have no data to even begin giving advice on the second.
So, that makes the first step very important: get on the same page, eliminate the assumptions about how to play a dungeoncrawl-type fantasy RPG, and replace those assumptions with explicit understanding that you're aiming for humanist drama in a fantasy context. Once you've had this conversation, only then can you even find out whether your players are interested in embodiment-focused play and what their individual barriers for that are, if any. Be aware that they may not be interested in this kind of play; be prepared to have a conversation that is about negotiating a common ground, and it may not lead to the sort of play you're looking for. It's possibly you'll all get on the same page, but if you can't, that may mean the group can't continue – but that's better than forging ahead with conflicting expectation and play goals.
You've run into a common problem - "Party RPGs with non-Party Characters". Same Page Tool can't fix groups who want different things, and it also can't fix game design that works against it's own game premise. You have a few options:
Class Limiting
"Hey, we're playing X kind of game and these classes/types in this
game don't fit that. Can we just not use them for this game run?"
Games that usually have classes antithetical to their goals usually also have a pretty broad set of class selection, so it's usually not too bad in terms of choice limiting. The other half to deal with is the social contract of your group.
(There's also a subset of gamers out there who deliberately pick the most contrary ideas to what you state the game is about. "Dude, why do you have a Navy Seal character in our game about civilians running from monsters?" Those players are their own problem...)
Building with Limitation in Mind
"Hey, for this kind of game I want to run, these kinds of characters
will need to fit these kinds of situations. Can you spend your
points/pick your skills/build your powers to better fit this?"
This is a relatively good option - you can get stuff like "combat rogues" and such that are better designed for situations rather than splitting off. This depends a bit on the system's ability to allow customization or choices within the class system, and also lets players know up front what they need to consider with a character class build.
Non-Party Play and Strong Pacing
If you can run a game which isn't dependent upon a party structure, all those character classes generally work fine as long as their goals and concepts line up. In these kinds of games you need to be able to cut scenes relatively quickly, not spend a lot of time on wasted scenes and the players need to have good goals to aim for.
That said, usually systems that are more mechanically light work better for this than ones attempting to balance out a lot of abilities, though games like Burning Wheel or Blade of the Iron Throne can work fine for it, mostly because the basic resolution systems allow for quick play and give good goal-building tools in the form of Flag mechanics.
Best Answer
Creating slang and dialect is an art, not a science, and there are two basic strategies: invent it, or steal it.
If you invent dialect, don't invent words
You can see that the slang used in Planescape isn't wholly made up--not even in real life. Sometimes slang is a reference to something (like a famous hunt), but most often it's just a matter of using existing words in unusual ways.
If you decide that fleugalsnorff is a local word for rice pudding, nobody's going to remember it at your table, especially if you make up a lot of words like that. But if rice pudding is called the jiggly, it's a lot easier to remember... and it's a lot more evocative, too. (I use the same technique for naming people, towns, nations, geographic landmarks, etc.)
Common words put to new use are easy to remember and give a good solid sense of the way a culture thinks.
Use words that people have forgotten
Quisling is an awesome word that most people have forgotten entirely. Bring it back!
Etymologists collect lists of words like that, and popular authors like Bill Bryson write entire books on the subject (not always entirely accurate, but that's irrelevant for our purposes).
But that's not how most of the greats do it:
Don't make it up: borrow and steal
If we look at the Planescape slang you mentioned, it's obvious that whoever came up with the Planescape dialect wanted to invoke a lower-class British atmosphere.
This is the most common and effective way to create dialect or slang: shamelessly rip it off. Tolkien's Dwarven script is inspired by Semitic languages, and just about everything on the planet Arrakis has Arabic or Islamic origins. For the record, this strategy is used not only for dialect and slang, but for culture, geography, history, art, and just about anything else.
First figure out what feeling you want to evoke
This requires some self-awareness, so sit down and write the things that are essential to you about the person/place. A strong caste system? No permanent home? Fetishizes the rules of hospitality?
Then find a real-world counterpart
You may have to do some research, but if you can identify a culture or period in real-life history with at least some of the elements you wrote down in the first step, you're practically done already.
You may be surprised to find that the cultural idea you thought was brand-new and original is actually very similar to something in the real world: our brains often pick up bits and pieces without us noticing, and then we put them together in new ways.
Use what you like
Commonly you won't find a single culture that fits your vision. That's great: take two and squish them together, discarding the bits that don't mesh. You may find new ideas as you read up on the cultures, things you'll want to add in.
...And don't be afraid to lean heavily on tropes for shorthand. Tropes are tools, learn to use them well.