First: English is not my native language (and not my players too), so there is may be some missunderstanding or mistakes. I'm sorry about it.
Second: I know about game-recommendation question's rules and I learnt other's questions on this site and reddit too.
Players: Different. There are 1 roleplayer, 1 "commander", 1 min-maxer and 1 "i can play almost anything".
Me: As GM I'm "min-maxer" and I'm always trying to maintain balance between classes, in combat and in adventure (site/event based). I much care about "fair play" and "fun" than about real dice-numbers, BUT I strictly follow the rules of the system if they exist with may be minor balance-homebrew fixes. Currently I'm GMing 3.5/PF fantasy campaign. I can learn the system of any complexity, but I should have a chance to explain it to my players (which not-reading, not-English).
Setting and Adventure's hooks: Almost any setting with fast travels between star systems (Earth may not exist or be wiped/destroyed). Aliens is acceptable and desirable (I hate only-human-based Sci-Fi settings). PCs are long-distance truckers, or/and traders/explorers. Adventures are unclear/mystic cargo, troubles with payment, old-story problems, new places/cosmos exploration (yeah, I like Firefly so much).
Not-to-suggest: Savage Worlds, Shadowrun.
Best Answer
Traveller. Traveller is an old game that's had a lot of editions, and several of them do what you want. It's... Well, it's basically Firefly. No, seriously: It's a game where the players are the crew of a small independent starship that hauls cargo and passengers between star systems to stay out of the red. I've described its features below, with the points that seem most relevant to your requirements in bold.
In most iterations, it's a traditional-style game, and should be flexible enough to accommodate the playstyles of all of your players. Balance is... A bit different to how it works in D&D. Characters don't have levels, and while advancement is available (in the form of equipment) for the most part, characters don't end a campaign a whole lot stronger in combat than they began - they just get a lot smarter about not dying. This means that the reward structure is different (it's about credits, not experience, and so combat isn't so central a theme), but it's not so different that it'll feel unfamiliar, and nearly player can find a mode of play that they feel comfortable with.
The setting is easily customisable (to the point where there's sayings about it in the player community) and every edition of the game has had strong and extensive rules for randomly generating vast swathes of space, so there's no risk of running out of content if you go "off the map."
A variety of alien races exist (and may or may not be playable, depending on the edition and the GM and which race you're talking about).
Long-distance trucking is the default assumed mode of play, with smuggling, mercenary work, piracy, asteroid mining, taking job offers from sketchy employers also being pretty common alternative sources of income. Most games start the players out with a considerable mortgage to pay off on their ship, so players are often willing to take on sketchy, risky or outright illegal jobs in order to make ends meet and keep on flying - which makes those rare jobs where everything goes right and you come out ahead seem like a real triumph, and gives you, as a GM, the ability to dangle all sorts of plot hooks in front of them and know they'll bite. Also, many planets both in and out of civilised space are underdeveloped and unexplored, so exploration can be as much a plot hook as any of the others I mention.
Character creation in most editions is a minigame where a player goes through their character's prior life in four-year blocks, deciding what career choices they made and rolling randomly to see what significant events occurred in that time. This means that A) character creation is more interesting than most other RPGs out there, B) Every character starts out with a roughed-out past that may come up in play, and C) Min-maxing is about risk management rather than point-buy.
In the last campaign I played in, the crew was a disgraced former army captain, a sociologist with a history of getting stranded on backwater worlds, a noble ex-diplomat who was 'slumming it', a pilot who had previously worked as a scout and belt-miner, a ship's doctor who was an illegal psionic, and an ex-intelligence operative trying and failing to stick to the straight-and-narrow. We started out with an rusty asteroid mining ship and forty years of mortgage to pay off before it was ours. Worried we might not make our first payment, we accepted a job from a military contractor who was suspiciously generous with pay and light on details about the work - work which consisted of seeking out a gone-dark installation on the other side of the subsector.
That... Went sour quickly. Only a few weeks later, we'd gained a criminal record on one world, barely managed to avoid getting caught on a second after an attempt to steal medical supplies went sour, and had our ship (technically) sold out from under us by a sneak before we managed to sort things out - and the military contractor was worse: After being jerked around on a string for a while, we decided to skip out on our mortgage, and spent some time fleeing imperial space for the border worlds. (Like I said, it's basically Firefly.) The campaign ended shortly after that, but it was awesome while it lasted.
I recommend Mongoose Traveller over other editions of the system; It's easily available (It's still in print), not too complicated (Beware T5), and contains some of the best features I've described above.