Better Teamwork
Bluntly put, the rest of the party are not doing their job if foes are attacking wizards and bards. The whole point of fighters and paladins and barbarians is to keep the baddies away from the artillery and support.
Remind them gently that if you are paralyzed by a ghoul then you can't cast that nice ranged healing spell or give them bardic inspiration (so the paladin can get off those wonderful smites). If the wizard is surrounded then she is going to be favouring short-range things (like burning hands) with chances of friendly fire (which isn't friendly).
Remind them that if you have to run back to avoid foes then you will be out of healing word range of the front-line fighters.
If they don't take the hint then just stop buffing them. Explain that if they are letting ghouls run past them then you are going to be spending your bardic inspiration dice on psychic blades, not inspiration, and you are going to be saving the healing spells for yourself.
Better Intelligence
You are the best character for scouting, with disguise self and good deception.
Ambush
Minor illusion is the perfect ambush spell. Make an illusory barrel or crate or outhouse and hide inside it.
Optional Rules
If your game is using the shove aside, tumble and overrun rules from the DMG then you can get good use out of your Expertise in Acrobatics.
Terrain
Fog cloud and darkness and similar spells can help keep your foes off you, or at least make it harder for them to hit you. Caltrops, ball bearings, and flasks of oil can also control where foes move. Illusions of bonfires can do the same - get creative with minor illusion.
Gear
You should be pushing for any magical light armour the party finds. The monk and barbarian won't be wearing it and the paladin and fighter will be in medium or heavy armour.
Being Passive-Aggressive
Wouldn't it be a shame if your fear spell happens to include a party member in its area of effect...
Wouldn't it be a shame if someone learnt the nasty little secrets that the party members have told you, in confidence, during those long hours on watch...
Other
Remember that Jack of All Trades allies to Initiative rolls. This will help you go before your foes.
Don't be a Wizard. Be a Divine Soul Sorcerer.
If you're explicitly not going for any of the utility spells, then you correctly see that there aren't that many good offensive or defensive spells that don't require you to have a good casting stat. My suggestion would be that, rather than going Wizard, you go Sorcerer, with the Divine Soul sorcerous origin (Xanathar's, p 50). In short, this sorcerous origin lets you pick spells from both the Sorcerer and Cleric class lists, as well as giving you an ability to add 2d4 to a save or attack roll once per short rest. You'll have a lot fewer spells, since you're a Sorcerer, but you'll have a lot more versatility in how you cast them.
My suggested spell list up to 3rd level would be:
Cantrips: Booming Blade, Green-flame Blade, Guidance, Resistance
1st: False Life, Shield of Faith, Shield
2nd: Mirror Image
In addition, you get one more 1st level spell depending on your alignment from the Divine Soul sorcerous origin. Law is a good choice for you, since bless gives you extra bonuses to attack rolls. I'd also suggest taking the Quickened and Twinned metamagic abilities, so you can cast while attacking and buff your allies, respectively.
Best Answer
Looks OK
I play a divination wizard. Overall, your list looks solid. I would not swap mage armor for something else, I would rather drop mirror image for something else.
Mirror Image is not going to help much with battlefield control, and you already have good defensive resources with mage armor and shield. I do have mirror image, but rarely if ever use it. I used mage armor and shield a lot. Mage armor holds for much longer, it only consumes a level one slot, not one of your precious level 2 slots. You also can use it to bolster your familiar's AC. If things go right, the fighters should cover you from attacks anyways, so you do not want to spend your first round casting mirror image which you may not even need, when you could instead incapacitate a key opponent. And if an opponent gets too close for comfort, you can always misty step away.
I would consider web in its place, which is a really solid level 2 spell for control purposes. If you want to take out single targets with Portent, suggestion is another great choice, that again works against many opponents, and has many other flexible uses beyond combat.
Other observations:
Magic Missile. If you goal is control/disabling instead of damage, you can consider Hideous Laughter on level one instead of the missile. It does allow for resaves, but it removes an opponent from action for a lowly level one slot, and it not limited to humanoids. I have taken dragons out of action with that spell, it will be useful even into the higher levels, and even removing a strong opponent for one round can save your team a lot of hurt.
Mind Sliver. Using mind sliver to push through other spell effects is a questionable strategy, because you have to burn two actions on it, and if they make the Int save, you just wasted a full round doing very little. It can help if you first target them with the main effect, and then use this to make it harder to get out of it with resaves.
Hold Person is solid because it paralyzes the opponent, and that is nasty when your buddies can attack them and may be worth it even for a round, but it does allow for re-saves every round, which is suboptimal if you want to maximize your Portent. You could instead pick levitate, which can take melee opponents out of the running without any resaves, and has nice side benefits as a utility spell.