I've found that the easiest way to stop your character getting dominated in big fights is to pump up your health each level increase.
On my last character, I was playing an archer whom I assumed would rarely get into a close-combat situation that I couldn't just kite out of. I spent each level increase on stamina so I could run further (as part of the hit-and-run tactic I was trying). It was great, except for every so often when I'd come up against something that pinned me down or shot at me from a distance. I was about to restart, but instead started investing in more health. All of a sudden, the game started to get easier. That character got to level 51 with over 300 health.
Other tips would be:
Invest in skills that increase your base damage for your weapon type. For example, maxing out the base skill for one-handed doubles your damage every single hit.
If you're looking for superior equipment, invest some skills in Smithing. If you go along the RIGHT hand side of the skill tree, you can craft the strongest weapons/armour for a light or heavy fighter or archer and improve them to legendary quality.
For fights against dragons and powerful enemies, keep some potions of fortify archery/one-handed/two-handed/destruction or whatever you're using to increase your damage. The sooner they die, the less damage they can do.
Carry potions of restore health attached to a hotkey so you can quickly restore health if you need to (if on the PC version, "favourite" them in the items menu, then press Q when out of the menu to open your favourites list. Move the selection over the item and press 1-8 to attach it to that hotkey).
Carry potions of resist frost/fire/shock/magic. A resistance of 40% almost halves the damage an enemy relying on that type of damage can do.
You can save money if you make these potions yourself. Consider investing some points in Alchemy (just the base skill, physician and benefactor is enough. Print an alchemy effects guide to find out what ingredients do, or just eat one of each ingredient to find an effect, then discover the rest through trial and error). Restore health, for instance, can be made quite simply with Blisterwort or Blue Mountain Flowers and Wheat.
If you rely on melee weapons, consider carrying a reasonable bow and some arrows (the arrows are weightless, the bow is fairly light too). In some situations, jumping on top of some rubble and shooting that warhammer-wielding master vampire is the smartest choice.
Let your companion attract the attention of nearby enemies and take the hits. They sound like a jerk for letting you die so often anyway. Target an enemy that is attacking your companion, so that you get a few hits in before they even realise they're being attacked by someone else.
If you let us know what character type you're playing, we can give more specific advice. Hope this helps for now.
Is there anything that I've done incorrectly for my build, in that I shouldn't have tried to get the best gear midway through the game?
This is very subjective - what is considered an "incorrect" build by one player, may be "optimal" for another. If you want to make the game harder, focus your skill points and perks on non-combat skills: Speech, Pickpocket, Lockpicking, etc. You can also put skill points and perks on combat skills that you won't be using in combat. This will level up your character (which will make enemy level difficulty scale to your level), while at the same time, not make your character stronger for combat.
With the Dragonborn DLC installed, perks may be undone and redistributed at the cost of one dragon soul per skill tree.
At the end of the At the Summit of Apocrypha quest, you will have access to different portals (one for each skill) which allows you to clear and regain any perks in that skill tree, at the cost of one dragon soul. You remove all perks from a single skill perk tree and can use these reclaimed perks on unlocking any perks you wish, including perks taken from said perk tree. By reading the Black Book, Waking Dreams, you can return to Apocrypha and alter the skill trees whenever you wish.
Are there any hard quests that I've completely missed, or do you think that now if I start new questlines with my increased stats, the game would level up the difficulty for me?
It is subjective to state which quests are "Hard quests". Just have a look here for a list of all quests and see which ones appear to be hard for you. As for the second part of your question, the unmodded game only checks your current level and the difficulty settings in the options to compute how difficult the enemies will be for you.
Excerpts from UESP wiki's "Leveling - Effects of Leveling" article:
Various aspects of the game are leveled. This means that as your character increases in level, some enemies become more challenging but also the quality of the items you find becomes better. However, the leveling system in Skyrim has been altered from that used in Oblivion, in response to criticisms of Oblivion's leveling system.
Different locations in Skyrim have different inherent difficulties. In other words, some dungeons are designed to be too difficult for low-level characters to enter. More challenging dungeons are generally located at higher elevations, meaning that early in the game players may want to avoid mountainous regions. However, more difficult dungeons contain better rewards. In addition, some high-quality items can be randomly found even early in the game.
... Bandit NPCs are always a fixed level for their name (Bandits are level 1, Bandit Thugs are level 9, Bandit Highwaymen are level 14, etc). The player's level affects the range of possible bandit types generated within a bandit dungeon, and probably the frequency, but does not seem to affect the resulting stats except in a few rare cases. Lower variant bandits remain reasonably common even when more dangerous bandits are available.
v1.9 Patch - 'Legendary' difficulty
Patch 1.9 adds a sixth difficulty level: Legendary. It reduces damage dealt by the player to x0.25 and increases damage taken by the player to x3.
Mods
If playing on the PC, you could also make the game harder with mods:
The mod, Pluto's Improved Skyrim Experience (PISE) has a "More Intense Level Scaling" component. It makes enemies more stronger relative to your level, compared to the vanilla enemy scaling. PISE also features harder sneaking, more enemy spawns, harder enemies and an overhauled enemy AI. The mod, Path of Shadows, a major stealth overhaul mod, also makes sneaking harder.
Other mods that may make the game harder:
- DFB - Random Encounters - adds different random encounters from vanilla: Vampires, Falmers, Werewolves, Dwarven Spiders, Spheres, Centurion, etc.
- High Level Enemies - has a feature that allows certain or all enemies to scale with your level
- Deadly Dragons - overhauls dragons to make them more challenging to fight
- Balanced Magic - designed to 'balance' the game's magic spells - make the spells do damage and consume mana appropriate to your magic perks, and lessens the effect of abusive perks like the 100% stagger change of the 'Impact' perk. It also affects enemy mage NPCs, making them more challenging
- Auto-cast Racial Powers Plugin - "Auto-cast racial powers will activate for NPCs, both enemy and friendly, making fights much more interesting and adding a new element to prioritizing targets in larger fights!"
Creature mods that increase creature spawn points, number of spawns, and also improves creature AI:
(I recommend using only one creature mod, to avoid conflicts and other issues.)
Best Answer
Googling a variety of different search terms eventually brought me to this thread where someone had, through a process of elimination, identified vibMUTEXSCT.pex in the scripts folder as the culprit.
Moving this file out of the folder allowed me to finish the quest, though I still need to work out which mod this file is from to disable it.
Edit: turned out to be Immersive Beds. (I worked this out by scrolling through my mod list in Wrye Bash looking at the file list in the right-hand pane for each)