I ran into this too, with the diverse characters and the hard first-session railroad. I became disillusioned because they wanted me to tell a story and I wanted a sandbox, but the result was the same: I had no desire left to run it.
It's awful advice, but my conclusion was that not every game is made to be continued, and it's only a failure if it's pushed beyond the number of sessions it can happily support.
You might be able to completely turn it around, but it's a tall order to change the game into something more like what you'd be happy with when it's contrary to what the players are already enjoying about the game. If you can pull it off, do it. You might consider offering to give up the GMing of this game to another player, and become a player yourself.
But if you can't pull it off, letting the game die peacefully is kinder to you and the other players. The sooner a doomed game is done, the sooner you can start on the next awesome one!
Tell Them Your Goals
If you haven't already, I would start by telling them essentially what you just said here. That there is no "one true plot". Tell them that introducing an evil person / problem does not make it the overriding campaign unless they want it to be. Tell them that you are willing to follow along with their character's background goals.
If they want to play that style of game, there is a good chance that will fix a lot (but not all, habits are hard to break) of it right there.
But its worth remembering that not everyone wants a sandbox. I prefer to play in games with good plots with well laid out characters. While there are some exceptions, GMs frequently are willing to invest more time into plotting and developping good NPCs if they are assured the PCs will focus on them. For that reason, I am perfectly happy to with what you might call "light rails". I want some character autonomy, but I am perfectly happy to follow "the one true plot" if it gets a better story and better NPCs.
By telling them your goals directly, they will either agree to go along, or directly ask you to put some rails back and then you can choose as a group how much of the game should be on rails.
Drop many plot hooks, make them choose
In the game itself, one way to force them to avoid rails is to drop so many potential plot hooks they have to choose which ones to focus on. You can even show that the world is a living breathing place by showing them the resolutions to the plot hooks they didn't take.
For instance, they sit in a bar and overhear of some great evil. They may be tempted to go after it immediately, and there are no rails, so they can! But then someone comes up and asks them for help urgently. And I don't mean Skyrim style urgently where everything is urgent but will wait for you without ever changing, give them an actual deadline measured in something like game days.
Now, they have a choice. Go after the big bad or help this person? In a game on rails, the answer is first help then go after the big bad. Obviously, the person asking for help is just there so they can level up. But you can break that by making it clear the big bad isn't standing still. If they go after him now, he won't have completed his McGuffin of Death and they can stop it before its an issue. But if they wait, they will have to deal with him and the McGuffin. Now, they have a real dilemma and rails won't help them.
To take it even further, highlight something unresolved or some goal on the character sheet. Point out that the world isn't standing still and any time they spend either helping this person or going after the big bad won't help them deal with that personal (to the character) matter. It too may get harder if they put it off.
Time is a great limiting factor in real life, it can be in games too.
Now, they have choices to make and no rails to help, they really have to choose. That is the first and biggest step.
After they get used to dealing with that, especially if you keep highlighting personal goals from the character sheet (and insist they add some if there aren't some obvious ones) they will eventually start adding more of their own, like becoming a dragon slayer. Then they will start actively trying to achieve those goals without further prodding.
Best Answer
What’s wrong with making friends with goblins?
The DMG says (p. 260) that you get XP “[w]hen adventurers defeat one or more monsters-typically by killing, routing or capturing them ...”
That means there are atypical ways of defeating monsters - turning them into friends is one of these.
Naturally, there should be risk, sacrifice and challenge in the social encounter just as there would be in a combat encounter and as DM you are free to adjust the XP award as you see fit. Remember that players respond to incentives: if war-war pays better than jaw-jaw then that’s what you’ll get and vice versa.
Oh, and friendships in D&D always create complications.
A recent session saw us come face to face with a blue dragon. This is in a sandbox campaign so we had no way of knowing if this was a fight we could win. So we humiliated ourselves, flattered and bribed the dragon and lived to fight another day.