Is it possible to remove Gwent mods without starting a new game

I completed that particular bit a few hours ago. If I recall, nothing happened after I'd beaten him, I'm pretty sure it was just to signify that you could play gwent against him.

There are at least 2 quests where being able to beat people at gwent gives you additional options to finish the quests.

There are gwent-specific sidequests where you need to beat specific NPCs at gwent. There is even a gwent tournament sidequest, which you can only start if your gwent deck has a sufficient number of high level cards.

There are gwent achievements too.

So if you are one of the people who wants to get all achievements, or finish all quests, you must play gwent.

If you only want to finish the storyline missions, you don't need to win a single game of gwent. This is my experience in the base game version 1.03, without any expansions.