Set a Firm Rule: Disagreements about Rulings are to be handled between Sessions, not during Sessions

Generally speaking, a player should not interrupt the DM to quibble about how rules work. What the DM says is final, and if players don't agree with how the DM is handling rules, they should wait until after the session is over to resolve disagreements.

So with this player, you need to be very clear and direct in telling them that, even if they're totally right, they need to wait until the session is over to discuss mistakes or changes to the rules. Then, based on that discussion, you can agree to retcon certain events in the prior session, or you can agree to follow different rules going forwards, or you can assert that you're going to be following your understanding of the rules instead.

But make sure you understand the rules as well as possible

As DM, you have a responsibility to make sure you understand the rules as well as you can. Players have specific class features, actions they can take during combat, spells they can cast, and so on, and so forth. If you don't follow the rules to the best of your ability—or you don't explicitly signpost when you're deliberately not following the rules-as-written—it makes it harder for players to make informed decisions about how to play their characters.

This isn't even necessarily limited to changes to the rules or homebrew material: A Wizard will probably keep Identify in their spellbook to help identify magic items they acquire, and if you're using the variant rules for Magic Item Identification (DMG, pg. 136), where the Identify spell alone isn't enough to fully reveal the properties of a Magic Item, they might get upset if you don't inform them that you plan to use those rules ahead of time, since that was a character creation decision they made under false premises. Now, that's a relatively minor example, and easily handled at most tables ("Alright, I'll let you swap that spell out from your spellbook with a different 1st level Wizard Spell for free") but more substantial deviations could be more frustrating to players, because if the rules aren't well defined, then players are going to feel like they can't make meaningful decisions.

You should probably avoid homebrewing mechanics if you're new to the rules

Related to the above, introducing new mechanics without fully considering how they interact with the world they've been introduced to can have negative consequences.

One example that came up in a recent campaign I participated in was my DM's decision to add a new creature type to the game to represent the central villains of the campaign. I believe the DM's intentions were to create a villain that was suitably alien to the Faerun setting, and arguably, they succeeded. But the issue then became that this new creature type had no interaction with any of the abilities or features that any other character had:

• They were a NEWTYPE, so spells that say things like "affects Aberrations, Celestials, Fey, Fiends, Undead, and Elementals" wouldn't affect them.
• Their special abilities were of NEWTYPE, so, for example, a Paladin Aura that says "gain Resistance to Spell Damage" wouldn't affect their "Spell-like" abilities
  • This also meant their features bypassed things like Anti-Magic Fields
• Their Damage Type was NEWTYPE, meaning no feature that grants Damage Resistance (or Vulnerability!) could interact with it, unless it read something like "Resistance to ALL damage", which is a very small subset of features

As a result, it was impossible, as players, to make meaningful decisions with respect to how to prepare for fights with these kinds of creatures. We couldn't revise class features without just homebrewing them, and we couldn't prepare spells to deal with these creatures because most of our spells wouldn't work on them at all.

Eventually, we were able to convince the DM to incorporate NEWTYPE creatures as a presumed addition to the "Aberrations, Celestials, etc." lists that many spells have, which allowed us to actually make tactical decisions when approaching these kinds of encounters.

^{Also in general he's dramatically dialed back how frequently they show up in encounters; we've gone more than a dozen sessions since the last time we ever fought a creature of that type.}

The point being, while homebrew mechanics can make the game more interesting, or handle scenarios that the base rules don't handle especially well, you have to spend time making sure that they interact with the rules in a balanced and fun manner, or you'll end up creating mechanics that either dull down the game, or rob your players of meaningful agency.

Getting better at handling rules

One thing that I do as DM is I'll spend a lot of time trying to examine edge-case scenarios in D&D to better understand the interactions of the rules. Sometimes, if I see something strange, like a long-range archer improving their attack odds by blinding themselves, I'll expressly decide that not following the rules-as-written is necessary to improving the game.

But here's the trick: even if I have no plans to follow the rules-as-written in a specific scenario, it's still important that I understand what the rules are, and why they were written that way, because that helps me better understand what problem I'm trying to solve, and better justify my decision to not follow that rule.

So given that these disagreements over rules are affecting not just your relationship to this player, but your relationship to other players, it's important you make sure you understand what the rules are, and develop a good justification for not following those rules as written. Planning encounters when you don't know what kinds of mechanics might interact with them is going to end in frustration, even if the players know better than to express that frustration in real-time at the table.

Perhaps have a conversation with your players about what kind of game they want to be playing

D&D historically has its roots in Wargaming gameplay mechanics. So as a broad principle, a player "playing to win" is not a strictly invalid method of playing. D&D, even 5th Edition, facilitates this kind of metagamey, "here's how we strategize to win!" gameplay, and many groups that don't play in this manner often do so by eliding mechanics of the game that they feel don't contribute to the gameplay style they want.

However, the evidence strongly suggests that that's not the kind of game you want to be playing. So what's very important is that you sit down with your players and decide what kind of game you want to be playing. If all of you are in agreement that you want to be playing a Wargame type game, or if you're all in agreement that you want to be playing a more social, less combat-focused game, then things will be good. But if you're each trying to play different games, then you're only going to have further problems.

Finally: Communicate with players

A lot of issues at the table can be resolved by openly and honestly discussing what isn't working, and what needs to change.

I've emphasized as much through implication, but if you're willing to talk to your players about what frustrates you about the gameplay, and they're willing to listen, I'm confident you'll find a solution that works for everyone.