By RAW: you can use a dart with Defensive Duelist. It just asks for a finesse weapon, not necessarily a melee weapon.
However, personally I would rule that it would require a finesse melee weapon, and I imagine it would probably be changed in a future version of the Errata to specify finesse melee weapons. Darts just aren't something you'd expect someone to even attempt to deflect a blow with.
To answer your question directly:
No, making this change would not be game breaking
This would impact the damage done by certain character builds - that is true. It very well may make certain character build less fun (like a strength based rogue).
Game breaking is a strong phrase. I don't believe that making this change (or not) would have that big of an impact to a campaign. Campaigns with Barbarian/Rogues can exist with no changes. Campaigns without a Barbarian/Rogue can exist as well.
A question that you really want to answer is: what mechanical problem you are solving by changing the mechanics of how abilities function?
This proposed change would impact strength based rogues. Some players find that in combat, more damage is fun. If you have a player like that in your group, playing a strength based rogue, this will directly impact how much fun that player can have in combat.
If this seems silly, or doesn't make sense, (but isn't causing mechanical issues) my recommendation would be: resolve that issue without changing the ways abilities work.
I see the hilarity in a barbarian being sneaky while also recklessly attacking. However, I'm very averse to changing abilities or mechanics as substantially as this without a clear mechanical problem that the change would fix.
As much as possible, I try to avoid 'fixing' things that aren't yet causing a problem. If you think this may cause a problem in the future, point that out to your players and make sure they clearly understand that if it becomes a problem, something may have to change.
Would making this change (or not) be game breaking? In and of itself, no.
It very possibly may hurt player fun though.
Best Answer
A weapon is only a ranged weapon if it is listed as a simple/martial ranged weapon. Every such weapon also happens to have the Range property
Note that fifth edition defaults to Standard English when something is not defined in the rules; however, what a ranged weapon is, is defined. The section on "Weapon Properties" includes the following:
If something requires a heavy weapon this doesn't mean the weapon needs to reach some arbitrary weight but that the weapon must have the Heavy property. Unfortunately this analogy doesn't hold for figuring out what ranged weapons are because it is not the Range property that matters but simply whether or not the weapon is classified as a ranged weapon.
We can look at the description of the wand and its statblock and see that no part of it states that the wand is a ranged weapon, nor is it listed on the weapons table as a ranged weapon.
Another way we could deduce this is from the fact that every single ranged weapon has the Range property and since the wand does not have the Range property, it can't be a ranged weapon (proof by contrapositive). But be careful, weapons with the Thrown property also have a range listed despite some of them not being ranged weapons; having the property is necessary but is not sufficient to show that something is a ranged weapons.
Ultimately, if something is a ranged weapon, it will appear in the ranged weapon section of the table, unless it is an exceptional case such as the Storm Boomerang, which is not on the weapon table and exists in a module for the game released well after the Player's Handbook and its weapon table. (Princes of the Apocalypse to be exact).
That said, a wand is not even a weapon
Wands are not listed in the Weapons Table and their statblock does not call them weapons so they are not weapons. For some more reading there is also the following Q/A: