[RPG] Does the Defensive Duelist feat work when wielding ranged finesse weapons

No, for three reasons:

First, Rapier specifically states:

You can’t wield a rapier in two hands in order to apply 1½ times your Strength bonus to damage.

Second, Weapon Finesse allows you to use DEX mod instead of STR mod on attack rolls. Wielding a one handed weapon in two hands allows you to add to your damage rolls. Two completely different cases.

Third, you cannot use Weapon Finesse when wielding two handed weapons, or wielding a one handed weapon in two hands. There are a couple of specific items that change this ( iirc, Elven Courtblade from Races of the Wild and the cheesechain Spiked Chain ), but it is limited to those items.

No.

The key here is that the Monk weapon ability doesn't actually add the keyword "finesse" to the weapon, it just lets you use Dex for attack and damage.

Defensive Duelist specifies that a weapon have that keyword, and thus, no, it can't be used.

Adding more fuel to this fire is Jeremy Crawford talking about natural weapons and coming down on the side that just because you are attacking with Dex does not mean you are adding finesse:

No. A weapon has the finesse property only if its description says so, and using Dex. to hit doesn't equal finesse.

Would this be a game breaking houserule? No, that's quite unlikely. The Staff does d6 or d8 damage, and there is already a finesse weapon that does d8 (Rapier). The monk isn't proficient in that, so it's not available, but it's not out of bounds for them to be able to use it.

Note: Brian points out that two monk weapons are finesse weapons, and as such would be usable with this combination (and get the Monk's proper monk weapon die). These are the dagger and the short sword.