[RPG] Should I request an ability check only when the player asks for it

I would say Y has to do it since Y does not see or know the danger involved in doing what you are suggesting. The only thing that I can see wrong here is that if Y somehow knows that what you are suggesting is 'eventually' dangerous to him.

I base this on the description of charm/dominate where it clearly states that the effected will perform any action that does not pose a threat to his own health. Literal copy of the RAW (I used the 3.5e rules here since they offer a nice example which the Pathfinder version does not) for charm person:

An affected creature never obeys suicidal or obviously harmful orders, but a charmed fighter, for example, might believe you if you assured him that the only chance to save your life is for him to hold back an onrushing red dragon for “just a few seconds.”

The suggestion that you made to have Y move to a certain square sure seems to be a lot less dangerous than what is described above, so if it works with charm it should work with dominate/diplomacy in my opinion.

Do you like your rpg old or new school?

You have asked a question that lies at the heart of the debate between "old-school" and "new-school" rpg. If you don't know what that is, then there are plenty of places on the net where you can have your brain bashed by intensely partisan views on both sides. I don't think its useful to go into this here.

At the core is the question "Who is detecting the untruth?". Is it the player (old-school) or the character (new-school). If it is the character then rolling the dice is a perfectly acceptable way of deciding. This seems to be causing you some cognitive dissonance (your old-school roots are showing ;-) ).

The simple answer

This is where I give you the mandatory and largely useless answer - you are the DM, however you decide to do it is perfectly correct.

Now that that is out of the way, I will tell you what I think should happen.

The complex answer

From the Players Handbook p.6:

How to Play

The play of the Dungeons & Dragons game unfolds according to this basic pattern.

1. The DM describes the environment.

2. The players describe what they want to do.

3. The DM narrates the results of the adventurers’ actions.

There is not one single word in there that says that anyone rolls dice. So you saying "... as (in my opinion) a player should only make a check when the DM calls for it" is spot on; nobody rolls dice to resolve anything unless you ask them to. Obviously, in a combat situation, your asking may be implicit but otherwise, the players' should be telling you what their characters are trying to accomplish; they describe the outcome they want, you determine the process. This may range from "You do it" to "Are you kidding me? That's clearly impossible."; its only when the result falls between these extremes that a die roll may be the appropriate resolution.

This position is clearly supported by this paragraph (p.186)

Ability Checks

In addition to roleplaying, ability checks are key in determining the outcome of an interaction.

Your roleplaying efforts can alter an NPC’s attitude, but there might still be an element of chance in the situation. For example, your DM can call for a Charisma check at any point during an interaction if he or she wants the dice to play a role in determining an NPC’s reactions. Other checks might be appropriate in certain situations, at your DM’s discretion

A player saying "I roll an Insight check" deserves the response "The NPC wonders out loud why you have suddenly taken a dice out of your pocket and thrown it on the table. "Do you want to play craps?" he asks.

Alternatively, you can do what I do. Since the player seems to love rolling dice I say "Go ahead" and then completely ignore their roll.

When wouldn't you roll

Going back to the flow of the game:

1. The DM describes the environment.

You know if the NPC is lying or not and, if they are, you know why. Interacting with the NPC should be one of the ways the players can find out if they are lying. It is important that the players know this then there should be at least 2 other ways of finding out (See The Three Clue Rule).

Describe what the NPC says and how they say it so that there are cues that clever players can pick up on to determine if they are lying. Signs of the truth may include "calmly", "quietly" etc. Signs of lying include "sweating", "shifty", "slowly" etc.

If they are lying then look at their deception score (or modified roll if you want to roll) - if its high then give fewer clues than if it is low. Personally, if rolling I would make this roll after the interaction was over; the frequency and obviousness of the cues I give would be based on the NPCs Charisma (Deception) modifier.

It is worth considering what the typical ranges of Charisma (Deception) modifiers are:

• -2 ( 6 CHA non-proficient)
• 0 (10 CHA non-proficient or 6 CHA proficient (L1-4))
• +2 (14 CHA non-proficient or 10 CHA proficient (L1-4))
• +5 (20 CHA non-proficient or 16 CHA proficient (L1-4))
• +10 (30 CHA non-proficient or 22 CHA proficient (L9-12))
• +16 (30 CHA proficient (L16-20))
• +17 (20 CHA expertise (L16-20))
• +22 (30 CHA expertise (L16-20))

A few points to note:

• an average person without proficiency has +0,
• the most charismatic person without proficiency has about +5 - equal to a proficient low level sorcerer
• the most charismatic con-man with a lifetime of experience has about +17

From this, I have the following rule of thumb:

• 0 or less: this person is going to give an obvious sign every time they lie.
• 1-5: this person is uncomfortable lying; general physiological signs and obvious signs if the lie is blatant.
• 6-10: happy to tell lies; minor physiological signs, distortions of the truth are undetectable, outrageous statements may cause them to blink.
• 10-15: accomplished liars, con-men, lawyers, politicians (non-pejoratively of course!), no physiological signs, maybe some slowness of speech as they have to think about the "correct" response instead of the true one.
• 16+: You get nothing.

If it is truly important that the PCs think he is lying just say "You think he's lying"!

2. The players describe what they want to do.

Is it enough for them to say - "We think he's lying?" Well, is it? As a DM my response to that is "OK, but what do you want to do? How do you catch a liar?"

Good players may well ramp up the stakes, you catch a liar by making them say something you know is untrue and that you know they know is untrue (otherwise they are just wrong, not lying). If they can do that, then there is no need for a roll.

What is this "insight" anyway

Let's have a look at what Insight actually says (p.178).

Insight. Your Wisdom (Insight) check decides whether you can determine the true intentions of a creature, such as when searching out a lie or predicting someone’s next move. Doing so involves gleaning clues from body language, speech habits, and changes in mannerisms.

Note: this is stronger than just detecting a lie - it gives you insight into the true intentions.

and for comparison, what Deception says (p.178).

Deception. Your Charisma (Deception) check determines whether you can convincingly hide the truth, either verbally or through your actions. This deception can encompass everything from misleading others through ambiguity to telling outright lies. Typical situations include trying to fast-talk a guard, con a merchant, earn money through gambling, pass yourself off in a disguise, dull someone’s suspicions with false assurances, or maintain a straight face while telling a blatant lie.

Clearly these are mirror images of each other. There are therefore four obvious alternatives to resolving a conflict between them:

1. Passive Insight vs passive Deception - No randomness at all, basically whoever has the higher bonus wins with ties going to Insight. This allows for no degrees of success or failure.
2. Passive Insight vs active Deception - Equally skilled participants will have a 50% chance of winning and a range of possible degrees of success/failure.
3. Active Insight vs passive Deception - This is the mirror image of No 2.
4. Active Insight vs active Deception - This gives a much wider range of possible results but the general rule is that a choice to use insight actively should not give you a worse result than the passive use so in effect, this significantly helps the insighter (to coin a word).

My preference

If the NPC is trying to deceive the PCs then this is something the NPC is doing deliberately. They should be making an active Charisma (Deception) check - you can do this before play so the PCs don't see you rolling.

Casual Interaction

These are the everyday interactions where the PCs are seeking information and have no premonition that a lie would be told. Have the conversation and give the players the appropriate cues.

If the players actively pick up on the cues (i.e. "He's sweating and stammering, I suspect he is lying" not "I think he's lying") then call for a Wisdom (Perception) roll. Success means that they get insight into the truth ("He says "The dungeon is north of here." but his eyes keep glancing to the west.")

Notwithstanding, their passive Wisdom (Perception) lets them feel that he is lying if they beat his active Charisma (Deception) roll. You tell them this at the end of the interaction - no backsies if they missed your cues.

Remember, this check is against the best PC Wisdom (Perception), other PCs are helping giving advantage (2 rolls on active, +5 on passive).

The Confrontation

This is where they go in suspecting that lies will be told, they should give you a reason why they think this; you are quite within your rights to say that they have no basis for their suspicion.

Here active versus active is probably most appropriate; I may or may not allow the passive value to serve as a floor, probably not as the player has decided to "go on the attack" and has left his "passive defence" behind. However, I would allow more nuanced results. Say succeeding gives the "truth", failing by up to 5 allows identification of "lies" and failing by 10 or more gives "false truth"; you believe the lie or you believe the truth that is not actually true (He says north, the truth is west, you tell the players south).

Play out this confrontation, depending on how it goes Wisdom (Insight) vs Charisma (Deception) may be the go but it could become Charisma (Intimidation) if the PCs threaten, Charisma (Persuasion) if they cajole or even Intelligence (Investigation) if they cross-examine; this allows the players to guide the conversation towards their strengths. In any event, Charisma (Deception) would probably be the response to all of these.

Again, remember to apply advantage and disadvantage as appropriate.

Player agency

Whatever you decide, you must tell the players how it will work; you can and probably should decide with the players. Remind them that this cuts both ways - they will be trying to deceive people too!

As an aside, much the same argument goes for using Wisdom (Perception), but that's another question!