During a discussion I had with @jgn based on the question discussing rolling twice on an investigation check, I realized we were operating with entirely different ideas of how dice rolls actually function inside of the narrative of the game.
For example:
A fighter searches a room he's never been in before. If he rolls a 15, he'll find the hidden switch that opens the secret laboratory of the mad doctor Fred. He rolls and it's a 3, he does not find the secret switch.
My point of view is that the fighter did his best to find something special about the room, didn't find anything, and has no reason to roll again, and he'd get a new try, or find it automatically, if somebody later informed him about the hidden switch in the room.
@jgn's point of view is that the fighter is aware of the fact that he did a poor job searching the room and can keep trying until he is confident that he did a good job. In essence, the fighter "knows" the dice roll and will stop trying when he rolls high enough.
To me, the later approach seems like it'd be better served with a taking 10 (passive) Investigation check and refluffing "Oh I rolled a 1, I did a poor job searching, I'll just roll again!" is essentially fishing for advantage based on meta game information.
In earlier editions, "try until you are 100% certain you gave it your best" was done with taking 20, but that no longer exists in 5e.
So which, exactly, is the official D&D 5e stance?
Notes based on comments:
-
There is, obviously, a risk of failure. The fighter can fail to find the room, so he should roll.
-
It is not critical that the fighter finds the room. This is not a case of "the fighter has to find the room".
Do characters know they did a poor job because of low dice rolls if a failed roll gives them no new information and their failure isn't visible?
Best Answer
It Depends
This is a matter of Player/Character separation of knowledge and what makes sense. The extent of RAW we have is that you are allowed to try again, but...
The kicker is the section I bolded. What does "In Some Cases" mean? We don't get a specific definition on that.
So you have to fall back on what makes logical sense. Think about if you, yourself, were trying to achieve the thing that your character is trying to achieve. Would you have a reason to try again? Is there some indication that you should try again?
We can assume that a dice result represents the results of your effort, not how hard you tried...so, at the least, characters know they were actually trying their best. This ultimately boils down to this: If you couldn't see the dice result (say, the DM rolled for you), only the actual output of what happened, would you try again?
Let's take a few examples...
Searching a Room
Suppose, IRL, you walk into a room and decide you want to look and see if there is a safe in the room. You search the room top to bottom and don't find the safe. There may, in fact, actually be a safe in the room--you just didn't think to tug the light-switch out from the wall and hit the button on the side, which would have popped open the hidden compartment where the safe is.
But here's the important thing... you were not certain there was a safe in this room. You searched the room, did your best...but didn't "roll" high enough to find the hidden controls to reveal the safe. You have no reason to assume you 'failed' to find the safe, and every reason to assume there's simply no safe there.
On the other hand...suppose you lost your keys in your apartment. Well, you know they're in there somewhere, so even if your first 'pass' of searching the apartment fails, you're going to try again because you are aware (or at least believe) that the thing you're looking for exists and is in the place you are searching, so you know that you 'failed' in your search, and will try again.
Climbing a wall
This is pretty cut and dry, honestly. You tried to climb the wall...you failed. Do you want to try again Y/N?
Crafting
Again, cut and dry. You made a thing...how good of a job did you do? Well, you can inspect the thing you made and find out. But you may no longer have the materials to try again.
Lie Detection
This one is also pretty cut and dry. In the moment, did you believe that person or not? You have no clear in-game evidence to show you that your 'check' was incorrect.
Tracking
Again, this one depends on the situation.
Suppose you're following a trail...you cut through some rough ground and lose the trail. Well, you know there's a trail there and the thing you're following probably didn't randomly cease to exist...so you'd think to double back to the last place you saw the trail and try again to follow it.
However, if you're searching a glade for signs that a deer was there and don't find any...well, you didn't actually know whether or not there was a deer there, so you can't tell the difference between "missed the signs" and "no deer here." So, again, your character has no ability to tell between 'failure' and 'nothing here.'
A bit of extra supporting evidence...
While this is not directly related to Skill Checks, it does relate in terms of a Character's ability to figure out how something worked, we have this bit on Invalid Spell Targets from Xanathar's Guide to Everything.
So, again...as there is no clear evidence that the target was invalid...you simply perceive that the spell failed to effect them. You cannot tell the difference between "They passed their save" and "they are immune" in the same way that, without evidence, you cannot tell the difference between "I didn't roll well enough" and "There's nothing to be found."
TL;DR:
In short...think about it realistically. If you couldn't see your dice result, is there some clear reason for your character to know that they failed? If yes, they have reason to try again. If no, they cannot tell that they failed.