There are no set rules.

But it is discussed at length under Chapter 8: Running the Game in the DMG. There is a section on Dice Rolling, where it talks about some of the things to consider before starting the game. Best bit of advice in there, IMO?

Establish expectations about rolling dice.

It all comes down to your relationship with the players, and how you like to run the game. Just ask them! Be sure to tell them what you prefer too! Some of them may like all rolls to be done up front. But you should also balance that with speed of the game, and keeping an air of mystery. Also, making every roll in the center of the table where everyone can see can get to be a pain.

Personally, I do a hybrid system:

• Default is to roll close to me. That is where my notes and Monster Manuals are. If that is behind a screen or concealed then so be it. But I am rolling for expedience, because in general the die roll should not get the constant focus of the game.
• If it is for a roll where the results would give the players metagame information that might crucially change the course of the game, I will probably conceal the roll.
  • example. If the guard figures out you are lying, you could lose your patronage with the Queen. The bard spins his yarn. (I have them make a roll). After eyeing you, he gives you an inscrutable look with a half smile, then he waves you on (I make a concealed roll. They will find out the consequences later when they return to the Queen.)
• If there is a lot riding on a particular roll, I will briefly summarize what the outcome of the roll means for the players then roll in the center of the table.
  • example. The barbarian has held off the giant for three rounds while you all have cleaned up its minions, but he finally fell. And now, the wizard is casting his last spell in the hopes that the giant won't come and make paste out of him. If Freja the Frost Giant makes this save, the wizard is likely toast. She needs to roll a 15 or better. (Makes a slow dramatic roll in the center of the table where everyone can see)

A Note on Passive Skills

Passive skills are akin to "rolling behind the screen". They are pre-calculated to provide an extra layer of concealment to avoid signaling to the players. Players will notice when the DM starts rolling dice. Especially after they said they take an action. Relying on a passive skill number allows the DM to assess a contested skill roll with no dice rolling at all.

There is a great Dragon Talk on Stealth that also talks about how to use Passive Skills. Give it a listen. It helped me grok how 5e handles Passive skills.

Also check out this great answer on Passive Skills and how to use them.

Using "Result-less" Concealed Dice Rolls To Motivate Players

Since players have a Pavlovian response to the DM rolling concealed dice, one very fun technique is to roll concealed dice to get the game back on track. Inevitably, every game stalls. It's just how it goes. I have found that sometimes, when the players are stuck discussing the best way to open a door, if you start rolling dice behind the DMs screen, they tend to get moving.

Fudging Rolls

Rolling everything concealed does afford you some more control over the flow of the game. If things start going south for the PCs you can turn hits into misses, lower damage, miss saving throws, etc. Conversely, if they are sailing through, you can turn misses into hits, etc.

Although this is possible, just be assured that there are other ways to control the flow of combat to make things easier or harder on the PCs. Like raising and lowering hit points mid-battle, changing monster tactics, decreasing monsters in an encounter, and others. So don't use this as your reason for concealing all rolls.

Overall, I would avoid any fudging. It can sap the excitement from the game, or turn the game into a slog. For me, there is one exception to this rule. When designing encounters, I have at times overestimated or underestimated a party's capabilities. If its not by too much, then that's OK. But if I grind them into the dirt, or they cake walk through the dungeon, I will probably fudge. And if it is a really bad PC rout, sometimes you have to fudge to-hit and damage rolls as immediate triage.

Roll Enhancing Abilities vs. Hidden Rolls.

Sometimes a player has an ability that allows them to modify rolls (Bardic Inspiration, Lucky, etc.) This is a limited resource, and players will want to use it judiciously. But, in the case of a hidden roll the DM is taking some of that decision making process out of their hands. IMO, the DM should honor player abilities, and give the player the benefit of the doubt. It lessens the fun to cripple a player ability because of how you choose to make your die rolls.

With that said, when a player has that ability, I typically give the players some indication of the result of the check, before I announce the results. I try to do this in game (as much a possible). Note: I also try to do this before initiative rolls - it allows the players to decide if they want to go to combat or try a different route. YMMV, but I find it to be much more fun as it gives the players a chance to surprise me. Example:

• As the player rounds the corner, they see up ahead four bugbear guards about to turn into the same alley (they are not actively searching, and have a passive perception of 15 - not a hidden roll, but it could be). The player hides and rolls a 13. The bugbears are moving slowly but purposefully through the alley, and three of them pass by but the last one in line stops just short of your spot and starts sniffing like it has caught a scent. At this point I would pause a little, and imitate the bugbear sniffing the air, turning its head, and squinting at shadows. This should give the player or their party members the chance to use their ability, or heck do anything to better their chances.

IMO, At the very least the DM should give the player some idea how difficult the task is, if not outright give them the DC, if it is not opposed. If it is opposed, then give some indication of the antagonist's ability. Wolves are known for their incredible tracking skills and ability to find hidden things or You have only ever heard rumors about the Shadowmaster, so researching his father's name is going to be a very difficult task. You might want to use your class abilities or cast spells.

I typically do this whenever an opposed or hidden roll is close, say within 3 points - be it a success or failure for the player. It just adds to the tension and makes the game more fun for everyone. Somethings I have not considered, but probably need to:

• Is is fair to get the players to spend resources by ratcheting up the tension even though I know they don't need to? Maybe? After all if I only do this when the player is failing but close, then I am sending a message. OTOH, it is a hidden roll so the players will not know that I only do this for failures. Maybe then it is OK. On the third hand, it sure is fun to slow down the action when the story reaches an inflection point.
• Can the player use skills to decision to spend a resource? I think yes, but I would make it a passive check (active checks take time). Maybe a passive insight or investigation (or other appropriate skill as needed) DC 15(?), modified by environmental conditions(?). Not sure, I would have to think on this.
• Should I increase the range where I indicate a potential failure? If so should it be out to the limits of what a bard could potentially change? Can I role play how close the roll is to the players satisfaction? Do I merely say, for example it is about to turn it's head to look right at you, do you do anything? Not sure here, other than to say if you are indicating close rolls when they are successes, don't expand that range upwards.

This comes with a caveat: if they have no way of knowing that there would be a hidden roll, then they must make the decision to spend their ability with little to no DM input. For example, if someone decides to stealth across a field, and they have no way of knowing that an owlbear nesting in the treeline, I would probably not indicate they might be seen. The only info they get to make a decision is their die roll.

So no firm answers, but in general, defer to they players abilities and them spending resources, and give them a fair shake. Follow the rule of improv, and typically answer "yes, and...."

Rolling For Players

One suggestion in the DMG is:

You might choose to make a roll for a player because you don't want the player to know how good the check total is.

I have to say that I have never tried that method. Most of the people I play with do not want others to roll for them. That is super old school though. Your players will probably be different. When the PCs are in such a situation I would then fall back to a passive skill.

In the end, just roll with it, and you'll get the hang of it. Just listen to your players and find a system that works for both them and you.