[RPG] How to have a sentient dungeon (or race thereof) in the story

An important difference between a torch and a flashlight, which you noted, is a torch is omnidirectional. What other omnidirectional sources of light are people familiar with? Campfires. Ever sit at a campfire on a dark night and look into the woods? What can you see? That's right... squat. A torch, unlike a flashlight, is always in your eyes. It's impossible to keep your night vision. Any space it does not light up will be pitch black. A great movie to watch for ideas is The Burrowers, a kind of Lovecraft/Cowboy/Horror movie.

Same goes for continual light and most every DnD light source except maybe a shuttered lantern, and since when do PCs think to pay for those?

So keep this in mind when determining what the PCs can see. They can see 20 feet away, 20 to 40 feet is indistinct, colorless. Beyond that is pitch black. You may even want to lessen the typical radius. This means they can't see down corridors. They can't see into the next room. If the non-torch bearer pokes their head around a wall they can't see the thing about to rip their face off.

Hollywood, and ubiquitous illumination, has trained us to think that dark isn't very dark. Indoors is pitch black. Outdoors totally depends on the phase of the moon, so keep track!

Racial night vision? Blinded while the torch is lit. Doesn't matter what the rules say, just make it so. Now the PCs have a motivation for putting the torches out and leaving the poor humans blind.

Torchlight carries way farther than the torch bearer can see, and you see the torch bearer distinctly from far away. This will attract lots of "fun" things. Smarter monsters can even use this to their advantage, seeing the PCs coming from far away, probably noisily talking and clanking, they can set up an ambush. Maybe, with no warning, spears and arrows fly out of the darkness! Punish the PCs for being so visible and so blind.

Torchlight, unless you're right up close to something, doesn't let you see very well. When a PC without their own light source is examining something, make the description indistinct. Make them want to get their face in real close to have a good look. Maybe brush their hand over it. Best way to find out it's a green slime. :-)

Things can hide in the darkness, but not always monsters. Pickpockets, spies, poisonous insects... all sorts of creepy crawlies can take advantage of the PCs being A) nearly blind and B) totally lit up.

Finally, there's lots of ways to get rid of the PC's light source. In combat, moving around wildly, there's always a chance of it blowing out. If the character gets hit, maybe they drop it... into a puddle. Maybe they drop it into something flammable. Maybe they need both hands to cast a spell or wield their weapon.

You can use your players' intuition that the sound of battle should be important without squashing their imagination of the dungeon, while also serving your GMing need to keep the whole dungeon from going "on alert" and dogpiling them at the first clash of swords. You do this by making sound weird in these strange, underground halls, and then telegraphing to your players how sound works in this dungeon by describing what they hear.

There are two standard ways to hint that fight noise will not normally carry, without simply revealing the man behind the curtain and showing that it's just a game and they "shouldn't" care about this.

The dungeon deadens sound

The hush of the tomb is expected, but there's something more to it. As you proceed down the corridor, you realise you don't even hear your footsteps echoing back at you. Conversation from the tail of your marching order sounds like thin, unintelligible whispers only yards away at the head. The very air itself seems to swallow sound.

Variations on this have been mentioned in various comments: muffling tapestries, loud or white ambient noise that masks the sounds of the party's incursion (machinery, noisy rituals, rushing water or air), and the like. Deadening sounds comes in a variety of forms, but they all have in common that a brief description of how the environment interferes with the party's hearing will let them know that fight noise is not going to travel as far as they would normally fear.

The location's acoustics confound the ability to tell where sounds come from.

This takes a bit more work. The best way to show (not tell) this is to insert random noises, screams, and whispers into the game. Creating a list and randomly rolling on it every in-game chunk of time is the traditional means.

For example:

Atmospheric Dungeon Sound Events Table
Roll 1d8 every 30 in-game minutes:

1. A scream, suddenly cut off, echoes from somewhere deep in the dungeon.
2. Mutterings that sound right behind your shoulder that slowly fade away.
3. The noise of battle, seemingly coming from directly in front of the party, where they can plainly see there is nothing.
4. The sound of dripping water, as if from very far away, but it follows the party without growing louder or fainter for several minutes.
5. Growls echo down the hall (1–3: ahead, 4–5: behind, 6: both)
6. A strong but quiet winds blows, but it seems to snatch away any attempt at conversation, carrying the party's words who-knows-where.
7. No event
8. Roll twice and combine, rerolling results 7–8.

(This is just am example. Expand as necessary before, between, or even during sessions.)

By giving a dungeon unusual acoustic properties, you can leverage how your players are already intelligently thinking about sound, making them aware in a natural way that the sounds of battle can't easily be used to locate them, unless the listener is already very close (and perhaps even not then). You also reward your players' engagement with the world by responding with in-world details about something they've shown interest in, making their investment deeper rather that fighting against it. As a bonus, it gives your dungeon more character and makes it a more unnerving and unnatural environment.