RAW the Rogue does receive the benefits of Expertise to his passive.
A passive check is a special kind of ability check that
doesn’t involve any die rolls. Such a check can represent
the average result for a task done repeatedly, such as
searching for secret doors over and over again, or can
be used when the DM wants to secretly determine
whether the characters succeed at something without
rolling dice, such as noticing a hidden monster.
Here’s how to determine a character’s total for a
passive check: 10 + all modifiers that normally apply to the check. - Player's Handbook P.174
The player has invested in making this character good at this, this is not a bad thing.
Rogues in general are supposed to be good at this type of thing, hence the proficiency bonus class feature and the player specifically spent a feat on making his passive perception better. The player could've spent this on another feat to specialize in another area (or gain a combat ability). This is a good thing, while it may make it difficult for the Rogue to be surprised by a trap or an ambush you should in no way try to outmaneuver this.
Don't worry, there are downsides to the Rogue for this.
To reliably spot traps and ambushes the rogue will need to be at the head of the marching order so that he has clear sight lines. Rogue's are not particularly hardy and the party as a whole is inviting more risk for this reward. Likewise while he is extremely excellent at spotting physical dangers, the Rogue will not be able to detect magic wards and other dangerous enchantments and may equally blunder into them.
Ways to handle his detection of a trap or an ambush
You are correct in assuming that only that PC has seen the trap/enemy. Unless he has some mode of telepathic communication he will need to speak out and announce the threat to everyone. The best way to handle this sort of thing is to pass notes or send text messages to the player(s) able to see/detect the issue and leave it to them to react and tell someone else. Intelligent enemies will see/hear the rogue warning the party and the combat should start immediately (no surprise round though).
The PC does not become omniscient of the trap upon detecting it.
The PHB itself is very, very vague on what information is received when a PC detects a trap. However, the DMG does have a nice little section about traps, their detection, and disarming them:
If the adventurers detect a trap before
triggering it, they might be able to disarm it, either
permanently or long enough to move past it. You might
call for an Intelligence (Investigation) check for a
character to deduce what needs to be done, followed
by a Dexterity check using thieves' tools to perform the
necessary sabotage...
...In most cases, a trap's description is clear enough
that you can adjudicate whether a character's actions
locate or foil the trap. As with many situations, you
shouldn't allow die rolling to override clever play and
good planning...
...Foiling traps can be a little more complicated.
Consider a trapped treasure chest. If the chest is opened
without first pulling on the two handles set in its sides, a
mechanism inside fires a hail of poison needles toward
anyone in front of it. After inspecting the chest and
making a few checks, the characters are still unsure
if it's trapped. Rather than simply open the chest, they
prop a shield in front of it and push the chest open at
a distance with an iron rod. In this case, the trap still
triggers, but the hail of needles fires harmlessly into
the shield. - Dungeon Master's Guide p. 121
Essentially though whether trap disarming is simply a dex check, a series of checks, and/or involves serious RP is up to you as a DM. I would encourage the open-ended approach the book suggests as it adds complexity and makes trap checking and disarming a more engaging process. Sitdown with the party OOC and discuss what the table as a whole thinks should happen for trap checks and move forward based on that consensus.
Short answer
Yes, the Observant feat helps a character only when they are not actively looking. It is possible to make sense of this in terms of how active Perception is framed by the rules, and with reference to real-world ideas about the conscious and unconscious mind. The +5 bonus only applies to Perception/Investigation skills when used passively, but it is still possible to exceed this score ("notice more") with active Perception/Investigation by rolling 16 or more.
Long answer
As the introduction to feats in the PHB says (my emphasis):
A feat represents a talent or an area of expertise that gives a character special capabilities. (PHB 165)
The talent/special capability which the Observant feat gives is that you are:
Quick to notice details of your environment (PHB 168)
This is different to an active Perception check, where you mostly have to specify exactly where you are looking to have any chance to find anything.
In most cases, you need to describe where you are looking in order for the DM to determine your chance of success. For example, a key is hidden beneath a set of folded clothes in the top drawer of a bureau. If you tell the DM that you pace around the room, looking at the walls and furniture for clues, you have no chance of finding the key, regardless of your Wisdom (Perception) check result. You would have to specify that you were opening the drawers or searching the bureau in order to have any chance of success. (PHB 178)
I can think of two reasons why it makes sense that the Observant feat only benefits passive Perception. The first is based on the wording of the rules, and the second on my basic understanding of psychology.
- Observant will only work with things which are potentially noticeable in the immediate 'environment', ie not more narrowly specified locations (this is the purview of active Perception)
- You could understand passive Perception as being what the unconscious mind sees, and active Perception as being what can be uncovered by the conscious mind.
You can take or leave the last point, as not everyone subscribes to this psychological model.
I've argued here from the point of view of Perception, but the same ideas can be applied to active and passive Investigation, which the Observant feat also covers.
In any case, this does not mean that your character's passive Perception/Investigation scores are always "better than the active ones", as the die roll could be 16-20, which would take you above your passive score, but the bonus only applies to passive checks for the reasons given. Of course, you could also roll lower, but even without the Observant feat, it is possible to roll better, worse or the same as your passive Perception/Investigation, and the feat just raises the bar of your passive scores.
Best Answer
The idea is interesting, but it affects too much other things in terms of the rules. Both Passive checks and Perception skill are core game mechanics, so the game balance will be shaken.
INT-based characters will have abnormal high passive perception checks
At the gaming table, any player should have a chance to shine. Wisdom-based perception proficient Ranger shines, because INT-based low-wisdom Wizards are supposed to have low Perception checks. With the house rule though, any character will have 10 + WIS Mod + INT Mod instead of the normal 10 + WIS mod + Proficiency bonus. That makes character virtually equal in terms of the Perception skill.
Perception proficiency becomes useless
Being proficient in Perception checks should affect passive Perception checks, but it no longer does. This is important for high-level characters, which are proficient in Perception.
Other passive checks seem out of place
There is some inconsistency in changing rules for the Perception skill only. Passive checks aren't tied to the Perception skill, but can be made for any ability check:
That is consistent with the rest of the rules - for instance, the Observant feat affects both passive Perception (WIS) and Investigation (INT) skill checks.
Combat encounters become non-lethal
Challenging, deadly combat makes your D&D games exciting. However, if your players can just pay for death saving throws, being dropped to 0 hp no more means risk of death. For a horror campaign that becomes crucial, since players won't face actual dangers anymore. Constructing house rules can be fun, but dangerous things should remain dangerous.
It will be hard to prevent meta-gaming
When you describe different things to different players at the same table, it is very hard for players to disengage themselves from that description, especially if they are beginners.
Alternatives
Instead of making a custom rule for passive Perception checks, consider introducing Wisdom ability loss. That gives you the same effects (madness at zero, bad perception results), but doesn't change mechanics so much. Also, the Dungeon Master's Guide has rules for Madness (page 258), that use similar cumulative Sanity score. How NathanS said in the comments below, "Perception would be to see if they notice it at all, but their Sanity score would modify how they interpret it".