The PHB, at p. 175, presents passive skill checks as a means to represent two different kinds of in-game scenarios:
Such a check can represent the average result for a task done repeatedly, such as searching for secret doors over and over again, or can be used when the DM wants to secretly determine whether the characters succeed at something without rolling dice, such as noticing a hidden monster….
(Emphasis mine.) Notably, the entire section on passive checks is general and agnostic as to any particular skill. It does not say these skills can be used passively, but not those. The implication, it would seem, is that any skill can potentially be used passively in the right circumstance.
For the detection-type skills — Perception, Insight, and Investigation — passive checks are so common and well-documented that I don't feel a need to cite sources for support. Other skills don't lend themselves as readily to passive checks, but it's at least conceivable to use them that way. For example, one could imagine using Passive Medicine to represent a character in a field hospital repeatedly diagnosing and treating injuries among troops at war, or using Passive History to represent a character's spontaneous recall of a particular fact without the player first asking "Does my character know any relevant history here?"
Is there any use-case for Passive Stealth? Or do the rules on hiding and detection (see PHB p. 177) essentially require that all Stealth checks be active?