As DM I have always some problems to apply the magic world of D&D onto the real world, which is the base of our common understanding of how things works. Especially, since I have experienced players who just wanted to kill kings to take over their kingdom, purely acting on impulse. I stumbled upon a new situation and I want to learn from your experience:
In my opinion, the spell gaseous form is very powerful for special-ops (by this I refer to rogues and assassins in particular). It allows to fly and, used at nightfall, it should generally provide a good amount of stealthiness as long as not used in very confined and lit spaces. Also, it provides the beneficiary of this spell the ability
to pass small holes, narrow openings, and even mere cracks,
Of course, it only allows slow movement (10 ft) and provides only resistance against common damage types, but the beneficiary can take all his potentially deadly equipment on its hour-long way. There might be a drawback; if the beneficiary is not the caster, the spell cannot be ended by the beneficiary, but that should be just a thing to be considered not a real problem here.
Consider a castle (or the home of a very wealthy merchant) which is well defended by regular means and maybe even some magical backup. An assassin who wants to follow his profession will certainly find a "mere crack" which can be used to enter the castle's building and even rooms (I don't think that any (land-based) door or window is air-tight).
This spell can be used from characters against anyone, as well as from assassins sent against the hero group (see Tyranny of Dragons, 2nd part for example). It could be used between hostile rulers – in fact, this could be nothing out of the ordinary: As gaseous form is a 3rd-level spell it even should be not a problem to find spell caster hirelings.
I don't see a realistic chance to protect a whole castle from such an intruder, do you? Certainly, spells like Alarm can 'protect' sleeping rooms, but they cannot prevent an attack (but, maybe a deadly success, depending in the fanaticism of the killer).
Do you consider a guard standing next to a bed all night as realistic?
Sure, while in gaseous form the intruder cannot interact with any object, but if that would be an issue I would send a special-ops team with potions of gaseous forms: whenever needed, a team member would end this effect, the remaining team proceeds.
Best Answer
Glyphs of Warding and Symbols on doors and windows, possibly paired with Alarms would probably be the go-to defense against such intruders.
Such spells could also be placed on bed frames of a castle's most important occupants or throughout their chambers to cover the case that an assailant managed to slip through cracks in a wall or floor rather than a door's key hole or drafty window.
Glyphs of Warding used to store a spell like Dispel Magic would be perfect for ending the Gaseous intruder's spell. For higher level casters, a Forcecage stored in a glyph would be perfect, as even a gaseous creature cannot escape from an enclosed Forcecage. Because Gaseous Form requires concentration, a Sleet Storm could be another useful spell to store, since it forces a concentration save against the caster's spell DC (rather than being based on the damage), and with only 10 feet of movement and the Sleet Storm creating difficult terrain, the caster would need to make multiple saves before leaving its area. Any spell that incapacitates, such as Hold Person or Hypnotic Pattern, would also break the gaseous interloper's concentration, as well as leave them temporarily unable to move.
Symbol offers several options that incapacitate, such as Pain, Sleep, and Stunning.
Otherwise, you could just try to do enough damage to knock the intruder out by placing multiple Glyphs of Warding (with the Explosive Runes option) or Symbols (with the Death option) in various places of import throughout the building.
If you placed Alarm spells in the same location as each Glyph or Symbol, you would also be aware of where your intruders were, and if the Glyph or Symbol worked as you hoped, you could quickly get to the location to find your intruder incapacitated or at least debilitated when you arrive.