So, I have run Curse of Strahd sometimes, but every time I run it I find a new problem in my DM'ing haha.
I will try to make it answerable without specifically CoS experience and keep it spoiler free.
Strahd, a powerful Vampire, is the main villain of the adventure, and, as the book describes, he is not a villain that will only show up for the final encounter. Strahd will, certainly, appear many times during the adventure, either to probe the characters, to scare them, or generally to play with them. In the many times I have run the adventure, I never had any problem with this encounter until now.
Recently, in one of the tables I am running, for reasons I am still unsure, my players felt very frustrated with this encounter. Let me tell how the encounter went:
The party met Ireena, helped with her request, and stayed a little bit too long in the Town. Long enough for Strahd's spies to inform him about Ireena leaving her house in company of the characters (they insisted) and, well, pay a visit to the characters when they were returning from the Tavern to the Burgomaster mansion. Strahd had no intention of actually harming the characters or even getting Ireena here – it was just to, well, make an introduction and… be Strahd.
However, the party has a very beautiful Half-Elf with 16 Cha who grabbed the attention of Strahd. He proceeded to Charm her and Bite her, just as a sign of "affection" – again, no intention of harming, I was not even rolling damage here, I only made the character in question roll the Wisdom Saving Throw and told her to "role play as if she was charmed by a vampire" when she failed – the player was fine with it and joined the role play nicely.
The other two players, however, tried many things. They tried to enter an abandoned house hoping that he wouldn't enter without invitation, or "talking the charm out" – all to no avail.
From one point, I can understand their frustration: they felt powerless. On the other, that is… kinda one of the points. At this point in the adventure, they are powerless against Strahd. He presents himself as an unbeatable encounter, who is there for his own amusing.
Other than that, there were many points – most sadly missed by them due to they worrying about how unfair the encounter was – in that encounter, which I tried to convey at the best of my abilities. These will be listed in the spoiler below as they are part of the Curse of Strahd adventure, but essentially, it was a nice opportunity to give some hints about the relationship of Strahd and Ireena, as well as introduce some Vampire Features to them.
I made Strahd constantly call Ireena "my beloved", while, even the high charisma half-elf, he only would call by "my lady" or stuff like that, never using anything resembling "love", since his love is only for Tatyana.
I used it as an opportunity to present some of the features and spells from Strahd, so the players will be better prepared when they actually face him. That's the main reason I used his Charm, a Ray of Frost and the Bite action on the PC.
Speaking of the Bite action, it was also my way to introduce them to the idea that just biting a character is not enough to transform them in a Vampire or Vampire Spawn. Until then, the players (and the characters) believed being bitten by a vampire would lead to immediate transformation.
One of my players also believed Silver would scare Vampires, but there is no such trope in 5e vampires. Sadly, he did not try out that, but it was the opportunity for him to find out that it does not work.
So, overall, I think there are many reasons for the encounter to happen, both from an in-game perspective, and simply to provide some useful information to the players and characters. However, they felt powerless, that the encounter was unfair and generally frustrated, even asking me "what is the point of this encounter?" in the middle of the encounter.
So, how can I present such an encounter, which is essentially unbeatable at this point, without frustrating them? I am asking because I plan to have them meet Strahd after the events in Vallaki again, and actually have a little bit of a combat this time, depending on how they behave, but I don't want it to end up being a "again this unfair fight? There is nothing we can do at this stage please stop throwing him at us".
Some things: First, I do not want to tell them "Don't worry he won't kill you" – they should be scared. And he will kill them, depending on how they act, and how bored he becomes with them. And I also would prefer to not disclose the spoiler'd reasons for the encounter, I would prefer them to find out by themselves that those were clues. I also think, even without them, there is enough in-game motivation for the encounter to happen from Strahd's perspective, and he is a character that I want to develop as well.
PS: I should note that this was fairly soon after the Shambling Mound encounter in the Death House, where they also felt powerless because they couldn't beat the monster in combat (and two characters actually died there – the players were back in this session with new characters).
PPS: I should also mention that, while Strahd himself was unbeatable, they had a feasible goal in sight: run to the Burgomaster mansion, where Strahd would not enter uninvited – which they understood quite quickly and managed to accomplish. That is to say that they had an objective in the encounter and even successfully managed to complete it, but obviously it lacked some sense of reward for them.
System agnostic answers get extra internet points.
How can I best invoke the trope of a foe who radically outclasses the heroes in Fate without compromising player agency? – but this is specific for FATE and the answers are very system-specific.
How can I present an "unsolvable right now" puzzle without frustrating my players? – Same thing, but instead of an enemy NPC, it's a puzzle.
How do I add a recurring fantasy villain without frustrating the players?
Not every encounter has to be "winnable"
There will always be encounters in games that are not beatable in the traditional meaning. Maybe characters encounter a god, or an enemy AI that they cannot fight against, an enormous alien swarm they cannot hope to beat or just a huge rock they cannot move with their current equipment. Maybe they encounter something that is important later in the adventure and they cannot beat it for meta reasons.
And that's okay. Many books and movies have those scenes where the heroes encounter odds or powers that aren't beatable.
But do not try to fake agency
A game is all about making decisions and have those decisions influence the outcome. And sometimes you cannot make meaningful decisions. That is okay. But do not fake it. Do not let players make decisions that you ignore on purpose. That is super frustrating. If the range of meaningful decisions is small, that is okay, just tell the players.
For example: if the players are to be captured by the city guards and there is no way the players can beat the odds, you can ask if they want to fight or surrender, but if they pick fight, don't have them roll dice for 90 minutes when the outcome is already fixed. Just make it a cutscene, tell them they fought heroically but there were just too many of the guards and they where overwhelmed.
For example: if the big rock blocks the road for meta reasons because they are not supposed to move the cart up the road at this point in the adventure, don't let them try all the different methods for a real-life hour. Tell them that their characters tried their best for half a day and then came to the conclusion that the rock is not currently movable. Maybe drop a hint to try again with better tools or more people of after the rain or whatever the adventure uses to justify that it can be moved later. But as frustrating as the blocking rock is for the characters, there is no reason to spent 60 minutes of real time frustrating the players.
So for your example: if they cannot beat the vampire, they can try a few tricks. Like the house or silver. But there is no reason to play that out. Rolling initiative to see if they get to the door, rolling dexterity to grab the silver, if that's meaningless and the dice rolls don't matter anyway, skip them. Take the idea, tell them what happens. Yes, on the surface they lost some of their agency if they cannot control their characters turn-by-turn, but in the end, if they never had the agency to begin with, this scene should be as short as possible so they are not stuck in this no-agency mode, but can move on to their old decision making selves as soon as possible.
If the outcome of the encounter is fixed, tell them. Get it over with. Make it a cutscene like in the movies, where the protagonist is surrounded and the next scene they are in chains with deep cuts and bruises. No need to bore people with the full extended directors cut, move on to the interesting parts of the story, where the protagonist can act meaningful.