I have tried this in two ways in the past. I think of the two, only one will be of use for your objective.
PC Villain in the Group
Create the villain with its player and discuss what their villainous goal actually is. Ensure the player can and will commit to being a villain. Their goal should require the villain to need to be close to or involved with the players to achieve. You will probably want the nature of your villain to be a big revelation at some stage of the campaign, so they need strong hooks to tie them to the heroes, and to tie them to what the heroes are trying to do.
Of course, the heroes will want to achieve a positive outcome in the scenario, while their supposed "friend and fellow hero" is actually trying to ensure a negative outcome. This ensures challenge for each player without necessitating that the players be targetting each other, limiting the sense of mutual dependence, or requiring that the relationship be entirely based on lies.
In truth, the variance between the heroes and the villain might only a slight difference in ideology, the burden of a dark family secret, or the interplay of noble but misguided intentions.
Example:
*Over time the group learns that an ancient prophecy claims a soul-collecting terror from ages past will rise again to plague a small village on the ancestral lands of one of the characters.
As they uncover more and more about the prophecy, they realize that the time will soon arrive. They must act now. They become desperate to discover a way to prevent or guard against the soul-collector.
Throughout this period, the villain is aware that it was their many times great grandmother which unleashed this foul curse on the village in the first place to spare her own children from the ravages of the soul-collector. If the villain chooses to delve deeper into the lore of the family, they will discover much more about the monster than the heroes, including that it was bound by the family to both stop greater depredations and to extract special favors from it. They can, if they have the will, learn how to bind it, and become tempted by what the being will offer for even a little bit of freedom. They will also learn that if the monster is balked it will come for them, their siblings, and other members of that generation within the family and nothing will be able to stop it. Worse, if they reveal the family secret they will be disowned, the monster will take back all the gifts it provided the family, and it will be set loose to kill indiscriminately. To ice the cake, if the villain chooses to master the rites involved, they can gain direct control of the monster and earn their own, very special favors from it.
The heroes will of course be committed to stopping the creature, and their flawed understanding of the situation will enable the villain, if they stick close to the heroes, to monitor progress and to try to redirect it when necessary to prevent success.*
Requirements
This style of scenario best pits character goal versus character goal, but not character versus character explicitly. It does not need to require that the villain receive lots of useful extra knowledge, nor that they go off for extended periods to plot or take care of things off-screen. Moreover, it allows all the players to play toward a goal of uncertain resolution, rather than forcing one of them to be a glorified NPC just there to trick the players. A great scenario will have lots of conflicting emotions on both sides of the group as the course of events plays out.
Key Points
- Everyone plays in the scenario, no ringers or PCNPCs
- Each has a related goal that they may or may not achieve
- once play starts there is no meta-fiddling with the villain's
knowledge or activities
- the heroes may remain focused on ending the threat little knowing
they have a viper in their midst
- the villain has the burden of keeping their secret agenda of foiling
the goal of the heroes without getting caught
At the end, there is a chance for a dramatic conclusion when the duplicity of the villain ultimately comes to light. If the bonds between all the PCs are tight, the heroes may even be swayed to the villain's side...
Start (experienced RPG.SE users will guess what I'm going to say next) by talking to them.
You obviously have a premise in mind: The player characters comes across something weird, and you expect them to investigate.
Your players are obviously on-board with the idea of a Cthulu game, and get that investigating is a thing they're expected to do. They also have a healthy sense of self-preservation, and so are trying to stay safe; There's nothing inherently wrong with that.
Both your premise and the way your players are acting are reasonable; You need to work out what your players aren't reacting to the hooks you're throwing them in a way you expect, and the easiest and best way to do that is to ask them why. Maybe they haven't got enough in-character information to be motivated to investigate; Maybe they don't have enough information to conclude that the Thing in the Woods isn't just a bear; Maybe they all made "reluctant investigator"-type characters and they want to be forced into investigating against their better judgement, or they have specific motivations for potentially investigating things that they hoped you'd pick up on but which you haven't yet noticed.
Whatever the explanation is, the only way to find out is to ask your players reacted the way they did to the plot hook you thought would work. They'll probably give you a completely reasonable answer.
Then, follow up by asking what kind of thing would have inspired them to investigate rather than flee. The answers they give you will grant you insight into both their playstyles and their characters' motivations, which will allow you to design story hooks that suit your players better.
Best Answer
Powerful drama requires powerful motivations. When everyone at the table agrees that they want a Horror game, they must craft their characters around these motivations. If they don't buy in, then you get the kind of power-fantasy where the heroes do the quite sensible thing of feeding Cthulhu a couple cases of dynamite and legging it. That isn't horror, that's fantasy (and it's totally cool if everyone wants that other kind of story.)
The mark of a sane person is to be able to see something that's likely to get them killed, incarcerated, driven insane, or otherwise disrupt their status quo and go "Nope! I've got taxes to do!." And then goes and does them.
Sane people, unfortunately, are boring. (This bug has been filed and is currently in the triage queue for Human 1.2. It's a very long queue.)
Take a look at True Detective, the protagonists are horrible, driven, people who cannot let things lie, who cannot take the easy route, and who, by existing, create drama. People like these are the fodder for horror stories, because at some level, all horror stories resonate with "don't be that guy."
Every single character must have an overriding need that cannot be simply fulfilled by going home and doing the (equivalent of) taxes. They may be fleeing something, trying to right an injustice, rightly (or wrongly) wanting something, or someone that they're willing to go through hell for. They must be able to point to their motivation as the answer to "What do you value more than life itself?" The purpose of the dramatic horror is to put that goal in jeopardy. "If you turn away, you'll never get the story." "If you go back to being sane... you'll always know that you could have been somebody."
The trick is getting player buyin. Horror stories require very specific protagonists, as they cannot be power-fantasies. The first requirement for a horror character to be effective in themselves is for the thing, duty, ideal, or person, that they're willing to make poor personal judgements for. Start character creation with that, and then figure out what kind of person is willing to make those poor life choices. Then watch to see how life puts them through the blender. If they ever achieve their goal, their arc is done. Either they must have some new, even more impossible dream, or they've won, and they can go retire and the player should make a new character.
Take a look at the types of horror protagonist:
Horror is a tragedy rife with various anxieties.
Tragedy:
Horror takes this and removes all of the "fate" bits. It presents us with a horrible uncaring universe, rife with the unknowable, and tells the story of the futility of challenge. The basic contract of the character with the story is "find motivations for doing this stupid stuff. In return, you may realise your own stupidity in time to regret it." **It's a story of hubris, the compulsions beyond the pale, set against a backdrop that simply doesn't care or even potentially comprehend.
Here are some further resources, so this doesn't turn into a conference paper:
Rationalists can experience horror, for they are driven by the need for understanding. Take Harry Potter James Evans Verres, a rationalist with an alarming amount of hubris. If you replace his antagonist with the literally unknowable, and force him to choose between the lies of the real, and the truth of the uncaring void, without allowing rationalism to win, you have both a really (even more) depressing story, and horror.