Cthulhu d20 has a magic system in which spells aren't tied to classes nor to levels. Anyone can learn and use a spell, if they're willing to pay the price, which usually comes in the form of a temporary or permanent ability loss (drain) plus a loss of Sanity points, besides whatever in-game consequences the DM sees fit.
This system, in my opinion, would work extremely well fluff-wise in a dark fantasy D&D 5e world as well — such as Ravenloft / Curse of Strahd, for example — either as an add-on besides the standard magic system or as a full replacement of that system. Let's focus on the previous, though:
What are the most obvious pitfalls of introducing the magic system of Cthulhu d20 into D&D 5e as an add-on, making cost-based spells available to practically anyone who has the chance to learn them in-game; and what are the best ways to avoid those pitfalls when integrating the two systems?
Would such a merger break the game? What steps and rule tweaks would ensure the smoothest possible extension of the core D&D 5e system? (Sanity would have to be used, obviously, as per the 5e DMG.)
Note, please, that there's another similar question out there on the site (which was also asked by me :)) about 3.x and the same Cthulhu d20 system. However, what I'm asking here is not the same (though even if I did that, the answer would have to be different, considering the difference between 3.x and 5e.) What I'm asking here is how to best port the complete spells section of CoC d20 into 5e as an addendum to work alongside the official magic rules, which would not be affected by the levelless cost system. Please, detail the major pitfalls your proposed solution helps avoid. (Nonetheless, it would be interesting to see a wizard who used both the leveled, costless spells and the levelless, costly ones. :))
Best Answer
My background: I've run some adventures using insanity themes. I once had a character who used insanity-magic and was about to go insane, when his player decided the campaign was too dark for him and stopped showing up at my table. I once had a character who used insanity-magic and went insane on purpose because he wanted to kill the party and destroy the world. Here's what I would do in future if I wanted to use insanity themes again.
I think you should start by looking at the Same Page Tool.
D&D 5e is (usually) about a group of heroes who work together to save the world. Call of Cthulhu is (usually) about a group of normal people who stumble into eldritch horrors, go insane, turn on each other, and die horribly. These are very different games. Which game do you want to be playing?
Suppose that a player deliberately drives their character insane, then uses that insanity as justification for killing the party and unleashing eldritch horrors on the world. Is that: (1) totally appropriate and in-genre, or (2) annoying because that player ruined everyone else's fun?
Suppose that the party as a whole embraces the use of insanity magic, becoming more and more dysfunctional and evil until eventually the group shatters and everyone dies. Is that what you wanted from the adventure? Or do you want something more along the lines of "people go a little evil for flavor, but not so much that it makes them lose"?
Decide in advance what sorts of actions are "in-genre" for your game, then make sure you and your players are in agreement about that. Once you've all decided what you want your game to look like, you'll have a much easier time getting it to look like that.
In terms of mechanics: the problem you're going to have is that D&D doesn't really do "permanent damage". Most reduction to ability scores goes away after a short or long rest. Even if you house-rule that certain spells cause permanent damage, the greater restoration spell can cure that stuff.
A lot depends on what your players will tolerate. If they're on board with it, you could improvise some sort of Sanity Points system for representing permanent sanity damage. Most players don't like having their character's impending insanity hanging over their heads, though. This could cause them to lose attachment to their character, or it could cause them to lose interest in your game.
I think a more awesome approach would be to impose narrative costs: when you use the forbidden magics, you take temporary Wisdom damage and you make a Wisdom check. If you fail, you release evil into the world. The evil could take the form of another monster you have to fight, or it could be that something you relied on has become foul and twisted and corrupted, or maybe it ignores your character and makes life miserable for the nearby villagers (depending, perhaps, on how badly you roll). I've done similar things in Apocalypse-World-based systems and it's been a lot of fun.