Is there a system for determining the difficulty of a combat encounter with a monster is in Call of Cthulhu 7e

There is no such system

It is easy to see from the monster statistics that most combat is very deadly, even against normal, human opponents with firearms. This makes sense: a human against another human, on average should be a 50/50 shot of who gets to walk away alive.

However, while there is no classification of monster deadliness now, old editions of Call of Cthuhlu grouped mythos monsters into classes, and this may be somewhat useful:

1. Class I were the least deadly monsters, things roughly in line with a human in terms of attributes and lethality: Sand-dwellers, Ghouls, Mi-Go, Deep Ones and the like. This would include gangsters, cultists, and natural animals such as wolves, and some "classical" horror monsters such as skeletons or zombies. A group of investigators might be able to win a fight against a group of such monsters with enough luck.
2. Class II were monsters where a whole investigator group might have a chance to win combat, if there only was a single monster (Hunting Horrors, Dimensional Shamblers and the like).
3. Class III is where fighting started to become nearly entirely suicidal. Hound of Tindalos, Flying Polyps, Shoggots and the like typcially were TPKs if you did not run. You might have a shot in rare cases, with explosives or cleverly set traps or good preparation.
4. Class IV: great old ones, star spawns, etc. Flee or die. Unless you happen to have a battleship to plough over the monster, a whole cavern rigged with dynamite, or something like that. In that case you might have a shot. Else you are toast.
5. Class V: The actual gods of the mythos, Nyarlathotep, Azathot, Yog-Sothot etc. Nothing helps. Despair, run, or die. Probably just die.

The newest version of the rules advises this:

It is quite possible for the investigators to beat some monsters in combat. Against others, their only chance of survival is generally to flee or hide. Deep ones, byakhees, dimensional shamblers, fire vampires, ghasts, ghouls, servants of Gla'aki, mi-go, moon-beasts, nightgaunts, rat-things, sand-dwellers, serpent people, insects from Shagghai, tcho-tcho, and of course human cultists can all be "taken on" in combat by several investigators with some hope of success. This is the reason that so many scenarios feature cultists, deep ones, mi-go, and ghouls—they make approximately human-scale adversaries.

The chance to survive a fight against those that can be defeated to begin with will depend on the size of your investigator group: the more people you have, and the better equipped and trained they are for combat, the better their chances to win a fight. This can fall apart however against supernatural mythos monsters with high Sanity cost, where failed Sanity checks can remove characters from the fight, turning them to ball up catatonically or babble incoherently.

Note, that the overall likelyhood of survival in COC often is inversily related to combat ability: combat against monsters is so deadly, that in my experience those characters with combat skills that end up trying to fight them, are those that die first. We played many campaigns where the remaining survivors were blind (great for keeping your Sanity), entirley unsuited to fight, or both.

Statistics

It is practically impossible to conclusively calculate probabilites of survival in general, as these will depend on party size, stats, armor, weapons (both of which in turn heavily depend on the era), as well as the individual type and number of monsters involved.

As a single shot from a handgun deals 1d10 damage. One such shot would not normally kill a character, but even a .45 revolver deals 1d10+2, and has a 20% chance of outright killing a character on a hit and a 70% chance to cause a major wound. A single shot of a 12-gauge shotgun at short range deals 4d6 damage, and has a 84% chance to kill a character and over 99% to deal a major wound. This level of damage then also requires a CON check to avoid unconsciousness on behalf of the victim.

So combat is going to be very swingy, depinging on who has higher DEX, gets to go first, and hits first.

Typical Investigator

If you look at a typcial investigator (in their 20s or 30s for physcial fitness), they have

SIZ 2d6+6 x 5 = 65
CON 3d6 x 5 = 52.5
STR 3d6 x 5 = 52.5
DEX 3d6 x 5 = 52.5
STR+SIZ = 117.5 (Damage Bonus 0, Build 0)
HP: SIZ + CON / 10 (Round down) = 117.5 / 10 = 11.75 = 11 hp

The default combat skills (without training) are: Handgun 20% Dodge 26%, Figthing (Brawl): 25%
Handgun: .38 revolver, 1d10

Most investigators do not walk around with rifles, as this creates unwanted attention by the police. If shooting with a handgun at point-blank range (within about 10 feet, 1/5th of DEX), the chance to hit increases from the bonus die to 36%, and there is an about 7% chance for an impale extreme success hit for another 10 damage.

Typically characters wear no armor, and unless they have an unusual occupation such as Soldier or Police Officer, they may not have a professional Firearms of Fighthing skills, and no access to automatic weapons or explosives either. (And depending on the keeper's choice, even those with skills may be limited to max 75%, see page 48 Keeper's Rulebook.)

The above character, firing a single shot per round and assuming the weapon does not jam and is shot at point blank, it will deal an expected 2.7 points of damage per round. Investigators are tyically best off using guns, as a readied gun gives +50 to Dexterity for determining initiative (although, as some monsters are entirely immune to bullets, an axe has also been proven a useful tool in our experience).

The average deep one would have 13 hp, 1 point armor, and will fight with a trident for 1d6+1d4 or spear for 1d8+1d4, 45% chance to hit. That means on average it will deal 45% * 6.5 damage per round, or about 4 damage with impales. Also here, attacks that connect are highly likely to cause a major wound and will force a CON roll or unconsciousness.

You would expect such a fight to end on average in the third round with the death of the investigator, while the deep one will still have about 9 hp (this overestimates the effect of armor, however, which has more effect against smaller "expected" damage every round than against none or full).

This would indicate, that you can probably pit 2 investigators against one deep one, and expect them to win. But again: all of this is very swingy, and you easily can end up with two dead investigators, with some unlucky rolls.

Specialized Fighter Investigator

Compare this to a modern-era military character who has combat equipment, and may have picked the profession due to his stats:

SIZ 14 x 5 = 70
INT 12 x 5 = 60
CON 17 x 5 = 85
STR 13 x 5 = 65
DEX 15 x 5 = 75
EDU 10 x 5 = 50
STR+SIZ = 135 (Damage Bonus +1d4, Build 1)
HP: SIZ + CON / 10 (Round down) = 155 / 10 = 15 hp
Professional Skill points: EDU (50) + DEX2 (150) = 200; Personal Skill points: INT2 = 120 (unused here)
Other Skills: Climb, Stealth, Survival, First Aid, Mechanical Repair (rest of skill points on these)

Firearms (Assault Rifle): 75% AK-47, 2d6+1, 1, (2) or full auto
Firearms (SMG): 60% Uzi, 1d10, 1, (2) or full auto
Fight(Brawl): 75%
Dodge: 60%
Modern Military Body Armor: 12 points

This investogator can shoot bursts of 6 bullets with his Assault Rifle, half of which will hit on a normal hit, or all on them will and half will impale on an extreme hit, dealing an expected 27 damage to the deep one, before he can hit back. And even if he could, little would get through the body armor, if he does not impale. He should be able to take down several deep ones all by himself, especially if he does not start out in melee.

Because of this wide disparity in investigators ability to deal damage, depending on era or build, if you wanted to have statisically exact calculations (which really do not make a lot of sense, as they do not generalize to diffrent monsters), you would need to provide the stats of the investigators, and the monsters you have in mind.