You also need to discuss expectations if you haven't already

If you just change the mechanics to better fit the style of play that a highly lethal old-school module is written for, you need to discuss the expectations of this style of play with your players too. If you don't talk about these expectations, all that your players will see is that you've changed the system and made them less powerful apparently for no reason. I would highly recommend having your players read a quick primer on old school play; if you can get past the bias inherent in the document, it's very helpful. Even if you don't have your players read the primer, you can still state the main points and present the reasoning behind you've changed the system. Making sure players know that build is less important, that death is frequent due to lack of caution, and that player skill is rewarded is going to be necessary to support the style of play you want here. Other than this...

Running an LotFP module with 5e is perfectly viable with some conversion work

As long as you convert the module's mechanics over, running Tower of the Stargazer should work fine. The main issues would be hit point inflation (which is sounds like you've covered); the Perception and Investigation skills (which it also sounds like you've removed), and Saving Throw DCs. I'm assuming (based on your question) that you feel comfortable making mechanical changes to both the system and the module, so I don't think this will be a problem for you. One point of caution I'd like to advise on, however, is length of time spent to make a character.

Avoid Lengthy Character Generation

The general expectation of a game like LoTFP is that characters die frequently, and thus, must be created frequently. You'll want to keep the time a player spends out of play due to a dead character as low as possible. A LoTFP character can be created somewhat more quickly than a 5E character, so I'd advise making some of your system changes reflect this. Particularly; I'd say that you should have a list of cantrips and 1st level spells available as handouts for your spellcasting characters so selection goes more quickly, that you either enforce rolled stats or use a standard array for scores to not spend a lot of time on point buy, and that you don't use feats and skills (which it sounds like you've already done). This is just a very quick and dirty method of reducing time it takes to create a new 5e character, but hopefully it will work for you.

Remove or minimize combat in your games

First, the low-hanging fruit. If you don't enjoy combat, then don't include combat in your design. There is nobody forcing your hand as the DM to include it. Its presence in your game is solely up to you, and since nobody likes it anyway, don't have it.

Put only thoughtful, reasonable NPCs in your games

In real life, there is a reason that we don't engage in combat with our boss to get a raise, or attempt to murder anyone we think is trying to stop us from achieving our goals. Violence creates a lose-lose situation for everyone involved. As a combatant, you risk physical harm, reputational damage, and even death.

This means your boss NPCs will be more intelligent and less likely to scorching ray someone, but rather they will be more open to negotiation and intrigue. After all, they probably got to their current position by being smart about it. Your guards (which you most likely only put there to obstruct the player's goals via combat) have extensive alarms and communication systems such that, if even one guard spots you, everyone knows there's an intruder -- which is a powerful deterrent for any possible intruder, including the party. Your mad dogs, wild animals, and random monsters will have a sense of self-preservation, and they will try to run away from the fight as soon as they feel the fight slipping outside of their favor.

This either removes combat from your games entirely, or cuts them down to such an extent that the only combats you will have are the combats that make narrative sense to have.

If you must have combat, only include it when the narrative demands it

Don't throw in fights for fights' sake, because as you established, nobody in your table likes that. Include it as the climax to a story arc, or the final battle before the completion of a subplot. This will make your combat narratively significant. There is no feeling of "get on with it already" here, because the success or failure of the party will swing the story in vastly different directions.

In a game I'm running, I presented the players with a choice. The party agreed to not attack a second party (of NPCs) or try to steal from them, even though the second party had valuable loot the first party needed, on the basis that one of the players was brothers in bond with one of the NPCs from the second party. If the players don't get the loot, though, there will be horrible consequences. But as the pressure mounted, their choice was this: does the player betray his brother, or does he keep his pact and damn his party?

They did eventually decide to betray the NPCs, which led to combat. This combat was significant: if they killed the NPCs, they could find the valuable loot they were after, but the player would have killed a brother. If they failed to kill the NPCs, then the bond of trust between the player's character and his brother will be permanently broken, plus they don't get the loot they need. It is a dilemma. Both choices are awful, but which one will they take? How can they get out of it?

This is the first combat I ran for the campaign, and it as the 6th or 7th session. I absolutely cut out all the unnecessary fighting. There is no dungeon crawl feel. What is left is an intense sense of tension and intrigue.

When you do have combat, minimize the total time in play

There are a lot of variants and techniques you can use to ensure you spend as little time in combat as possible. The bottom line is, you must streamline your combat.

• You already know this combat will be narratively significant, so prepare all the cheat sheets beforehand: rules you think will be important to reference.

• Prepare index cards for each monster, pre-roll their initiative, attacks, and damage rolls. When it's their turn, quickly describe what they do (which you'd already decided before combat began) and throw the spotlight back to the players.

Use Variant rules that will shorten combat

The DMG provides several options that effectively reduce the HP of all the combatants.

• Massive Damage (DMG 273): When a creature suffers damage in one attack equal to or greater than half its maximum hitpoints, it experiences a system shock. The effects range from not being able to use reactions, to instantly dropping to 0 HP. This means you only have to deal half a creature's HP to put it down, instead of its full HP.

• Morale (DMG 273): When creatures are surprised, reduced to half its HP for the first time in battle or has no way of harming the opposite side (when alone), or when the leader's HP drops to 0 or the group has shrunk to half its size while the opposition hasn't suffered losses; then creatures may flee if they fail a DC 10 Wisdom save. A fleeing creature is out of the battle, so for the purposes of combat, their HP has already effectively dropped to 0.

• Side Initiative (DMG 270): Instead of each PC rolling initiative, the group makes one check and performs all their turns simultaneously, allowing them to gang up on any one creature and potentially end combat very quickly.

On actually reducing the importance of HP

Introduce save-or-suck mechanics

Working with the system as designed, you can introduce a lot of save-or-suck spells to your player characters and your NPCs: forcecage, wall of force, banishment, polymorph, phantasmal force, etc. These are spells or effects where, if you succumb to them, it doesn't matter what your HP is, as it brings you out of the fight immediately.

If both sides of combat employ save-or-suck tactics, then combat is a game of rocket tag. This isn't desirable under normal game requirements, because combat becomes too short and is usually determined by who goes first rather than using smart tactics. However, I do think your table could do away with combat tactics (and combat in general), so this may be worth a try.

Reduce HP drastically in exchange for a very high AC

In designing monsters, there is a trade-off between AC and HP. Higher HP creatures can have a low AC because they can take strong hits and survive. I recommend that you adjust your encounters and reduce the HP of your monsters, but increase their AC drastically in exchange. The goal is to have a fragile NPC which can be killed in one or two hits, thereby devaluing their HP, but who is very hard to hit in the first place. This still has the effect of presenting a very hard encounter, and it forces players to adapt and strategize once they know any one hit will kill their opponents, if only those hits land.

Introduce dynamic and interesting terrain

You can get away with having high HP opponents if you also put a pool of lava that the players can push them onto, or a high pit they can fall off of. Put in a large body of water that someone can drown in (suffocating brings you down to 0 HP instantly). Essentially, design the encounter to have ways to instantly defeat the opponents by using the terrain creatively.

Take note that the opponents should also be able to use the environment to their advantage. Just as the PCs can push their enemies into lava, the PCs themselves can be pushed into the lava. This creates the sense of a high stakes arms race: I have to do it to them before they do it to me. That creates tension, and tension is immersive and interesting.

Don't make it about murder

A thing that happens when you introduce HP is, it creates this idea that you have to reduce someone's HP to 0 to "win." But in reality, you rarely win encounters by smacking someone until they agree with you. Realign your campaign so that it isn't about reducing a monster to 0 HP in combat. This removes the meaning from HP entirely, allowing you to reduce its significance to 0 without replacing it for a houseruled mechanic.

For example: the players have two minutes to reach the end of the hall, insert the MacGuffin into the place it goes, and stop the villain from completing his plans. But, standing in their way and blocking their path is a really high HP goon who cannot possibly defeat the party, but is resilient enough to take their attacks and hold the party off for the next two minutes. Reducing this creature's HP to 0 will actually be according to the villain's plans, because the party will have wasted all that time. This combat doubles as a puzzle: how do the player characters defeat this goon without bringing him to 0 HP?

Another example: the party's "opponent" turns out to be an innocent woman who is trying to protect her children. The woman has been possessed by a devil, and if left alone, she will turn into the devil's vessel and lose her mind, potentially killing a lot of people, and definitely killing her own children. However, right now, she is still herself, and she is afraid, thinking the party is out to kill her and take her kids away. The crux: they party has been tasked with killing her and bringing her kids to the government. She is a commoner with 4 HP and 10 AC: any one hit will kill her. But will the the party do it? This "combat" doubles as a moral dilemma, and it becomes a vehicle to explore and develop the player characters' personalities and beliefs.