The perception rules give sample DCs. It is reasonable to equate "hearing someone speak in a strong voice" to "hear the details of a conversation" at DC 0. The DC increases by 1 for every 10 feet away, and a further +2 for the festivities (unfavorable conditions).
So if you try to dominate a guard who is patrolling 100 feet away from the nearest other guard, that nearest guard will need to succeed on a DC 12 check (+10 for 100 feet, +2 for background noise) to hear that something is going on. He would probably need to make an Intelligence check to realize that the noise was a spell being cast. If he doesn't know it was a spell, you could be long gone, and your new puppet can make up some reason like "Oh, I just stubbed my toe and cursed." Of course, the other guard would also be entitled to a Sense Motive check to determine traces of enchantment...
Note: By pure RAW, you do not need to actually speak in a strong voice, merely be able to, but you may have difficulty convincing your DM that it works this way.
To sum up the mechanics of counterspell:
If a creature with counterspell available can perceive any spell within range being cast, they can attempt to counter it. They don't need to know what spell it is, or even have it on their class list to make the counter. Based on this, and what I gather from your context, the short answer is: yes the NPC could have cast counterspell.
Let's tackle the issues one at a time with this knowledge:
Issue 1:
The NPC doesn't have to know that a silence spell is being cast. All they need to know is that a spell is being cast at all. Counterspell counters any spell after all. How would they know that a spell is being cast? Because silence has verbal and somatic components. So, as long as the NPC can see or hear the caster casting something, the NPC has the option to counterspell it.
Issue 2:
Yes. It's been established in the Sage Advice Compendium that (emphasis mine):
If a spell that’s altered by Subtle Spell has no material component, then it’s impossible for anyone to perceive the spell being cast. So, since you can’t see the casting, counterspell is of no use.
So, if the NPC can't perceive the spell being cast, then they can't counterspell it. In most circumstances (casting during combat or from a different room) it's up to the DM to determine if the NPC can perceive the casting. But if the DM rules (or is convinced) that the NPC can't perceive the casting, then the NPC can't counterspell. In this case, I assume that the NPC could see you casting, and you were within range, so counterspell would have been a legal reaction.
Issue 3:
It could, though it doesn't have to. There are some optional rules that allow creatures to identify a spell as it's being cast, but those are... well... optional. Unfortunately, the DM just has to make a call on what this particular NPC would do, and stick to it. There is no rule pulling the decision one way or another. It's reasonable for the DM to rule that the NPC would take the risk and not counterspell whatever you decide to cast. But, it's just as reasonable for them to rule that the NPC would assume that any spell you cast is a threat that can't be allowed to pass, and so would counterspell it regardless of what it turned out to be.
Issue 4:
Technically, a creature doesn't automatically know anything about the effect of whatever spell a different creature is in the process of casting. But the rules are, again, silent on this matter beyond that. So, if the group decided that minimizing metagaming is something that they want to strive for, neither they, nor the DM should know what spell is cast before deciding to counterspell. But that is a conversation that you all will need to have as a group. Some groups are more lenient where everyone knows what spells are being flung around. And still others let the players know what the enemies cast, but not the other way around. You all have to decide for yourselves what would be the most fun.
Best Answer
RAW
Most of this is something that will have to be determined by your group / DM:
The rules don't clearly state whether or not the suggestion itself is the verbal component for Suggestion.
The rules do state that someone who makes a mental save is unaware of the spell unless the spell has a "perceptible effect" (PHB, page 204).
The rules do state that you need a material component for the spell, meaning you either need the component itself, a spell component pouch, an arcane focus, or a holy symbol.
But the rules don't really state how obvious it is when someone "uses" an arcane focus or holy symbol. Only that you must have a hand free to use a material component.
RAI
A few Suggestions
Here are what I see as the best practices for this scenario.
If you want to take an action, but don't want players to know you're taking an action, you should make a stealth check.
See: How loud/obvious is a wizard casting a spell?.
This stealth check covers stilted arcane language, somatic gestures, glowing arcane foci, etc.
There would be a variety of skills to detect this
Insight — "Huh, he just said these weren't the droids they were looking for, and they repeated it back in a mono-tone? Weird."
Arcana — "That stress pattern. The eye contact. His hand in his component pouch. That's a Suggestion spell!"
Perception — "Why is this guy keeping that glowing green crystal hidden while he talks?"
Players should pick one, whichever is most meaningful to their character.
The victim of Suggestion is unaware of the spell unless given cause to question it.
Points 1 and 2 above apply to external observers. The spell wouldn't be much good if the victim could detect it.
I would not typically give the victim a chance to notice the spell, unless they made their save.
Q&A
It doesn't have to be stealth. You could make a good case for deception, or sleight of hand instead. I picked stealth as a "catch all," and a skill that many casters would benefit from anyway.
It isn't clear whether or not you can whisper your suggestion. Or if you need to speak it loudly. Or like Obi-Wan Kenobi in Star Wars. Or how obvious using a material component is (does it glow? does it need to be manipulated in the hands? etc.?).
If you like, you can define all of these aspects of the spell. If Suggestion comes up a lot, this might add quite a bit to your campaign.
But most of the time, I find it's just easier to let the dice decide. Rather than having a prolonged debate about the exact motions required to cast suggestion, or where everyone is standing, or how easy it would have been to notice a Jedi mind trick in the wild, it's more expedient to make a skill check.
None of this is intended to be a way to stop Suggestion. The spell should do what it says it does, regardless of what anyone sees or notices. This answer is about who is aware of what happened after the fact.
In general, if a character can make an active insight/perception roll to notice something, they should be able to use their passive scores without an action. You could have them roll against a DC if you want to emphasize the use of active skill checks (and even have different DCs for passive and active checks).
I generally prefer opposed rolls (or rolling against passive values), simply because it gives the player a bit more control, agency, and time in the spotlight.