Background: In my campaign, my players (each are level 5, having just finished LMOP, and have magic weapons) took up a quest for a group of lycanthropes, and as a reward, were bitten and granted lycanthropy. Soon after, I realized this was a terrible mistake, as lycanthropy is a very overpowered thing to have. However, I don't want to immediately take the lycanthropy away, as that would spoil my players' fun. For the same reason, I don't want all the players' enemies to immediately gain silver weapons. In my campaign, lycanthropes are sparse on the surface, but have a small city in the underdark.
Question: How do I create encounters to challenge my players, without outright negating or countering their abilities?
Best Answer
Use enemies with alternative damage types
The lycanthropic immunity to mundane weapon damage is indeed pretty powerful, but it does only apply to bludgeoning, piercing, and slashing damage. Attacks that deal different or additional kinds of damage - poison, fire, cold, etc - are still effective against lycanthropes. Lots of monsters automatically deal this sort of extra damage with their attacks, and they will remain dangerous (albeit less so) to lycanthrope characters. Normal humanoid-type enemies wielding normal weapons are going to be mostly ineffective against the characters, but that's difficult to avoid while allowing the players to remain unaltered lycanthropes.
Level 5 is one of the major milestone levels where player character power gets a dramatic jump (extra attacks, level 3 spells, etc.) anyway, so you could take the opportunity to shift focus so that the players are now getting into the realm where their challenges are more and more the sorts of fantastic creatures who do deal this kind of damage and powerful enemies of their own tier who have the magical resources and special equipment necessary to deal with them. They should still get to have a go at mundane foes from time to time, but they can be the sort of fluff encounters they're meant to handle easily and quickly rather than a serious challenge.
Use their reputation against them
It would certainly feel unreasonable if enemies who had no reason to do so suddenly started coming equipped with silvered weapons and other anti-lycanthrope measures purely because they need to be able to pose a challenge to your players. However, it would be reasonable for enemies to start using such equipment and tactics if they know they are likely to be fighting lycanthropes. If the party can't keep their nature a secret, their enemies will learn to fight them with silver and fire. Even if the characters are never seen actually transforming, there are only so many conclusions that can be drawn if they are seen to be obviously immune to mundane damage, and silvered weapons are one of the standards for overcoming such immunities.
Groups and opponents who are specifically opposed to the PCs can therefore justifiably arm themselves with silvered weapons and other anti-lycanthrope tools after the secret gets out. If the PCs are well known and active in a region, even groups who aren't specifically opposed to them might invest in having a few such weapons available, just in case these adventurers (a breed who are notorious for getting involved in other people's business) cross their path. You don't need to make every common guard or thug start carrying silver swords, but silvering is relatively inexpensive, so leaders and lieutenants at least could readily be so armed.
Retcon the damage immunity
You could houserule the nature of lycanthropy so that it only grants resistance, rather than immunity, to such mundane damage. This way your players still get a considerable advantage from becoming lycanthropes but it's not quite so game-breaking as outright immunity is. You could even flavour this as a difference between natural-born and afflicted lycanthropes, saying that natural lycanthropes have full immunity but afflicted ones only get resistance (there's already a mechanical difference between the two in that afflicted lycanthropes can be cured by Remove Curse, whereas natural ones require nothing short of a Wish).
Hopefully, you have good-natured players who would understand if you nerfed the ability a little bit in the interests of keeping your campaign grounded.