In my homebrew setting, one god gave his cultists the recipe for a special alloy, and a rival god gave to a kingdom that opposes the cultists the same recipe.
The PCs acquired a warhammer made of this alloy—called dense iron—, and it alone is sufficient to render insignificant a large number of encounters the PCs face.
The problem is that I've already equipped the cultists and the kingdom's army with dense iron armor, weapons, and shields, so the PCs will need their own dense iron weapons to defeat those foes, but I've made it so dense iron is so powerful that it makes many normal adventures now impossible.
This question's answers are on a much smaller scale. I've not introduced a lone overpowered item; I've introduced an overpowered campaign-spanning plot device.
What's the best way to reduce this alloy's power without leaving my plot in shambles and the PCs wondering Why's my dense iron hammer not working as well as before?
Best Answer
I see three possibilities: Either you change the world (and with it the campaign), change the enemies or you make the ore something more.
Change the ore: Liesmith gave a very nice possible answer, so I'll just put in a few extra possibilities. For example: What makes this ore special? How is it so powerful? Maybe it is stabilised by God #1. Maybe it feeds on the blood of those killed by it. Maybe every kill made with a weapon like this is a sacrifice to God #1. Make the consequences slow to show but long-term. If only the PCs recognize the flaws and problems with the ore, they will have to simultaniously gather evidence about it, fight the cultists of God #1 who rapidly gains strength and convince the kingdom to lay down their powerful super equipment at the same time.
Change the enemies: Adapt or die is something that happens fairly often. What would happen if every generic soldier had very good equipment? Random monsters, bandits and others would adapt, or quickly die out. If the usual tactics only lead to death, the innovators survive. Ambushes become more common and frequently only use ranged weapons. Everybody makes traps to negate the advantages of the ore. People will try to steal the superhammer before they confront the heroes. The monsters who survive become accustomed to being cautious and will avoid close combat whenever they can. So change the enemies, make them use tricks, because those who survived until now would be the tricky ones.
Change the world: A new superpower with powerful, easy to mass produce weapons who hates the gods? That is a massive background to any campaign, and would have far reaching consequences in the greater world. Other kingdoms trying to steal the weapons and the tech. Crusades against the godless. Crusade against the believers. Small groups of soldiers deciding to strike out and use their powerful weapons to become kings on their own. All in all: Total Chaos. Embrace the changes and use them to their full effect.
So these are the three possible ways I think this can go, I think they are all role friendly, but the path you take should depend on what kind of campaign do you want to run.