Smithing capacity

There shouldn't be an armour cap for items, but there is a cap on the amount of damage you can mitigate (80%) this comes at 567 displayed armour rating.

With that said, even if you go over it should still display what your actual rating is, just won't mitigate more than 80% of the incoming damage.

My guess is a bug or you're doing something wrong.

Source

So there are several things going on here, none of which you can control (without the console):

1. The Mace of Molag Bal, despite requiring an ebony ingot to improve, is uncategorized for its classification, so it is unaffected by any smithing perks (other than arcane smithing since its enchanted).
2. Without the proper perk, the best improvement you can make on an item (with a smithing skill of 100), is the Flawless level unless you stack a lot more fortify smithing on top of that.
3. You will need a lot of fortify smithing enchantment to push it beyond flawless (+34 for epic, +68 for legendary). +25 isn't enough.
4. The message you are getting is rather vague and wrong, its the default message for anything you can't improve beyond its current state. What you will need to do is make 4 pieces of enchant smithing gear using this technique.

In the end, even if you stack on the fortify smithing gear, you will not be able to make as good of an item (damage wise) than if you had the perk. Some numbers as example with 100 smithing:

• No fortify smithing no Perk - Elven Armor (Flawless) 45 defense
• Maximum Fortify smithing no Perk - Elven Armor (Legendary) 52 defense
• Maximum Fortify smithing /w Perk - Elven Armor (Legendary) 74 defense

Finally, you can further improve these numbers by making a potion of fortify smithing, but I'll leave that up to you.