True polymorph is the only spell that will work

There are no spells that transform any cloth objects into stone or metal by default. Flesh to stone comes the closest though, as the name implies, only works on flesh (actually living creatures). However, if you are a very high level caster and have two days to work with then true polymorph should work.

True polymorph can transform a creature into another creature, an object into a creature, or a creature into an object. However we are trying to transform one object made of cloth into another object made of stone or metal. That means you have to cast the spell twice - once to transform the cloth object into a creature and then again to change the creature into an object made of stone or metal.

Note that the cloth object must abide by the restriction:

the object must be neither worn nor carried by another creature

Since this is a 9th level spell and a caster only gets one 9th level spell slot per day by default (outside of boons and the like), this means a single caster with no access to a free 9th level slot will have to cast it once, take a long rest, and then cast it again. Not exactly ideal, but it seems to be the only way to accomplish what you want.

However it is also good to note that the effect will last an hour after the second casting and can be ended at any time by stopping concentrating. After an hour it must be dispelled.

Level 14+ transmutation wizard can get kind of close

As spotted by @BlueMoon93 in their answer, transmutation wizards have a feature that gets kind of close to doing what you want but are not spells.

At level 14 you would get a feature that would allow you to change a cloth object into a metal or stone one.¹ However it is not going to a perfect transformation.

You can transmute one nonmagical object—no larger than a 5—foot cube into another nonmagical object of similar size and mass and of equal or lesser value. You must spend 10 minutes handling the object to transform it.

However, you the new object must be of the same mass. So if you changed a cloth shirt into a stone object it would have to be much smaller since stone is much denser than cloth. So if you wanted to change the cloth object into stone while keeping it otherwise the same it isn't going to work. It also takes 10 minutes to do and would require another 10 minutes to change back. But, it does successfully allow you to change a cloth object into a different one made of stone or metal.

^{1 - At level 2 you get a feature that seems like it might be helpful, but actually is not for you:}

^{You perform a special alchemical procedure on one object composed entirely of wood, stone (but not a gemstone), iron, copper, or silver, transforming it into a different one of those materials.}

^{However, as written it would not work on cloth so would not work for your purpose.}