Yes, it can

You can make any non-magical trinket that can fit in your hand, and it disappears after the end of your next turn. One of the trinkets you can make, as listed under PHB 160, is "a whistle made from gold-colored wood" as well as "a tiny silver bell without a clapper"; these are both things that you could make such that they can fit in your hand.

Furthermore, they fit the criteria of "musical instruments," because you can make music with these. While the PHB does list some of the things it considers as instruments, it is not meant to be an exhaustive list:

Tools, PHB 154

Musical Instrument. Several of the most common types of musical instruments are shown on the table as examples.

This means prestidigitation can create a musical instrument that can fit in your hand, and you can use any musical instrument as a spellcasting focus.

Of course, it is inefficient

You must use your action to cast prestidigitation, and then you cannot use your action again to cast another spell. On your next turn, you could use the instrument to cast a spell, but then it disappears at the end of that turn, and you have to recreate it on your third turn. This allows you one spell every other round that isn't prestidigitation (unless it's a bonus action or reaction spell).

Intrument of the Bards only works on spells that impose the charmed condition

When you use this instrument to cast a spell that causes targets to become charmed on a failed save, the targets have disadvantage on the saving throw.

Spells that don't say that they impose the charmed condition do not count as imposing that condition and thus do not meet the criteria in Instrument of the Bards.

For example, otto’s irresistible dance does say:

creatures that can’t be charmed are immune to this spell

However, it says nothing about imposing the charmed condition which is the only thing that matters for the effects of Instrument of the Bards.

Jeremy Crawford also supports this here:

"Charm spell" has no meaning in the rules. A spell refers to the charmed condition if that condition is relevant.

and also here:

When the game refers to being charmed, it's referring to the charmed condition. There isn't a general category of charm effects.

Side note: RAW/RAI the Instrument of the Bards' effects can only be used on spells with material components

There is another reason why otto’s irresistible dance would not work with Instrument of the Bards as well.

Since Instrument of the Bard says:

When you use this instrument to cast a spell

This means in order for the Instrument of the Bards' effects to apply to a spell you cast, it must be used as a spellcasting focus (or be the source for the spell). And spellcasting focuses are only used when a spell requires material components.¹

Jeremy Crawford also backs this up:

Q: If attuned to an Instrument of the Bards and cast Charm Person, does the target make its save with disadvantage?

A: Charm person doesn't benefit from an instrument of the bards, since the spell has no material component.

^{1 - It also means that it can only be used on bard spells (see Does using Instruments of the bard gives disadvantage to enemies if I use it to cast wizard spells?).}