Yes. Given the nature of the item and spell, and given that how they interact isn't explicitly written and therefore the rules put it on the DM's shoulders to decide, how your DM handled it was correct.
The item
Without spoiling too much, a bard's instrument that:
- involves “attunement”
- is able to cause psychic damage to unattuned players
… matches a specific magical item in the DMG's list of magic items.
For those who want to be spoiled, they can read about it in
the DMG on page 176.
The interaction
Whether using a mage hand to play the instrument counts as personally playing the instrument, at least enough to suffer the damage, is in the realm of DM's judgement. There are no rules for how the item interacts with "indirect" playing, so the game falls back on the basic rule that the DM decides based on their best judgement.
Since it's psychic damage and the mind doing the playing is the rogue's mind operating the spell, I would agree with your DM's call. In another game with another DM though, the damage might just fizzle out with no available target instead. Another method of indirect playing may also have a different result — for example, building a machine to strum it would seem (to me) to not count as the builder playing it, but using a stick to play it would. Again, this is explicitly down to DM's judgement, so your DM's call is supported by what the rules say is up to the DM, and I think the call is pretty reasonable.
(And aside, isn't it way more interesting to discover this? That's another, separate reason to appreciate the DM's call here!)
A saving throw
Normally a saving throw is required before suffering this particular damage. If the DM skipped it entirely, that isn't (normally) their call to make and a save should have been rolled.
It's possible that the DM forgot, or maybe rolled the save for you. (I'm generally fine with making rolls for players when it's done to preserve a good mystery. If it was me that would be likely why you didn't roll, but I don't know your DM's habits.)
If the DM simply left the save out, that's something that could be corrected next time. A nice non-intrusive way to do this is to ask "don't I get a saving throw or something?" This nudges the DM if they forgot, without the player engaging in "backseat DMing".
Mostly balanced except for Dance of the Dead
Overall this subclass doesn't appear to be too unbalanced. Playtesting it as you are probably gives you a better idea than any of us would have purely from reading the rules.
Grim Secrets
Part one of this feature seems fine. Though no other college gain a cantrip at this level I don't feel this is fundamentally different to any of the others.
Part two is harder to guess at as it grants an ongoing improvement rather than a static benefit. To fully know if this is OP I would have to compare all necromancy spells to all bard spells and see if this adds a significant amount of power. I feel as though it may be a little over powered compared to the existing subclasses.
Potentially consider moving this to the sixth level to mirror Additional Magical Secrets of the Lore Bard.
Haunted Eyes
Advantage against fear is common enough from many sources so not a big deal. Necrotic resistance is good but situational so not game breaking.
Dance of the Dead
This feature seems to be both over powered and goes against the normal design principles. For reference the other subclasses gain at this level:
- Additional Magical Secrets (Lore), two spells of your choice
- Extra Attack (Valor/Swords), as it says
- Mantle of Majesty (Glamour), Command as bonus action for 1 minute. Once per long rest
- Fool's Insight (Satire), detect thought CHA mod per long rest
- Mantle of Whispers (Whispers), situational disguise
This feature also uses Bardic Inspiration in a way that no other subclass uses them, more akin to the way superiority die are used than bardic inspiration.
This is the feature that needs the most work and I would consider scrapping entirely for something different.
Rule the still heart
This feature is fine, it could actually be considered weak since it only affects one creature per long rest and they get a saving (possibly with advantage).
Suggestions
This isn't too bad as a first attempt of the subclass. I would suggest re-doing dance of the dead. Potentially replacing it entirely. Allow me to propose some features for you, not sure if it suits 6th or 14th level better though.
Grave Humour
When an Undead Creature under your control, that you can see within 60 feet of you makes an attack roll, an ability check or saving throw, you can expend one of your uses of Bardic Inspiration, rolling a Bardic Inspiration die and adding the number rolled to the Undead's result. You must decide before the DM says whether the roll succeeds or fails.
Puppet Master - credit to Doc in comments
When you use the Attack action on your turn, if a creature you control can see you, it can use its reaction to make a melee attack. Only one creature you control can do this per round.
Best Answer
Bardic Colleges are not modern Colleges.
A "college education" means something specific in the modern world: it means that someone has gone to a specific kind of educational institution to get a specific kind of education.
A bardic college is not a modern college. Instead, it's basically a club (PHB 54):
Consider the most learned-sounding college, the College of Lore:
Notice how the first sentence draws a distinction between the bardic college and "actual colleges"?
Your character's progression through levels is how you explain "getting a college education". As they spend time hanging out with people in their college, they learn more and more and gain better and better abilities, which is represented game-wise by your class levels.