Mostly balanced except for Dance of the Dead
Overall this subclass doesn't appear to be too unbalanced. Playtesting it as you are probably gives you a better idea than any of us would have purely from reading the rules.
Grim Secrets
Part one of this feature seems fine. Though no other college gain a cantrip at this level I don't feel this is fundamentally different to any of the others.
Part two is harder to guess at as it grants an ongoing improvement rather than a static benefit. To fully know if this is OP I would have to compare all necromancy spells to all bard spells and see if this adds a significant amount of power. I feel as though it may be a little over powered compared to the existing subclasses.
Potentially consider moving this to the sixth level to mirror Additional Magical Secrets of the Lore Bard.
Haunted Eyes
Advantage against fear is common enough from many sources so not a big deal. Necrotic resistance is good but situational so not game breaking.
Dance of the Dead
This feature seems to be both over powered and goes against the normal design principles. For reference the other subclasses gain at this level:
- Additional Magical Secrets (Lore), two spells of your choice
- Extra Attack (Valor/Swords), as it says
- Mantle of Majesty (Glamour), Command as bonus action for 1 minute. Once per long rest
- Fool's Insight (Satire), detect thought CHA mod per long rest
- Mantle of Whispers (Whispers), situational disguise
This feature also uses Bardic Inspiration in a way that no other subclass uses them, more akin to the way superiority die are used than bardic inspiration.
This is the feature that needs the most work and I would consider scrapping entirely for something different.
Rule the still heart
This feature is fine, it could actually be considered weak since it only affects one creature per long rest and they get a saving (possibly with advantage).
Suggestions
This isn't too bad as a first attempt of the subclass. I would suggest re-doing dance of the dead. Potentially replacing it entirely. Allow me to propose some features for you, not sure if it suits 6th or 14th level better though.
Grave Humour
When an Undead Creature under your control, that you can see within 60 feet of you makes an attack roll, an ability check or saving throw, you can expend one of your uses of Bardic Inspiration, rolling a Bardic Inspiration die and adding the number rolled to the Undead's result. You must decide before the DM says whether the roll succeeds or fails.
Puppet Master - credit to Doc in comments
When you use the Attack action on your turn, if a creature you control can see you, it can use its reaction to make a melee attack. Only one creature you control can do this per round.
Scorching ray is always more damaging if you have the same chance to hit
I could tell you even before running the numbers that this would be the case. As long as both spells are on equal playing ground (both have adv/dis, etc.), the higher average damage will always win out. This is because even though scorching ray is split among 3 attacks, the average damage will stay the same. The difference between scorching ray and chromatic orb's damage is only going to be the deviation (the likelihood that the result is close to the average) in the damages:
Notice that the scorching ray damage is slightly more likely to hit near the average (since there are more dice involved). As you can see, I had to correct the scorching ray by 3 points for them to line up (to show the different deviations) meaning scorching ray will deal 3 more damage than chromatic orb if they all hit. Since each ray has an independent chance of hitting, the damage output approximates to the same.
The numbers
Linked here, you can find a spreadsheet with all of the calculations you requested (feel free to download a copy if you want to check out the formulas I used). Essentially it only involves figuring out the chance that the character hits and multiplying the average damage by that chance. Since the scorching rays are independent, you can simply multiply this adjusted damage by 3.
In summary, scorching ray will always be more damaging if the chance to hit is the same throughout.
The case for chromatic orb
There is one instance where chromatic orb deal more average damage, and that is when only the first attack will have advantage (such as if you used true strike). The modified damage in this instance is in the spreadsheet linked in the previous section to show how this changes the numbers. In this case, the greater damage that is applicable to an advantage attack does more than the extra average damage of scorching ray. However, as your normal chance to hit increases, this advantage to chromatic orb decreases and scorching ray catches up.
Versatility of chromatic orb
Chromatic orb still has a comparative advantage over scorching ray and that is its versatility. Fire is one of the most resisted damage types in the game, so if you are dealing with a monster with such a resistance, scorching ray falls off hard, but chromatic orb can select a different option. This lets you go around resistances and target vulnerabilities, greatly increasing utility (perhaps at the cost of raw damage).
Best Answer
For the sake of argument, we're going to assume that the way this character uses Elven Accuracy (which, when a creature has Advantage, allows one of the Advantage dice to be swapped with a new die, at the user's whim—XGE 74), is that this character will always swap their lowest rolled die with the new one, which is the mathematically optimal way to use it.
Swift Quiver is always going to offer greater damage output
Unfortunately, while having Advantage, especially Advantage with Elven Accuracy, is a pretty decent boost to damage output, it's just not going to compare to having two additional attacks.
Below, I've constructed a table representing the statistically average damage output per round of this character, assuming no other buffs or expended resources than what you have specified, and assuming that with using Swift Quiver, one attack gains Advantage (and thus Elven Accuracy), and with Greater Invisibility, both attacks gain Advantage (and thus Elven Accuracy). I've also included single attacks with a longbow, using a normal roll and an EA-Advantage roll.
We can see that having Advantage is producing a modest, ~9.6% improvement in overall damage for an individual attack, but it's just not going to compare to a direct, +100% improvement in damage when using Swift Quiver.
Note an important fact however: this improvement becomes far less pronounced at higher ACs on the target. Indeed, I extended the array to calculate for AC 29+, and found that at that point, the difference becomes less than 1%. However, there is no point where mathematically, Swift Quiver deals less damage, no matter how high the AC goes, and no published creature in 5th Edition D&D has an AC higher than 25, which is the AC of the Tarrasque.
So ultimately, the choice this character needs to make is whether the 50-20% improvement in damage is worth losing their Bonus Action used for other purposes, like healing allies or providing Inspiration. If so, they should use Swift Quiver. If not, they should use Greater Invisibility.
Full stats for this analysis were posted here, for anyone curious.