Not Necessarily
Vampire's Charm
The specifics of Vampire's Charm are more impactful than a standard charm:
The charmed target regards the vampire as a trusted friend to be heeded and protected. Although the target isn't under the vampire's control, it takes the vampire's requests or actions in the most favorable way it can...
In this case, the charm increases from Charm Person's friendly acquaintance to trusted friend to be heeded and protected. Agency should still not be fully removed from the character, but this is a much stronger effect and should treat the interactions with the Vampire differently. You wouldn't necessarily go against all your beliefs, but you are going to protect the vampire and listen to it more than you would just a friendly acquaintance.
Combine this with the mechanics of the Charmed Condition and you also gain advantage on ability checks to interact socially.
A charmed creature can't attack the charmer or target the charmer with harmful abilities or magical effects.
The charmer has advantage on any ability check to interact socially with the creature.
Does advantage translate to anything more?
The big question is whether or not you can persuade someone to do something against their beliefs. Mechanically, the charmed target is still under their own control, they just view the charmer as a trusted friend and therefore will listen to them more than they normally would. Whether or not that means you can take their agency away is going to be dependent on the DM and table.
How I'd rule
At my table, I would not allow one creature to persuade another to do something that is against their beliefs, unless they could come up with an argument that makes that action feel like it goes with their beliefs when it really doesn't. But that's going to be on a case by case basis and a very tricky effect to pull off. If a party member can't persuade another party member to do something against their beliefs (which they shouldn't), then neither can someone casting Charm. Vampire's Charm is still going to have them protect the Vampire, but not necessarily allow the Vampire to dictate actions (even with a successful Persuasion Check.) To do that, you need something like Dominate Person.
In your specific case, Protecting the vampire is your major concern. Allowing him to have a trial rather than summary execution would likely fall under that. But I could also see the Vampire trying to persuade that you that he wouldn't get a fair trial. It doesn't mean you'd help him escape given your beliefs, but it would mean you'd try to make sure it was a fair trial.
3.5e dominate person and calm emotions—no interaction
In 3.5e, dominate person is not a charm spell, but rather a compulsion spell—the two have different rules and different effects. For that matter, calm emotions has nothing to do with either unless they specifically altered emotions—so calm emotions could suppress the rage spell,1 also a compulsion, but does nothing against dominate person.
5e dominate person and calm emotions—total suppression
In 5e, dominate person applies the Charmed condition, and then also supplies an effect that reads “While the target is Charmed,” and calm emotions says it suppresses any effect that causes the target to be Charmed—which means that the “While the target is Charmed” clause is no longer met, so yes, 5e-calm emotions suppresses 5e-dominate person.
About being a “10th-level” spell—that sounds like a misreading
Dominate person isn’t a 10th-level spell in either edition—it’s 5th-level in both. However, in 3.5e, sorcerers get 5th-level spells at the 10th of that class—and thus the spell would have caster level 10th. That wouldn’t affect very much—just how close you have to be and how long it lasts. Meanwhile, 10th-level spells, as such, didn’t really even exist—there were rules for 10th-and-higher level spell slots, which meant you could cast lower-level spells from those slots for increased effect,2 but it wasn’t the same as there being printed “10th-level spells.” There were also Epic spells, which counted as 10th-level spells, but used entirely different rules. So I kind of doubt that the module has dominate person as a true 10th-level spell, because that isn’t really a thing in 3.5e. It’s not impossible, but seems unlikely.
Whatever “10th-level” really means, it doesn’t protect against calm emotions
Whether we’re talking about caster level 10th or spell level 10th, that doesn’t really matter to 3.5e-calm emotions. For example, it does suppress all fear effects—even if they’re cast with caster level 10th, or Heightened to a 10th-level spell slot. Neither of those would matter, because calm emotions does not indicate any limitation that prevents it from suppressing fear effects based on level.
Conclusion: if using 5e-dominate person and 5e-calm emotions, should have total suppression
In 3.5e, calm emotions would not be a valid response to dominate person at all, but in 5e, it is. And in both editions, the spell level doesn’t matter at all for this purpose—calm emotions does not indicate any difficulty suppressing higher-level effects in either edition.
And I recommend using the 5e versions of both spells here, because you are playing with the 5e rules. In 3.5e, there were other, common effects that could block dominate person,3 but those aren’t available in 5e. You instead used the 5e spell that is supposed to protect against Charm effects, calm emotions. That was the correct choice, and it should have worked.
Probably; 3.5e is not always clear about what is and what isn’t an emotional effect. Calm emotions calls out the morale bonuses of the rage spell, but doesn’t mention the other effects.
Using metamagic, which weren’t a sorcerer class feature but instead feats that made spells use higher-level spell slots in exchange for some increased effect. 3.5e also does not have default effects for casting spells from higher-level spell slots—even basic things like a higher saving throw DC, which was based on spell level, required the Heighten Spell metamagic feat.
For example, the core 1st-level cleric spell protection from evil could block all domination—the 5e protection from evil and good only blocks the Charmed effects of certain creatures. And note the wording—like calm emotions, it blocks the Charmed status. This is the way 5e handles blocking domination, instead of having charm and compulsion as separate things.
Best Answer
Effects only end when they say they will end
Some abilities have lines such as "won't do anything suicidal" but enslave does not. The only reasonably applicable end condition is a save when they take damage, not before.
Conceivably this could result in a situation where the target does as they are told, gets on the altar, then makes the save and comes to their senses as the sacrificial dagger is being thrust into them, but they would certainly climb onto that altar willingly. They are under the aboleth's control quite unambiguously.
The lore on aboleths also states
Which pretty much confirms it in my eyes. Seized control and slaves are very strong terms. The victim does exactly what they are told, until they come to their senses.