Like many things in the BoVD, yes, this spell is horribly dysfunctional. There's no rules coverage for a lot of obvious consequences of the spell's use, and what rules there are are at best unclear and at worst contradictory.
Focusing on the part of the spell text that deals with the actual casting and not the aftermath:
Soul Shackles
Necromancy [Evil]
Level: Brd 5, Sor/Wiz 5
Components: V, S, F, Location
Casting Time: 1 [standard] action
Range: Close (25 ft. + 5 ft./2 levels)
Target: One living creature
Duration: Instantaneous
Saving Throw: None (see text)
Spell Resistance: Yes
The caster draws out the soul of a dead creature and imprisons it within a specially made talisman. The subject must have had the talisman in his possession when he died, or the spell cannot function.
[...]
Focus: The talisman that will be the receptacle for the soul.
Location Component: An area under the effect of a desecrate or unhallow spell.
If we try to extract from that the circumstances under which the spell can be cast...
As per standard spellcasting requirements, the caster must:
- speak the verbal and act out the somatic components of the spell
- possess the focus (the talisman)
- be in the required location (the Desecrated area)
- spend a standard action (the casting time)
The target needs to be:
- within 25+(5 per 2 levels) feet
- be a single living creature
The talisman:
- must be in your possession on casting, as you're using it as a focus
- must have been in the target's possession when it died, despite the fact the spell must be cast on a living target
So, as written, this only works on a single living creature that has previously died with the talisman in their possession, but who is now alive; and the caster has to be nearby, with the talisman and stood in a Desecrated area. That's obviously silly, and not how the spell is meant to work. Although the spell isn't completely impossible to use as written, it's nonsensical garbage - it effectively works as a Trap the Soul with no save against anyone living if the caster possesses a talisman the victim held the last time they died (i.e. it only works on people who have previously died and been resurrected).
What I think this is meant to do:
I'm generally in agreement with your first reading. The immediate contradiction between "Target: 1 living creature" and the first sentence of the description calling the victim "a dead creature" seals it as an error for me.
The second reading is pretty much ruined by the use of the past tense in the text - "The subject must have had the talisman in his possession when he died, or the spell cannot function" - which places the target's death chronologically before the casting of the spell can take place. The talisman focus required to cast the spell is only complete once the target dies while holding it, and without it "the spell cannot function".
I think this spell was meant to be a lower-level version of the second usage of Trap the Soul (in which you trick someone into touching a prepared vessel, and it sucks their soul out), with the attendant extra restrictions and casting hurdles because it's level 5. In this case, you're meant to somehow get them to hold the talisman (reverse pickpocketing, disguise it as loot, a Sympathy spell, etc), then kill them in a Desecrated area while they still have it and quickly cast this spell on their corpse to trap them. You could also force the talisman onto an uncooperative enemy or sacrifice while they're restrained, then kill them to trap their soul with this.
How I would fix this broken mess:
First, change the target line to "1 dead creature (see text)". Add extra specifications to the text limiting the time the creature is allowed to be dead for (probably something like 1 minute/level), and specify that the target must also be in a Desecrated area.
Second, add some sort of description of how one prepares the talisman, and what forms it can take. I'd probably go for any object with some small gp cost for the process (something like a 1 hour ritual, one talisman per ritual, max simultaneously prepared talismans equal to level), and make it radiate faint Necromancy once "prepared" (presumably the caster would go to some effort to hide that, otherwise it's a potential tip-off for the victim).
Third, add extra rules describing what happens if the Talisman is destroyed or damaged - "the spell ends and the soul departs to the afterlife", as with a successful saving throw (or, if a victim hasn't yet been trapped, the spell simply ends). That plus the revised target requirements eliminates the possibility of one person being affected by multiple spells, as only one spell can affect their corpse initially (it's instant, and after the first cast there's no soul left to trap - if they had multiple talismans on them, the Necromancers responsible will have to race to be the first to cast this spell on the body) and once the soul is in the talisman it'll be outside this spell's targeting restrictions.
The highest effect applies. If there is more than one effect, the highest effect still applies to each individual effect. The rule doesn't differentiate between occurrences or individual spells.
The effects of the same spell cast multiple times don't combine, however. Instead, the most potent effect—such as the highest bonus—from those castings applies while their durations overlap.
When spells overlap, you aren't saving against one spell. You're saving against them all simultaneously with a single roll. So in this instance, the highest DC would be 15, and the highest damage would be 4d8.
The rules clearly states they don't combine, only the most potent effect applies. So you can't add them together to get a spell save DC of 20+ and damage of 11d8, but you can replace their values completely by a better one.
This means the target makes it's save against the highest DC, in this case 15, and damage is rolled at 4d8.
Effectively, multiple instances of the same spell function as one spell modified for maximum benefit until their durations individually expire. Note that this conlcusion is derived specifically from the wording of the quoted text above. At no point does it say only the most potent spell applies. It specifically states the most potent effect does. If there's multiple effects, the most potent of those must apply by this rule.
Best Answer
Multiple of the same spell never stack. Player's Handbook, “Combining Magical Effects”:
With three casters of spirit guardians you can cover more ground, but you will never get more damage or saves out of them by overlapping them.