The whip blade is unbalanced for dexterous characters and pointless for strong characters.
(I am assuming that the interaction to toggle forms is completely free because no action is mentioned in the whip blade's description)
Toggling forms
In most cases, a character can change their reach once during their turn by dropping a weapon and unsheathing another weapon that has more or less reach.
By comparison, the whip blade can toggle its form to change the reach as many times as necessary per turn without the hassle of dropping stuff onto the ground. I think that the toggle ability is decent in the hands of a tactical player, but not game changing.
Dexterous characters (dex ≥ str)
The whip form's lack of finesse is inconsequential because the whip form uses dex anyway.
The whip form is strictly better than whips because of the extra damage.
The whip form is at least on par with rapiers and scimitars. However, if the dexterous character has the dual wielder feat then the whip form is strictly better than scimitars.
Moreover, the whip form is strictly stronger than the blade form too, so the toggle ability is useless outside of some really rare scenarios where having reach is a negative.
Overall, I would almost always use the blade whip with a dexterous character over any other melee weapon.
The exception are rogues because they need the finesse property for sneak attacks. However, rogues are not proficient with whip blades, so it's not worth analyzing.
Strong characters (str > dex)
The finesse property is meaningless to strong characters and so the blade form is strictly worse than all one-handed martial melee weapons except for the whip and the lance, and generally worse than all martial melee weapons.
The whip form is strictly worse than whips when you consider the reduced chance to hit from using dex instead of str.
Overall, I would almost never use the whip blade with a strong character.
My recommendation
Just flavor a regular whip to be a whip blade. When the enemy is within 5' describe the attacks as belonging to a solid blade, else describe the attacks as belonging to a whip-like sword. Guaranteed balanced.
Once the game progresses to the point that magic weapons are available, put something like this in the loot.
Whip Blade
Weapon(whip), rare
You have a +1 bonus to attack and damage rolls made with this magic whip. When you make an attack with it during your turn, you can forgo its reach property to deal an extra 1d4 slashing damage.
This is profoundly unbalanced as written. During downtime days, as written, the item permits the user to pour unlimited numbers of hit dice into the bracers. All you have to do is ensure that you don't end the day with less than half your maximum number of hit dice, and that you finish storing at least an hour before you go to sleep, and you're losing nothing for the next day. You can then tap those whenever you like for major blocks of healing and/or curing the blinded/deafened/paralyzed/poisoned conditions.
The cost of storing is meaningless because no one's going to take the Store action while in combat anyway. There's no reason to, and actions in combat are far too precious to spend unnecessarily. If they're safe enough to store, then the couple of rounds won't matter. Spending a die to cancel the debuff is if anything even more pointless, because then you're burning the die you just stored in order to get rid of the (minimal) side effects of storing it. Between that and the action cost, it means that you're effectively burning a hit die and two actions in order to give yourself a durability penalty for a round while in combat. It's just a terrible idea all around.
In order to have any hope of being balanced, there needs to be a serious limit on how many hit dice can be stored in the thing. Either there's an explicit cap, or the hit dice that are stored in the thing simply don't refresh while stored. The way that gets handled is going to be a significant part of how balanced it is and for what rarity. When the answer is "there is no limit", it breaks the HP economy and therefore cannot be balanced. None of your proposed fixes address that core issue in any meaningful way, and therefore none of them are particularly pertinent here.
For where to put the hit die cap, if you want to go that route? Well, the ring of spell storing is an attuned rare item (a very nice attuned rare item) that can store up to 5 spell levels in a somewhat similar way. Those spell levels could be healing spells (but generally aren't). Figure out how much healing you could reasonably get from packing one of those things full, and that's a floor... because the ring of spell storing is way more flexible than that. For another benchmark, compare it to the max cap of the staff of healing (using only 9 charges rather than 10, because you don't want to risk a 5% chance of it just vanishing). That, too, is an attuned rare item, and its ability to heal others makes it somewhat more flexible, but it's getting closer. The "staff possibly vanishing" and "only cleric/druid/bard" aspects bring it down a bit.
As a sidenote, by my understanding of Brandon Sanderson's work, this sort of thing may be a recurring problem with trying to convert his ideas into magical items for 5e. 5e is pretty careful about not letting things break the system. Sanderson is all about having people take the powers they have access to and figure out ways to use them to break the system. The two philosophies don't mesh particularly well.
Best Answer
I see no real problem with it.
There is precedent for a longsword-type weapon having both the Finesse and Versatile properties (Sun Blade). Beyond that, this is a +X Weapon that hurts you in order to use its properties, making it inferior in that regard to a normal +X weapon.
As a character levels up, taking 1-3 damage becomes an increasingly non-issue. One minor exploit is that if you attain Resistance to whatever damage type the sword does to its wielder (I'd go for Necrotic, since it is drinking blood), then you could use the sword in +1 mode at all times with no drawback. But the fact that it hurts you at all already makes it inferior to a basic +X Weapon.
If you want to shut down the 'Resistance' case, then make sure the damage dealt by the weapon is explicitly stated to be irresistible.
I would not suggest making it state that actual the damage taken by the wielder translates to the +X applied. While this would prevent resistance abuse, it would open the weapon up to Vulnerability abuse. If a character was made vulnerable to Necrotic damage and it followed that rule, they could boost the weapon to a +6.
This weapon would be fairly hard to abuse, because you must be attuned to a magic item to use its properties, so the blood drinking property could not be used on enemies. As mentioned above, it's basically inferior to a normal +X sword...though the Finesse on a Longsword does give Dex-based classes a viable two-hander option.
A few things other I will note in your description, however.
I will note that some degree of care should be taken if you hand this sword out at low level. I would rank it as at least 'Rare' and treat it accordingly (inferior to a +3 Weapon, which is Very Rare). At low level, the ability to spike your chance to hit by 15% for only 3 damage is well worth it for some classes--particularly bursty ones like Paladins who, at level five (with a single round to prepare) can drop 2 Attacks, 2 Spell Smites, and 2 Divine Smites in a single round...bumping their chance to hit by 15% would be a significant boon to such a character attempting to Overnuke a target like this. I am not saying this to note that it is unbalanced...simply that, like most high +X weapons, passing it out at low level can have a significant impact.
Note
It has been brought up that using a Versatile/Finesse Weapon may allow some cheese with a Rogue's Sneak Attack by taking a 1 level dip into Fighter for Great Weapon Fighting. This is a non-issue because GWF only applies to the actual, raw weapon damage...not to bonus sources like Sneak Attack, Smite, etc. See the Question here as well as the following Sage Advice quote:
Source
This pretty strongly states that it wouldn't apply to Sneak Attack either.