The rules on somatic and material components say:

If a spell requires a somatic component, the caster must have free use of at least one hand to perform these gestures.

[...]

A spellcaster must have a hand free to access a spell's material components--or to hold a spellcasting focus--but it can be the same hand that he or she uses to perform somatic components.

So yes, the hand holding your focus can also perform the somatic components. It doesn't matter what the other hand is doing.

But, this only works if the spell has Material components. If it has Somatic but not Material, you will need a free hand to cast it (unless you have the Warcaster feat).

You are interpreting the rules correctly, but a bard of valor can cope with the restrictions.

We had a bard of valor in a campaign I DM’d for, and we stuck to the rules, which you described accurately. The bard was able to cope with these restrictions, although it required a little more “power gaming” than some players might enjoy.

Verbal-only Spells

Spells with just a Verbal component, such a Healing Word got special consideration. Healing Word is a good fit for a front-line healer. Casting Dissonant Whispers while within melee reach was another go-to tactic, letting the bard and his companions make opportunity attacks.

And the signature bardic cantrip, Vicious Mockery, gave our bard the ability to attack at range, while keeping his melee weapon and shield equipped.

Spell components are relevant

As you mention, War Caster is a great feat for a bard of valor, and our bard eventually attained that. At that point, Verbal-Somatic spells joined the list of spells that deserved extra consideration.

Juggling Sword, Board, and Focus

If the bard wanted to cast an attack spell with a material component in combat, he could drop or sheathe his sword (as a free object interaction) and to access his arcane focus.

The bard typically sheathed his sword for VS spells, or dropped it for spells with material components. If he dropped it, he would draw a spare weapon next round. (Dropping the sword became a less attractive option, though, after the bard acquired a magic sword.)

If an opponent provoked an opportunity attack while the bard had no weapon in hand, the bard still had the option to kick as an Unarmed Strike (see PH 195) and shield bash as an improvised weapon.

In some situations, such as fighting a spell caster with weak melee attacks, the bard would drop his shield and keep his sword ready for opportunity attacks. (But note that “doffing” a shield requires an action.)

I want to stress, though, that the juggling described above was not a constant thing. If a spell is cast before engaging in melee, for example, not having a weapon drawn was no issue.

A fine point about whether an arcane focus needs to be “interacted” with

To use a material component, “a spellcaster must have a hand free to access a spell’s material components — or to hold a spellcasting focus” (PH 203). But unlike “drawing or sheathing a sword,” (PH 190) accessing a material component is not explicitly mentioned as something that requires your interaction “with one object or feature of the environment for free.” (PH 190)

Whether the rules require your free object interaction to cast spells with a material component is beyond the scope of the question, but for our purposes, just find out how your DM will rule. If they rule that holding your focus does not require your free object interaction, then you probably won’t ever need to drop your weapon or shield.

Other Bard Powers

A bard often has a fair deal of Bardic Inspiration to hand out in combat, which does not require a free hand. While these aren’t spells, they are part of a bard’s action economy and their pool of limited-use resources.

Non-Combat Spells

One final point, many bard spells are non-combat spells, or have a long enough duration that they can be cast ahead of combat (e.g., Enhance Ability).

Complexity, not Disadvantage

A bard of valor may require a little more power-gaming, in terms of character build and tactics, than some character builds. But sword-and-board Rangers face similar constraints. These constraints don’t make bad character builds.