It is imbalanced, but not because of Spell Compendium

The imbalances you note are very real. They have often been noted, commented upon, and even codified in the 3.5 tier list.

• Clerics and druids, along with archivists and wizards,¹ simply are the most powerful classes in the game because they have access to all of these spells and can change them every day.

• Sorcerers are a tier below, because they have access but cannot change them.

• Bards are a tier below that, because they only go to 6th-level spells.

• Rangers are another step down: only 4th-level spells.

• Fighters are a tier below that, because they have no spells.

This marginally over-simplifies the tiers, but it's pretty close to true: spells are, in 3.5, just about equivalent with power. Only a few classes meaningfully change tier despite their spells or lack of them.²

But this is not because of Spell Compendium. This isn't even because of supplements in general. This is due to some fundamental design mistakes that Wizards made early on. They underestimated spells, overestimated the significance of what non-magical types were doing. Player's Handbook is the most imbalanced book they published; they slowly learned from their mistakes and did better in future products (well, mostly).

So clerics and druids have this advantage; yes, absolutely. And it's a big one. But (very nearly) all the best spells are core anyway. Adding Spell Compendium (or any other source of spells) doesn't matter very much to them, since optimally 90% or more of their spells will be still be core.

The bard, paladin, and ranger are actually the big winners: instead of just getting mostly the same spells the bigger classes got many levels ago, they actually get some unique stuff, greatly expanding their ability.

Fighters are still left in the dust, but they were always there. Not much can be done about them; they weren't well-designed. Better to replace them, such as with Tome of Battle classes. But even failing that, Spell Compendium doesn't make them worse, it just means a few classes that would otherwise be down there with them get to start to move upward.

Spell Compendium is a good book, full of interesting material, that avoids a lot of the overpowering mistakes of the core spells. Clerics and druids may find a few things to pick up, but ultimately most of the spells are weaker. The book is really best for half-casters.

1. ^{2 spells per level plus any others you come across is plenty, particularly off of those spell lists. There are numerous ways to build a wizard to improve on the 2 spells/level if one anticipates a game where spells will be few and far between, but ultimately you don’t even need to – forty spells from the Sor/Wiz list is just a phenomenal amount of potential power.}

2. ^{Some example exceptions from “more spells = higher tier” include
   • The Eberron Campaign Setting artificer: Tier 1 along with archivist cleric, druid, and wizard, despite only 6th-level “spells,” thanks to incredible versatility provided by their item-creation abilities
   • The Miniatures Handbook healer, which is Tier 4, with the ranger and below the bard, despite cleric-style spellcasting up to 9th level, since their spell list is incredibly one-dimensional and weak.}

The rule for Quicken Spell implies that swift-action spells do not have their casting time affected by metamagic. In the case of a spell that is swift because of Quicken Spell, other metamagic feats explicitly have no effect: the rules state that spells modified are entirely exempt from the casting time rules. That would include any casting time increases from other metamagic feats.

By comparison, then, to spells that have a swift-action casting time due to Quicken Spell, we can see that spells that have a swift-action casting time naturally should also have their casting time unaffected by metamagic. However, this is not clearly spelled out in the rules, and furthermore Paizo has explicitly declined to address the issue (see this FAQ candidate thread, marked “no reply required”), so this implication is the best we have available to answer the question.