Yes, both effects can apply to the same spell.

There's nothing in the wording of either feature to indicate that it excludes other features adding damage as well. Many other combinations of features can work this way -- a barbarian's damage bonus from Rage is cumulative with their Strength damage bonus and other bonuses such as the Dueling fighting style. The word all of these have in common is add.

Contrast this with Thirsting Blade, which says:

You can attack with your pact weapon twice, instead of once, whenever you take the Attack action on your turn.

It doesn't say "you can attack one additional time", so the reason it doesn't stack with Extra Attacks is that neither one adds attacks, they both just change the number of attacks from one to two.

This is also similar to how the barbarian and monk Unarmored Defense features don't "stack" with armor. Armor provides one way of calculating AC, Unarmored Defense provides another. A given character can only use one.

So, in general, effects or features that let you add something to a roll are cumulative, but effects or features that change a value are not.

You can only add your Charisma modifier once: "when you cast a spell ... add your Charisma modifier ...". However, you can choose (before rolling damage) if the primary or secondary target suffers the extra damage. A kind DM (like me) may even let you split the extra damage as you desire.

Naturally, Elemental Affinity is more effective on area of effect spells (Burning Hands, Fireball etc.) where even though the extra damage is added only once it can affect multiple creatures.