they were in an area of magical darkness and were blind.
This is correct; the vision rules state that being inside darkness without a way to see through it is equivalent to being blind.
we ruled that he was attacking an invisible creature.
Technically incorrect; based on the rules for the Blinded condition, the creature had Total Concealment from them.
We then ruled that they could not attack the creature was due to it having effective invisibility to them.
Well, mostly correct; invisibility and Total Concealment are slightly different in some cases, but in this it’s the same. Total Concealment says “You can't attack an opponent that has total concealment, though you can attack into a square that you think he occupies. A successful attack into a square occupied by an enemy with total concealment has a 50% miss chance.”
They then tried to use the Ricochet Splash Weapon feat to gain a new chance to hit the unseen creature. After more lengthy discussion it was ruled that because he attacked the square and not the creature, that it was considered a successful hit and could not make the attempt to hit the creature.
Total Concealment gives a 50% chance to miss. If they fail the coin flip, then they do in fact miss. For a thrown weapon, this means you now roll on the misdirection table, and they can then use Ricochet Splash Weapon from there. Since the miss means they actually land in a different square, they cannot use Ricochet Splash Weapon to hit the same target they were going for. But they can roll an attack (at −5) to hit some other target, if they happen to land in a square that has something to hit. This attack would also be subject to the 50% miss chance.
Was this the correct way to deal with what I viewed as a possible abuse of the feat's intent (which I already feel is a little overpowered). Did I rule fairly?
“Fair” is largely a matter of opinion, but it does sound as though you were incorrect in your ruling, with respect to the rules. However, you also seem to have some confusion about how Ricochet Splash Weapon works in general.
Ricochet Splash Weapon does not give you a second try at attacking the same target. When you miss with a splash weapon, you have to roll 1d8 to pick a direction that you miss in. Your attack therefore ends up in a different square than you originally targeted. You can then use the feat to attack whatever is in this new square, not the original target.
This is, in fact, a very weak feat. Touch attacks are extremely easy to make, so misses should be rare, and the odds of happening to miss into a square that has an enemy is going to be very low most of the time. Thus, it will be incredibly rare for this feat to do anything at all.
The only requirement for Shape Breath Weapon is "Breath weapon as a special attack." 'Special Attack' isn't well defined for non-npcs, however 'natural' breath weapons (from a dragon, gorgon, etc.) use the Universal Monster Rules 'Breath Weapon' entry, which specifies that they are supernatural abilities. The Bomber class says that it uses the Bomb class feature of the alchemist, which is also a supernatural ability.
Since both of these abilties are the same type, and that Breath Weapon Bomb specifies that it is being used as a breath weapon, I'd conclude that it is sufficient to satisfy the requirements of Shape Breathe Weapon.
Best Answer
It doesn't work in a strict rules-as-written reading of the rules:
Healing Bomb(emphasis mine):
But strictly speaking Splash Weapon Mastery doesn't increase the splash radius: it only makes creatures on an other case suffer the splash damages.
However ruling that "creatures in the splash radius" should be read as "creatures in a case affected by the splash damages" works fine. There is no official statement on this topic that I know of.