they were in an area of magical darkness and were blind.
This is correct; the vision rules state that being inside darkness without a way to see through it is equivalent to being blind.
we ruled that he was attacking an invisible creature.
Technically incorrect; based on the rules for the Blinded condition, the creature had Total Concealment from them.
We then ruled that they could not attack the creature was due to it having effective invisibility to them.
Well, mostly correct; invisibility and Total Concealment are slightly different in some cases, but in this it’s the same. Total Concealment says “You can't attack an opponent that has total concealment, though you can attack into a square that you think he occupies. A successful attack into a square occupied by an enemy with total concealment has a 50% miss chance.”
They then tried to use the Ricochet Splash Weapon feat to gain a new chance to hit the unseen creature. After more lengthy discussion it was ruled that because he attacked the square and not the creature, that it was considered a successful hit and could not make the attempt to hit the creature.
Total Concealment gives a 50% chance to miss. If they fail the coin flip, then they do in fact miss. For a thrown weapon, this means you now roll on the misdirection table, and they can then use Ricochet Splash Weapon from there. Since the miss means they actually land in a different square, they cannot use Ricochet Splash Weapon to hit the same target they were going for. But they can roll an attack (at −5) to hit some other target, if they happen to land in a square that has something to hit. This attack would also be subject to the 50% miss chance.
Was this the correct way to deal with what I viewed as a possible abuse of the feat's intent (which I already feel is a little overpowered). Did I rule fairly?
“Fair” is largely a matter of opinion, but it does sound as though you were incorrect in your ruling, with respect to the rules. However, you also seem to have some confusion about how Ricochet Splash Weapon works in general.
Ricochet Splash Weapon does not give you a second try at attacking the same target. When you miss with a splash weapon, you have to roll 1d8 to pick a direction that you miss in. Your attack therefore ends up in a different square than you originally targeted. You can then use the feat to attack whatever is in this new square, not the original target.
This is, in fact, a very weak feat. Touch attacks are extremely easy to make, so misses should be rare, and the odds of happening to miss into a square that has an enemy is going to be very low most of the time. Thus, it will be incredibly rare for this feat to do anything at all.
Magic Missile cannot be 'targeted'
The Rogue's sneak attack...
If a rogue can catch an opponent when he is unable to defend himself effectively from her attack, she can strike a vital spot for extra damage.
...must be aimed at a 'vital' part of the creature. Magic Missile...
The missile strikes unerringly, even if the target is in melee combat, so long as it has less than total cover or total concealment. Specific parts of a creature can't be singled out. Objects are not damaged by the spell.
...cannot be aimed in such a manner. It simply 'hits' the target.
Best Answer
From the Magic section
You cannot see the target therefore they cannot be targeted by the spell using sight. If the player was within 5 feet, they could try to touch the target.
To explain why this touch will cause an AOO and why normal touch spells do not provoke attacks of opportunity:
The spell in question is not a touch spell, so you are not normally considered to be armed (Unless you have a natural weapon or improved unarmed strike)
Although, if it was just normal invisibility, actually taking that AOO would break the spell, which is also useful.