Bob is Frightened but there is almost no effect on him
The Frightened condition still persists, as there is nothing stopping or dispelling it in this case, but there is no effect because Frightened says:
(PHB 290)
Since the source of its fear was transformed into a frog, Bob does not suffer Disadvantage on checks and attacks, as Bob cannot currently see the Dragon.
As to whether or not Bob can move closer to the frog is contentious. Unlike the first bullet, the second effect does not require line of sight. I know I certainly wouldn't move closer to a frog that was, just a second ago, a Blue Dragon (What if the spell ends or it breaks free like one of those pokemon balls?!) I could see a DM ruling in either way, if the second effect persists or not, but I'd personally let the second bullet remain in effect, because it's funny.
Both bullet points take effect, but the second one is vacuous.
In its title and in its body text, the question explicitly asks about what happens if you are frightened of yourself. This is exactly what I am going to answer. Whether casting the cause fear spell on yourself (intentionally or by Spell Turning) becomes a non-issue because being frightened would cause you to lose your concentration on the spell immediately, would be a different question.
Considering the first bullet point: The crucial question here is: is a creature considered to be within its own line of sight? The only precise definition of line of sight is given in the context of the optional rule of playing on a grid (DMG, page 251, emphasis mine):
To precisely determine whether there is line of sight between two spaces, pick a corner of one space and trace an imaginary line from that corner to any part of another space. If at least one such line doesn't pass through or touch an object or effect that blocks vision -- such as a stone wall, a thick curtain, or a dense cloud of fog -- then there is line of sight.
The word another seems to prohibit using this definition in order to decide whether there is line of sight within the same space - which would be the relevant case in this scenario.
In the absence of a general mechanical definition of line of sight, we will have to go with its plain English meaning. I just tried around a bit, and I find it difficult to move my body naturally without seeing any part of it (or the clothes I'm wearing), unless I close my eyes. In fact, no matter where I look, my nose and part of my beard are always in my peripheral vision. It takes active effort not to see any part of my body, so I'd find it rather unnatural not to consider a creature to be within its own line of sight. Being the source of its own frightened condition, it should thus be subject to the consequences of the first bullet point.
Note that this interpretation of line of sight is consistent with the rules on Targeting Yourself with a spell (PHB, page 201), as is discussed e.g. here, here, or here (thanks to NautArch for referring me to these questions).
Considering the second bullet point: Since there is no additional condition in this bullet point, it definitely takes effect. The only remaining question is whether it actually makes a difference. "Moving closer to something" is obviously supposed to mean "reducing the physical distance to something", not some metaphorical or spiritual distance, or whatever other notion one could come up with. The rules on movement in combat say:
On your turn, you can move a distance up to your speed.
This seems to equate the distance to something with the movement it takes to get there. Since staying exactly where you are costs no movement at all, the distance to yourself is therefore 0. As a negative distance is impossible, no conceivable move could bring you closer to yourself than you are at the moment. Consequently, you cannot move closer to yourself - regardless of whether you are frightened of yourself or not.
However: That said, I think your ruling sounds much more reasonable, more in line with the idea of the frightened condition, and simply more fun than sticking to this strict reading of the rules.
Best Answer
This sounds like a DM problem more than a rules problem
It sounds like you were fighting a singular enemy in a square room with no line-of-sight-breaking cover and only one exit. In this case, Fear is an extremely potent spell for the reasons you gave. However, your situation seems like an outlier, because usually the above conditions are not all present at the same time.
In 5E, especially with casters, having a single enemy is a sure-fire recipe for DM heartache. There are simply too many encounter-ending save spells available. Hold person, Fear, Hypnotic Pattern, and others can very easily lock down a single enemy with almost no hope of them breaking out.
In a 'normal' encounter, there would be cover (so the enemy could get behind it and make a save), there would be multiple enemies (one errant hit on the Wizard who probably doesn't have much of a CON modifier and the concentration breaks down), and there would be multiple exits so the frightened creature could run out and raise the alarm. Your encounter seems almost perfectly tailored to the strengths of Fear, so it seems Fear is overpowered.
I liken this to someone saying:
In normal circumstances, Fear will not have the dramatic effects that you experienced.
To specifically answer your question of 'Is Fear in line with other spells of the same level', there is nothing in the book that allows you to calculate the relative or absolute power of a spell, so the answer will come down to opinions. However, given the restrictions placed on the spell, and the fact that the creature maintains its other defenses and its ability to call for aid, the spell certainly isn't drastically more powerful than other spells of its level. It is certainly possible to craft situations in which Fear would be exceptionally powerful, but this is possible of nearly any spell regardless of level. In a general, level-appropriate encounter with a general, level-appropriate number of enemies in a generalized setting, Fear will not be an encounter-ender most of the time.