Since the answer @AshleyNunn posted didn't contain enough information (it was more or less exactly the same as what the game tells you), I looked into it a bit.
Highlander and Overkill mode, seem pretty straightforward, as Ashley mentioned - bigger XP boosts and some limitations in the former, or just ludicrous powerlevelling in the latter.
In the Score Attack mode it seems that though you are underleveled from not encountering random enemies, you still have to walk through empty dungeons to reach the bosses. So it isn't exactly a boss rush in the common meaning of the phrase.
The Cthulhu's Angels mode seems to be substantially different. The main storyline is heavily altered, including the dialogue and characters. (E.g. one of your enemies in the standard mode becomes your ally, etc.) There are only four characters available (all of them girls, of course.)
You can go back to an easier area and continue to farm enemies by going to the menu and selecting "Fight" as many times as you need to in order to level up.
This will manually kick off a fight, even if you've exhausted the random battles for an area.
Best Answer
Try pressing alt and enter at the same time.