It appears I have gone and created more questions then I have answered.
First of there are 3 ways to level up once you hit Veteran Ranks; Veteran Overwold zones (questing), Veteran Dungeons (same dungeons you did while leveling but much harder) and PVP.
The short answer is more like your 3rd option. Not in the way you described it though. Level 50 A.K.A. Veteran Rank 1 is when you are sent to the Veteran Overworld Zone, which is the Veteran questing area where you do the questlines of the other two factions. Only in these areas everything is Veteran Rank. Veteran NPCs, Veteran quests, Veteran Rewards, Veteran Gear, Veteran level resources, you name it, its Veteran.
If you so choose to go back to your home faction questing area the highest loot you will find is level 50. Nothing will be Veteran except for the occasional resource you might find in coldharbour that lets you make Veteran gear. And yes you can go back simply by teleporting.
Also the first time you do this, (speculation at this point) the questing will only put you between veteran 4-6 which you will then start the next factions questing zone. The idea behind Veteran ranks is to take a very long time.
Alright time for my long answer full of spoilers and such (this is my attempt at not causing more confusion)
So you have finished Coldharbour, foiled Molag'Bal's plot of destroying Nirn. You have just hit level 50 (AKA Veteran rank 1(hopefully, those that don't have to grind to 50 cringe)) Now you are being summoned to the harborage to fight molag'bal. Once defeating him you are asked if you wish to see what it would be like if you had washed up on a different shore, saved by a different faction. If you accept, you will start your journey through whatever faction the game chooses for you. I am unsure if it is random or if there is a linear path of which one is next. You start out just like if you had started out making a new character for a seperate faction, except you are still veteran rank 1 and still have all your skills and gear. Also everything is veteran as well as mobs which by the way are much harder to kill. Traps are just as deadly, one misstep will hurt, two missteps will be deadly. So if you couldn't help yourself and read this part of the answer enjoy veteran content its way more challenging and fun then normal mode.
See if I can answer all of your questions.
What exactly is the point of leveling a skill?
To gain access to higher level skills.
I sure do get that you can unlock new Skills and perks but does It do something else?
No.
Will you gain more Armor/Damage if you have a Higher rank in a certain skill in ESO or is it solely for the sake of unlocking new stuff
Yes. Not sure how much it scales.
Also if you Level up an Active Ability, will this ability get stronger or is it also just for upgrading an ability. If it's only for upgrading, then why can I further level the upgraded skill (I've noticed that you can upgrade an Ability only once.
Yes. Not sure how it scales though. I believe they are planning on implementing one more morph to the skill lines in the distant future.
Another thing I'd like to know is: What is the max level of an Ability/Skill
Rank IV > Morph Rank IV. Max Level of weapon/armor/class skills is 50.
Best Answer
All your stats are boosted, no matter whether they're based on your character (like health or stamina) or your actual items.
Just keep in mind that due to this items will scale similar to how your character is scaled. For example, if you're indeed level 10 and your weapon is level 5, then don't expect to have the equivalent of a level 50 weapon. It will be more likely around level 40 or even lower.