Although this question already has an answer, and rightfully so. I believe that anyone who visits this page with the same question should know the ups and downs of wielding over-sized weapons before coming to a conclusion.
Lets start with the Core Rulebook
A creature can't make optimum use of a weapon that isn't properly sized for it... The measure of how much effort it takes to use a weapon is altered by one step for each size category of difference between the wielder's size and the size of the creature for which the weapon was designed. For example, small creature would wield a Medium one-handed weapon as a two-handed weapon. If a weapon's designation would be changed to something other than a light, one-handed, or two-handed by this alteration, the creature can't wield the weapon at all.
A medium creature can use a large one-handed weapon as a two-handed weapon. Because you cannot increase a two-handed weapon, this means that a medium creature cannot wield a large two-handed weapon.
A medium short-sword is a light weapon on a Medium creature, and a one-handed on a small creature. Both creatures would utilize its 1d6 damage in this way, except the small creature takes a -2 penalty because of the size difference (when they could simply wield a longsword for the 1d6). A shortsword becomes a longsword, just as a longsword becomes a greatsword, as you increase size categories. The only difference is that you can take a -2 penalty to wield a smaller version of the weapon made for the next size category higher, at the same damage modifier.
Why would you wield a weapon of a size category not of your own?
1) You are a dual wielder (lets say a bard for example) and in your main hand you hold a longsword, and in your offhand you wield a whip. According to the equipment section, a whip is treated as a one-handed weapon, which means that in order to dual wield (assuming you have the two-weapon fighting feat) you will be taking a -4, -4 penalty. Wielding a "small" whip will treat the weapon as a "light one-handed" at the cost of a -2 penalty. This makes your two-weapon fighting penalties drop to a -2/-2 but increases your whip (offhand attack) by 2 because it is sized differently. Making your end result -2/-4 (essentially you are increasing the chance to hit with your main hand)
2) You have specific feats in weapon specialization and weapon focus, that give you a bonus to damage and hit with a specific type of weapon. Lets say my focus is longswords. Rather than wielding a short sword, it would be more beneficial to me to wield a small longsword because the -2 penalty to size is more than made up for with feats that give +2 to hit and damage when I use a longsword. This is linked in with the first reason.
3) You are playing a class that allows you to bypass the size restriction. For example: The barbarian archetype Titan Mauler. The Titan Mauler does not specifically say that you can wield weapons larger than yourself outside of the archetype description, though it is assumed through her abilities with the minor exception of Jotungrip: A titan mauler may wield a two-handed weapon in one hand though this weapon must be appropriately sized for her. Her level 3 ability allows her to reduce the "over sized" penalty by 1, meaning that for wielding a weapon that is larger than yourself you would take a -1 penalty instead of the -2. You would have to talk to your dm about this first to clarify.
How significant is the attack penalty compared to the damage?
A -2 is a difference of 10% on your hit chance. If you have +1 to hit at level 1, and you are versing someone with 12 ac. All you need to do is roll an 11 or higher to hit. Because there are 20 sides to the dice, each side represents a 5% chance, thus you have a 50% chance of hitting your opponent. If you take a -2 penalty, that would require you to roll a 13 to hit, thus making your chances of hitting a 40%.
Mathematically you can multiply your hit percentage by your average damage output to determine which is most optimal to use. In almost every situation, especially early on, the ability to hit your opponent is far better than the potential damage you could have done. As mentioned above, one of the major reasons to wielding an inappropriately sized weapon is to reduce your penalties to hit.
However there is more to take into consideration. As you level, your class will gain BAB (Base Attack Bonus). This bonus gets applied to your hit chance as well as attack bonuses from Feats, Equipment, Ability Modifiers, and Buffs. Your AC on the other hand, does not scale with your level, it instead only increases through Equipment, Buffs, Feats, and Relevant Ability Modifiers. It is because of this, that as you level you will begin hitting your opponents more and more. Your opponents AC will increase for the exact same reason that your attack bonus will increase -> You will have better equipment, and more feats / buffs at your disposal.
As a rule of thumb, do not invest in AC unless you can keep it roughly 20 + your level, (This does not apply right off the bat) example: Level 10 -> 30ac, Level 14 -> 34ac. Else your opponents will be hitting you so much, that it pays off more to simply have higher HP. This is because the attack bonuses at these levels is so high (from both Players and NPCs) that you will be getting hit anyway.
What exactly are the damage numbers?
On page 145 of the Core Rule Book is the chart for damage for weapons of larger or smaller size categories. This chart covers what is not mentioned in the equipment section -> The equipment tables only show the weapons medium and small damage, thus this chart shows it for Tiny and Large.
Each weapon has a BDR (Base Damage Rating), this number is not shown in the book and it is an average of the weapons potential damage. You simply take the lowest roll, and the highest roll and average the two: 1d6 ~ 1+6 = 7, 7/2 = 3.5. A 6 sided die will give you a BDR of 3.5. A greatsword is 2d6, thus 2+12 = 14, 14/2 = 7 (or simply put 3.5 + 3.5). If we were using a Dwarven Waraxe: 1d10, a large version would be 2d8 -> 5.5 increased to a 9.0. So when we say that you are increasing the damage by roughly 3, we are saying that you are increasing your average damage.
Simple so far? That is only the first step, we covered what the average damage potential was, but now I will talk about consistency. When a barbarian has a choice between a greatsword and a greataxe which one do they pick? When we look at BDR the GS is a 7, the GA is a 6.5. This is because both have the potential of reaching 12, but the GS will never do 1 damage. When you graph out all of the possible rolls on a dice, you notice that the more dice you roll, the more and more the outcomes are consistent and average. Rolling 2d6 has the highest percent chance of totaling a 7 than any other number, the next highest percent roll is either a 6 or an 8. When you roll multiple dice, you create a curve, and you make your numbers more consistent. With a single dice your numbers are much more random.
The third step is considering the damage range. As levels progress your opponents AC will not be the only thing you must overcome. You also have to take into consideration damage reduction. If your opponent has DR 10 silver, and you do not have any silver weapons. You are relying on your melee classes doing more than 10 damage. In this case you may hit your opponent 20 times, but if you cannot roll above a 10 on your damage, then your hit was for nothing. In the example of wielding a Large Greatsword (3d6 = 10.5 BDR) in comparison to a Medium Greatsword (2d6 = 7 BDR) taking a small penalty to hit to increase your BDR from a 7 to a 10.5 is extremely worth it!
Food for Thought: If you are versing an opponent that has DR 9 for example (Though this is rarely a case) you will have a higher percent chance of bypassing his DR with the GA than you will will the GS.
I realize the significance of attack bonus, but when should I sacrifice this?
1) When you are versing an opponent that has High DR
2) When you are versing a caster opponent, and the damage you deal applies to your opponents concentration check
3) When you are versing an opponent you believe to have less AC than the average and higher HP
4) When you have reached the point where your attack bonus is so high, that you will gladly take the extra damage (This typically happens after level 10)
My final words
I realize that this post may spark a lot of attention and negative approval. There was a huge argument about this and I believe that both sides of the argument should be presented. Yes a -2 penalty is significant, yet it is the exact same penalty that a dual wielder takes with both the two-weapon feat and a light offhand weapon. There are far more ways of increasing your attack bonus than there are your damage output. Thank you for taking the time to read this.
Best Answer
Tieflings can get a variant heritage that allows you to use Large weapons:
There is a bit of debate on whether or not you need the Fiendish Heritage feat, but the answer seems to be "most likely no, but talk with your GM".