Declare Thaneship
If you are the Thane of a hold, you can declare this to the guard. This will immediately clear your bounty, but can only be done once per hold. This only applies to bounties less than 3,000 gold.
Pay your bounty
If you pay your bounty, you will be immediately transported outside the nearest town jail, and all stolen items in your possession will be seized. Thieves Guild members have an additional option to pay a much lower fine, if they so choose. This additional option will allow you to keep stolen items.
Bribe guard
Thieves Guild members have the option to bribe the guard. The amount of gold required to bribe a guard is usually much larger than the bounty for small crimes, but less than one for murder and doing so will raise the Speech skill, immediately clear your bounty and avoid having items confiscated. To do this however, you have to do five jobs in the region and complete the special task. The Bribe Guards Speech perk also grants you this option. Note, the bounty cleared is only for the crime you were recently caught doing. If you had some previous bounty as well, this won't get cleared.
Persuade guard
Occasionally another option will appear if you have a low bounty. You can attempt to persuade the guard that you are not worth the effort of dealing with, due to your low bounty, and should be allowed to go free. Success is dependent on the Speech skill. If successful, you will raise Speech, but will not clear your bounty.
If your speech is 75 or higher, this option will always be present (given that the guard has not been persuaded already). If you wish to persuade the guard again, wait 2 days and the option will be available again.
Go to jail
If you cannot pay your bounty, you will be sent to the nearest town jail.
You are able to keep one lockpick in your jail cell, which can be used to try to escape from prison. Escaping from jail can be much easier than in Oblivion with an option to pick the door or a loose grate on the floor which is much easier. There are also other methods to escape, such as pushing down a crumbling wall. You can also pickpocket the jail key from a passing guard to escape the cell. However, it is still difficult to recover your equipment.
To serve your sentence, simply sleep in the bed in your cell, and you'll sleep through your entire sentence. Depending upon the length of your sentence, your progress towards one or more skill advancements will be reset to zero. Unlike Oblivion, you will not lose any past skill increases; instead, it will take longer to obtain the next skill increase(s).
If the guards don't give you a chance to submit and go to jail or pay your bounty make sure that you sheath your weapon. The guards should then stop attacking and should hopefully hear your out :) If they don't you can try this:
Best Answer
Crime is generally localized to the hold you're in: if you are kill-on-sight in one hold, running to the next hold over should cause the guards to drop pursuit without new guards joining the chase.
However, in general, you should always be able to pay the guards, even if they decide to attack you. As soon as they do, sheathe your weapons/spells/shields and hope they trigger the dialogue to allow you to go to jail or pay off your bounty.
If they don't, there are a few things you can try to do:
As Bri and RoughNeckRage mention, sometimes all you need is time. Wait out a few days and see if the situation hasn't died down by then. The guards will still want you to make reparations, though.
An anonymous user mentions that you can try sneaking up to a guard from behind and trigger the dialogue screen to pay off your bounty.
Stof mentions another possible thing, the bounty collector. The UESP suggests it's not guaranteed, but if he does visit you, you can pay off your bounty for a 20% premium (and even keep your stolen items!).