This is quickly going to become too complicated to tell you for sure if your deck will be able to support a structure like a pergola, but if it is even half the weight of your typical hot tub then your deck will very likely not support it.
To accurately determine this you need to figure out the additional #'s/sq ft of load that a structure as large as a pergola will place on top of the deck. Doing this is simple with something like a spa where you have a clear square footage area and a relatively even weight distribution when filled with water and people. This is not so easy to determine with a structure where certain points of the structure are going to carry the weight of the structure differently. The weight is not as evenly disbursed.
Right off the bat there I would advise you to consult an engineer with your structure plans to determine if this is feasible and safe. This is completely disregarding the fact that your deck is only as strong as its weakest point. If your house is brick and your ledger board is attached with carriage bolts then even if the joists were on 12" centers then you are still likely putting entirely too much stress on the ledger board. You also could have a lot of problems if any part of your structure is going to sit over that 2' overhang.
In the end, an experienced engineer will probably be able to confirm pretty easily if this will be feasible or not.
First, your question
how do I calculate what that new beam has to be given the existing joists and the desired post options?
This is a bit long winded, but it is how we figure it out accurately.
Figure out a design load per square foot (PSF), we typically use 40 PSF live load and 10 PSF dead load for floors and decks. In a simple span, the beam carries the load halfway to the next support or tributary width; in this case 5'. Multiply the PSF x the tributary width to get the pounds per lineal foot (PLF) on the beam, in this case 50 PSF x 5 Ft = 250 PLF. Then, using software or load tables find a beam that can carry that load based on the span of the beam; allowable loads reduce as the span increases. In your case this requires (3) 2x10 DF #2.
I typically use Forte for beam sizing instead of load tables.
For the post sizes, calculate the tributary load which, in simple spans, is the load halfway between supports. In your case, this would be half the beam length between post multiplied by PLF. For your center post that is 250 PLF x 11.5 FT = 2875 LB. Then, using load tables or calculations find a post that can support that load based on height and bracing of the column. Take into account bearing area of the 2x10 at about 625 PSI perpendicular to the grain for a required bearing area of 4.6 Sq. In. In your case a 4x4 is plenty.
Try a column load calculator here
For multiple span beams, cantilevers, and anything outside of "simple span" things are a little different. For this reason I typically use the beam sizing program.
If this is too intense, maybe just be guided by the following.
If I read correctly, you have deck joist spanning about 10' and on one end a single 2x10 "beam" or rim joist that the joist hang into that spans approximately 11'-6" between (3) 4x4 post.
Based on this, your deck joist are structurally OK although maxed out at a 40/10 load, but are deflecting (bending) a good amount giving part of the bouncy feel. You can add 2x6 or larger joist down the middle of the existing joist to firm things up, or just live with a safe bouncy deck.
The 2x10 is definitely undersized and should be addressed. It also contributes to the bounciness. The simplest solution to this is to add (2) intermediate post mid span reducing the span of the 2x10 to below 6'. OR beef up the 2x10 by adding (2) more 2x10's to it nailing 3 16D nails per foot per beam lamination would easily transfer the load between beams.
Finally, the 4x4 post are more than adequate to support a load of 2,875lb. An 8'-0" length of 4x4 can support over 6,000 lb in wet service.
We applied over 10,000lb to a 7' 4x4 in jacking up a floor with no indication of possible buckling; we were compression limited on the member being jacked.
Best Answer
Notch the support columns? 2x6's for joists when 2x8's fit? Yeah um, stay away from those people. Concrete would be best overall since it's impervious to most everything from water rot, to termites to air leaks.
With concrete you'd want to cinder block the perimeter, throw down 4"s of rigid foam insulation & pour a proper 4" thick slab on top of the insulation. However, there's no problem with wood either & that would be much more DIY-friendly.
With wood you'd do 2x8's everywhere & would box-around the support beams, definitely do not notch the supports. 2" rigid insulation would go on the bottom & on the perimeter frame, then fiberglass batts can fill in the rest.