[RPG] How does Monk damage combine with other sources

I have few ideas based on your criteria but the best option really boils down to the situations the party finds themselves in and how the party functions in combat. Because of that, I'll list some options and give the pros and cons of each.

Idea 1: Monk 17/Ranger 3

(Take Hunter's Mark, Dueling, and either Colossus Slayer or Horde Breaker)

Possible Damage: 4d10+4d6+[1d8(CS)]+24 for 3 hours (Attack, Extra Attack, Hunters Mark x4 (each weapon attack) [, and Colossus Slayer])

Pros:

• Up to 3 hours of use split into up to 1 hour increments 3 times per long rest.
• Even if you are out of ki and Hunter's Mark uses, you still get Dueling bonus to weapon damage.
• Bonus 1d8 if target is under its max HP, even if you don't have ki or Hunter's Mark. (If you take Colossus Slayer)
• You can hit 1 additional target for 1d10+7 each. (If you take Horde Breaker)
• Ranger class features are nice to have for adventuring.
• d10 hit die

Cons:

• Hunter's mark only recharges on Long Rests and might be ended earlier than intended if you fail a concentration check.
• You don't get the full effect on the first round. You need to use your bonus action to cast Hunter's mark which means you don't get your bonus unarmed or Flurry because it requires a bonus action, only your 2 attacks +2d6 (+1d8 if you have CS and target is below max HP).
• Every time your mark dies, you have to use your next bonus action to pick another mark.
• Because Hunter's Mark is a concentration spell, you risk losing the effect if you take damage.

Idea 2: Monk 17/Fighter 3

(Take Dueling and Battle Master or Champion)

Possible Damage: 4d10+1d8+24 up to 4 times between rests or you run out of ki. (Attack,Extra Attack, Flurry of Blows, 1 damage dealing maneuver)

Pros:

• Superiority Die and Ki recharge on a short rest.
• Maneuvers have utility effects (Battle Master)
• Dueling
• Action Surge gives high burst potential once per short rest. (6d10+1d8+38)
• Second Wind (though 4-13 HP isn't that much at level 20)
• Proficiency with 1 artisan's tools. (Battle Master)
• d10 hit die
• Increased Crit Range (Champion)

Cons:

• You only get 4 rounds of bonus 1d8 damage.

Idea 3: Monk 17/Rogue 3

(Take Assassin or Swashbuckler)

Damage Potential: 4d10+2d6+20 (Attack, Extra Attack, Flurry of Blows, Sneak Attack)

Pros:

• Bonus damage not reliant on recharge. Only have to worry about ki
• If you don't have advantage, you can still sneak attack the target if one of your other party members is in melee range.
• Expertise (double proficiency bonus in 2 skills)
• (Swashbuckler) Sneak attack if no one else is within 5 ft of you.
• (Swashbuckler) Add your Charisma to initiative (if you want)
• (Swashbuckler) Your target can't make AoO against you.
• (Assassin) Crit on surprised target for 8d10+4d6+20
• (Assassin) Proficiency with Poisoner's Kit for potentially more damage or other utilities.

Cons:

• Smaller hit die (d8).
• Smaller minimum damage potential.
• Monk weapon must be finesse. (Not too bad, just limits weapon choice.)
• Can't sneak attack with unarmed strike
• Very reliant on party placement

could this work?

No. All weapons with the "heavy" property have the two-handed property as well. Here's exactly what each property does (PHB 147):

Heavy. Small creatures have disadvantage on attack rolls with heavy weapons. A heavy weapon's size and bulk make it too large for a Small creature to use effectively.

Two-Handed. This weapon requires two hands when you attack with it.

The heavy property itself isn't a problem here, but the two-handed is. Since all heavy weapons are also two-handed, you must use two hands to wield them. The enlarge spell doesn't give you the ability to wield a two-handed weapon in one hand, even if that weapon wasn't enlarged with you. Your DM might allow it, but it's certainly not part of the Rules As Written.

So what can you dual wield, then? Here's the rules on two-weapon fighting (PHB 195):

When you take the Attack action and attack with a light melee weapon that you're holding in one hand, you can use a bonus action to attack with a different light melee weapon that you're holding in one hand. You don't add your ability modifier to the damage of the bonus attack, unless that modifier is negative.

Based on this, you can only dual wield light melee weapons. You don't even need the two-weapon fighting style, since all the fighting style does is allow you to add your ability modifier to the bonus action attack. There is a way to get around the "light" limitation, namely the Dual Wielder feat (PHB 165, emphasis mine):

You master fighting with two weapons, gaining the following benefits:

• You gain a +1 bonus to AC while you are wielding a separate melee weapon in each hand
• You can use two-weapon fighting even when the one-handed melee weapons aren't light
• You can draw or stow two one-handed weapons when you would normally only be able to draw or stow only one.

This feat allows you (among other things) to use any one-handed melee weapons for two-weapon fighting.

In conclusion, you're not going to be two-weapon fighting using greatswords, even if you're enlarged. The closest you're going to get would be two longswords with the Dual Wielder feat

Side note: If you want to do two-weapon fighting purely from an optimisation perspective, you may want to reconsider. It is generally accepted as one of the weaker playstyles, as it uses your bonus action to be effective (needed to cast smite spells on a paladin, for example), and still can't quite compete with great weapon fighting, especially if you factor in the Great Weapon Master feat. It doesn't fall completely behind other playstyles, so if you want to do it because of a character concept, go ahead. But if it's exclusively for optimisation, it might be better to just use a single greatsword.