There's no way to get the damage in a general case, as whichever actions you should choose depend on the circumstances and it's impossible to know the circumstances ahead of time.

You can calculate the damage amounts based on certain assumptions, though. For example, if you assume you'll have advantage about every other turn, just take the average of the damage you do when you have advantage and the damage you do without advantage.

This has the caveat that it relies on the accuracy of your estimate, but it's literally the best you can do. Try to get a good estimate on how often you'll use each of the options and weigh each option's average damage accordingly. For example, if you deal 10 damage every fourth turn and 8 damage otherwise, your average damage should be:

$$ \left( 8 \times \dfrac 3 4 \right) + \left( 10 \times \frac 1 4 \right) = 6 + 2.5 = 8.5 $$

So does this become (9 + 7.5 + 9.9 + 5.4 + 11.25)/5 = 8.61 at level 1?

Only if all the five different cases are equally common, which I don't think is likely. Without knowing the 5e mechanics very well yet, I assume some of these five options happen far more than the others, meaning they'll have a higher impact on your overall average damage per turn.

Dual wielding daggers does scale way better with your sneak attack than using only one rapier.

At first if all attacks hit the damage calculation would be:

$$ \text{Two Daggers}: 2d4+\text{DEX}+\text{SNEAKATK} \\ \text{Rapier}: 1d8+\text{DEX}+\text{SNEAKATK} $$

With Dex = 3 and Sneakatk = 1d6 that would be:

$$ \text{Two Daggers}: 6-17 \text{(Avg. 11.5)} \\ \text{Rapier}: 5-17 \text{(Avg. 11)} $$

Variable Hit Chances:

The problem is, that not all attacks hit.

We know that daggers and rapiers use the same attack bonus, thus having exactly the same chance to hit.

$$ \text{Two Daggers}= \text{Hit Chance}\times(1d4+3)+\text{Hit Chance}\times(1d4)+\text{SNEAKATK}\\ \text{Rapier}= \text{Hit Chance}\times(1d8+3)+\text{SNEAKATK} $$

Calculation for Sneak Attack:

If we assume, that every hit is eligible for sneak attack, we can add the damage to the daggers if any of the two attacks hit. If my math does not fail me that calculates as follows: (For two Daggers)

$$ \begin{align} \text{One Attack Misses} &= (1-\text{Hit Chance}) \\ \text{Both Attack Miss} &= (\text{One Attack Misses})^2 \\ \text{Sneak Attack Chance (NOT Both Attack Miss)} &= 1-(\text{Both Attack Miss}) \\ &= 1-(1-\text{Hit Chance})^2 \end{align} $$

Also let us talk in average damage for ease:

$$ \text{Two Daggers}: \text{Hit Chance}\times(8)+(1-(1-\text{Hit Chance})^2)\times(3.5) \\ \text{Rapier}: \text{Hit Chance}\times(7.5) + \text{Hit Chance}\times(3.5) $$

Base Damage:

Regardless of HitChance the daggers deal more damage in average. As the HitChance can never be less than 0.05:

$$ \text{Hit Chance}\times8 > \text{Hit Chance}\times7.5 $$

Sneak Attack damage:

The damage would be equal in both cases, so the relevant part is the trigger chance. Assuming the attacks are eligible for a Sneak Attack, the relevant part would be whether you hit or not.

When attacking twice, your chance to hit is higher than when only attacking once.

$$ (1-(1-\text{Hit Chance})^2) \geq \text{Hit Chance} $$

Assuming 0 < HitChance ≤ 1

Proof (Thanks to @Glen_b):

$$ 1-(1-H)^2 = 1-(1-2H+H^2) = 2H-H^2 \\ = H+H(1-H) \geq H \forall 0\leq H \leq 1 $$ with equality only possible at the endpoints and a maximum difference at $$H=0.5$$

Here is a graph to visualize the chances that you can apply your sneak attack damage:

Y = Sneak Attack Chance; X = Hit Chance per Attack; Red = Two Daggers; Blue = Rapier

Thus, two daggers are better with and without sneak attack.

Here is another graph visualizing the average damage for both weapons (Assuming +3 Dexterity modifier, 1d6 sneak attack, no magical bonuses, all attacks eligible for sneak attack.)

Y = Damage; X = Hit Chance per Attack; Red = Two Daggers; Blue = Rapier

Opportunity Costs

Through OPs comment on another answer, I know his rogue is level 2. This means he could use his bonus action to Dash, Disengage or Hide.

Later on, depending on your subclass and table rules, you might gain Feats or other class features like Fast Hands which require your bonus action. So, why would I take the Two Daggers instead of the Rapier? This would trade in positioning possibilities or other maybe useful opportunities for a measly 0.5 damage.

While the damage difference might appear to be 0.5 in average, the real advantage of dual wielding comes with the sneak attack chance.

Look at it this way:

If you have only 50% hit chance the difference would be

$$ (0.5\times8+0.75\times3.5) - (0.5\times(7.5+3.5)) = 1.125 $$

The daggers now deal 1.125 damage more in average.

Same scenario, but without sneak attack:

$$ (0.5\times8)-(0.5\times7.5) = 0.25 $$

Only a 0.25 damage difference. This gap increases drastically as sneak attack damage increases.

So your goal is to get your sneak attack damage through. If your first attack hits, your rapier deals in average 2 damage more than the dagger, but if the first attack misses you deal no damage with the rapier, while dual wielding gives you another attack to possibly deal your sneak attack damage.

Why is this so important?

When looking at your weapon attack damage, how does it scale?

• Your Dexterity modifier might go up (but would benefit both weapons equally)
• Your sneak attack damage increases with your level (better for daggers, because of higher hit chance if you attack twice)
• You may get a magic weapon

The 2 Avg. damage difference from the damage dice, if the first attack hits looses its relevance while the increased chance of sneak attack gains in relevance.

Let's look at a level 5 rogue who used is ASI on Dex (+4) and has now 3d6 Sneak attack damage. With a 50% hit chance: [the numbers in square brackets are without sneak attack]

$$ \begin{align} \text{One Dagger: } 0.5\times(2.5+4)+0.5\times(10.5) = 8.5 \text{ [3.25]}\\ \text{Two Daggers: } 0.5\times(2.5+4)+0.5\times(2.5)+0.75\times(10.5) = 12.375 \text{ [4.5]} \\ \text{Rapier: } 0.5\times(4.5+4)+0.5\times(10.5) = 9.5 \text{ [4.25]} \\ \text{Rapier+1: } 0.55\times(4.5+4+1)+0.55\times(10.5) = 11 \text{ [5.225]} \end{align} $$

Y = Damage; X = Hit Chance per Attack; Red = Two Daggers; Blue = Rapier

Even a Rapier+1 does not reach the Avg. damage of two daggers.

Here an level 20 (+5 Dex, 10d6 Sneak attack damage) example:

Y = Damage; X = Hit Chance per Attack; Red = Two Daggers; Blue = Rapier

The damage gap increases, but to get the most out of your turn I would recommend the following:

If your first attack with dual daggers hit, accept that you might have dealt more damage with a rapier and use your bonus action to get away (improve your survivability). If this is no option you can attack another time to maybe deal 1-4 extra damage.

If the first attack misses, attack another time, this is where dual wielding shines.