I have not playtested this (specifically not the advantage system you propose), but I will give some statistical analysis on it. Hope it helps.

From my understanding of the question, you want a comparison between current system and yours.

All the plots made are assuming no modifier. If you want to add a modifier, simply translate both the curves by the modifier. For example, in the first figure (as follows), a +5 modifier means the DC actually starts from 6, instead of 1.

# Advantage

Essentially, your method makes it easier to succeed at easy tasks with advantage and harder to succeed at harder tasks. But this is true for the 3d6 system by itself.

It might be interesting to see the actual difference between the two curves, so...

Essentially, you are giving +5% chance of succeeding at low difficulty tasks for a -30% of succeeding at harder ones. If you just keep the DCs as they were in the original form, this means the players are having a really harder time.

The thing is: you can adjust the DC, so comparing the probabilities from the original system to the current system might not be the best. We can, however, compare **how impactful is advantage** in both systems.

This is where things get tricky: your advantage is not as helpful as original advantage. For every DC, the **advantage gain** (this is how I'm calling the difference between the probability of success with and without advantage) from your system is lower.

However, this doesn't tell us completely how **impactful** the advantage is. Finally, plotting the relative increase

\$ G = \frac{\textrm{Pr}_A}{\textrm{Pr}}\$

where \$\textrm{Pr}_A\$ is the probability of success with advantage and \$\textrm{Pr}\$ the same without advantage.

So, technically, your advantage means they are succeeding relatively more than before, even more than the original system for medium/higher DCs. This happens because increasing the probability from 1% to 2% is "more impactful" (if it would take 100 tries, now it takes 50) than increasing from 33% to 50% (if it would take 3 tries, now it takes 2). So, from this point of view (how many times I'll have to try until succeed), your advantage **is more impactful**, since it should not be needed in low difficulty tasks (variance is already low enough that they already succeed on it almost everytime) and gets to 3.5x more likely to succeed at 18 DC (again, remember this is without modifiers).

## Note on modifiers

From the first plot, you can see that your system's probabilities drops way faster (inclination is higher). This means **modifiers are also more impactful**. Things like expertise and ASI become more valuable, since sliding the graph now means a higher increase in chance of success than before. The same goes for negative modifiers.

In the original system, people used a "rule of thumb" that advantage roughly meant a +5 modifier. This is no longer true in your system - a **+5 modifier is way more impactful than advantage**.

# Disadvantage

I'll just skip to the third plot I've mentioned in advantage. So, advantage got less helpful from an absolute point of view, but more impactful . What about disadvantage?

Welp, turns out this is basically the same, but inverted.

But wait, the relative curve is pretty different.

This time around, disadvantage is less impactful, overall.

# Ability Contests

We're also interested in ability contests. This plots how often you will win against someone with the total modifier in the x axis. The total modifier is given by (His modifier - your modifier), so if you have a +7 and he has a +5, the total modifier is -2. I'm using \$\geq\$ condition, i.e., the ability contest is won in case of a draw.

Again, the effect is: the easier it is, the easier it becomes. The harder it is, the harder it becomes. While you still had a ~15% chance of winning a contest against someone with +10 bonus against you using the 1d20, you have a ~1% chance of doing so using 3d6. For a +5 modifier, it goes from ~30% to ~15%.

The consequence is simple: a good liar will be deceiving people way more often than before.

# Conclusion:

These changes modify the game balance considerably. Actually, modifiers become so impactful that getting them through ASI or feats that give flat modifiers is now better than having advantage. **Bardic inspiration** becomes alot more powerful. **Expertise** becomes alot more powerful. Overall, **bards and rogues get most benefits**, since they get more proficiencies (which are now more impactful, as mentioned) and more flat modifiers.

That is, again, assuming that you are being consistent with DCs, rather than making them up on the run based on how often you want the players to succeed. In this case (changing the DC according to the probability you like to achieve), the system itself doesn't matter, i.e., there is **always** a DCX in the d20 system (almost) equivalent to a DCY in the 3d6 system. Note: almost because the values are discrete, not continuous, i.e. you can't achieve a 99% chance of success in a d20 system (without adding extra rolls and doing some math), since it increases in 5% steps.

# My experience on D&D 3d6

I've mentioned I didn't playtest the advantage/disadvantage system. When I did play 3d6 in D&D, we used the sum as "a roll". Advantage would be "roll 3d6, roll 3d6 again, pick highest". I won't bother you with more plots and math, but essentially it's even more extreme than your changes (honestly, because we didn't think on yours back then). I have applied them to combat as well.

Basically, it **breaks bounded accuracy**. And it felt bad. Characters would just not be able to do things that they were not good doing, even with help (check first plot). Things they were already good doing would become even easier, but honestly these were things I didn't even ask for a roll most of the time. The way I had implemented it, advantage and specially disadvantage became way more useless. Even by your method, it doesn't matter the PC will succeed twice as often when they are essentially never succeeding anyway.

The major difference from our experiences, though, is that I and my players discovered we actually liked the swingy-ness of the d20 rolls. Specially true for Goblin Dice. And I'd already decided by then to not put important plot elements or important things overall behind dice rolls, which, by the way, I suggest: exactly because of the swingy-ness, don't put important things behind dice rolls. Even with the 3d6 system that's not a good idea, mainly because if it's hard they will fail even more frequently.

**The thing is: decreasing the variance means you know the outcome more often than not. For us, that wasn't fun.**

# Notes for Combat

OP mentions he is aware of the problems that using 3d6 in combat leads to. Nonetheless, I'm maintaining the combat section for other people interested.

Feats like sharpshooter (if you are applying this to combat as well, not only ability checks) get a lot worse, since the -5 (de)bonus gets the probability to decrease a lot faster now.

As an example (for the sharpshooter), if you were at DC10 (which would happen, for example, against an AC 14 target and a +4 attack modifier), it becomes DC15. Originally, with advantage, it meant decreasing your probability from ~80% to ~50%. Now it decreases from ~90% to ~20%. It means it's almost never worth to use feats like these.

Moved from my experience section: Also, the overall change means Goblins just wouldn't be able to hit the 18 AC character. Basically, for combat, AC and Attack Bonus both become way more important than HP and Damage done when you use 3d6 instead of 1d20.

# Additional notes on modifiers

I was talking about this in the comments and I've decided to edit it into the answer. For the normal scenario (i.e. without advantage, disadvantage, lucky or whatever), a +5/-5 modifier (in the 1d20 system) around the DC11 changes the probability of success from 50% to 75%/25%. This is achieved by a +2/-2 modifier in the 3d6 system. The 3d6 curve starts to get a little non-linear after the +2/-2, but it can be **approximated** as linear until around +5/-5, so a +10/-10 modifier in 1d20 system can also be approximated by a +4/-4 modifier in the 3d6 system.

Thus, as a rule of thumb, if you want to keep the power levels of modifiers the same as they were before, **divide the modifier by 2.5**. This also shows how much stronger modifiers are in the 3d6 sistem: 2.5 times.

I find the main reason to use the 3d6 system is exactly to make modifiers stronger, though, and reward characters for being experts in what they are doing. So, if you are changing to 3d6 to begin with, you probably don't want to cripple the modifier powers levels again. But I'm adding this to people interested anyway.

## Best Answer

http://anydice.com/ will do this quite easily for you:

Roll 8 drop lowest 3 (which is just keep highest five) :

Click "at least" to see the probability of getting 20+, which is 79.97%