You'll need to "translate" the glass bottle data tag into something that works with target selectors (@a and so on). This can be achieved most easily using dummy scoreboard values or, in 1.9, scoreboard tags.

In 1.8, create an objective using

scoreboard objectives add bottleInOne dummy

Create a fill clock and run the following two commands

scoreboard players set @a[score_bottleInOne_min=1] bottleInOne 0
scoreboard players set @a[score_bottleInOne=0] bottleInOne 1 {SelectedItemSlot:0,SelectedItem:{id:minecraft:glass_bottle}}

If the team part is necessary, you can now add @a[score_bottleInOne_min=1] to the team, and remove @a[score_bottleInOne=0] from it, using the same fill clock. You could also use these target selectors directly for the /effect and /replaceitem commands, if the team part was just your idea of selecting that player for the effect.

In 1.9, you can omit the scoreboard objective in favor of tags. Using a Repeat/Chain command block chain, run

scoreboard players tag @a[tag=bottleInOne] remove bottleInOne
scoreboard players tag @a[tag=!bottleInOne] add bottleInOne {SelectedItemSlot:0,SelectedItem:{id:minecraft:glass_bottle}}

In the following commands, you can then use @a[tag=bottleInOne] or @a[tag=!bottleInOne] to target players with and without the bottle selected in slot 0, respectively.

Ok, so it turns out, my logic works fine, unfortunately sometime between 1.10 and 1.12 nested execute command support was removed. pity. If anyone has a different way of doing this, please let me know. In the mean time, I'm going to try and figure out how to do this without nested execute commands...