Minecraft – How to search for a specific item with attributes

Your issue here is that your values are not integers. Because enchantment level and id are both stored as shorts, you must add an 's' after them. Thus, the working command would be:
/testfor @p[r=5] {Inventory:[{tag:{ench:[{id:16s,lvl:1s}]}}]}

You'll need to "translate" the glass bottle data tag into something that works with target selectors (@a and so on). This can be achieved most easily using dummy scoreboard values or, in 1.9, scoreboard tags.

In 1.8, create an objective using

scoreboard objectives add bottleInOne dummy

Create a fill clock and run the following two commands

scoreboard players set @a[score_bottleInOne_min=1] bottleInOne 0
scoreboard players set @a[score_bottleInOne=0] bottleInOne 1 {SelectedItemSlot:0,SelectedItem:{id:minecraft:glass_bottle}}

If the team part is necessary, you can now add @a[score_bottleInOne_min=1] to the team, and remove @a[score_bottleInOne=0] from it, using the same fill clock. You could also use these target selectors directly for the /effect and /replaceitem commands, if the team part was just your idea of selecting that player for the effect.

In 1.9, you can omit the scoreboard objective in favor of tags. Using a Repeat/Chain command block chain, run

scoreboard players tag @a[tag=bottleInOne] remove bottleInOne
scoreboard players tag @a[tag=!bottleInOne] add bottleInOne {SelectedItemSlot:0,SelectedItem:{id:minecraft:glass_bottle}}

In the following commands, you can then use @a[tag=bottleInOne] or @a[tag=!bottleInOne] to target players with and without the bottle selected in slot 0, respectively.