You'll need to "translate" the glass bottle data tag into something that works with target selectors (@a and so on). This can be achieved most easily using dummy scoreboard values or, in 1.9, scoreboard tags.
In 1.8, create an objective using
scoreboard objectives add bottleInOne dummy
Create a fill clock and run the following two commands
scoreboard players set @a[score_bottleInOne_min=1] bottleInOne 0
scoreboard players set @a[score_bottleInOne=0] bottleInOne 1 {SelectedItemSlot:0,SelectedItem:{id:minecraft:glass_bottle}}
If the team part is necessary, you can now add @a[score_bottleInOne_min=1] to the team, and remove @a[score_bottleInOne=0] from it, using the same fill clock. You could also use these target selectors directly for the /effect and /replaceitem commands, if the team part was just your idea of selecting that player for the effect.
In 1.9, you can omit the scoreboard objective in favor of tags. Using a Repeat/Chain command block chain, run
scoreboard players tag @a[tag=bottleInOne] remove bottleInOne
scoreboard players tag @a[tag=!bottleInOne] add bottleInOne {SelectedItemSlot:0,SelectedItem:{id:minecraft:glass_bottle}}
In the following commands, you can then use @a[tag=bottleInOne] or @a[tag=!bottleInOne] to target players with and without the bottle selected in slot 0, respectively.
the heads show up as a 3d block in the hotbar while items don't. try making a recource pack where you change the texture of items that you dont use to those numbers, or anything you want. then use the items instead of blocks. items should display just fine as they are displayed in 2d.
Best Answer
I figured it out with MBraedley's help. First I have to tag the player with "redstone".
Then I need to use the execute with store to see how many redstones I have...
Then I have to use replaceitem to get rid of all the redstones in my offhand...
Then I have to subtract my second score by my first score, and the resulting negative number will by how many redstones I got rid of times -1.
Then I just remove the redstone tag. I had to use a tag because there are commands that come after when I clear the redstones from my offhand so I couldn't detect the player using -nbt=...].
Thanks again MBraedley!
EDIT: Just realized that last scoreboard command only finds the correct number of used redstones for some equations like 1 - 2, but not for some like 1 - 4, which would suggest I used 3. The solution is to deplete both scores of the player until one score is 0, and the other will be the answer times 1. To do that, remove the player of score check1 64 if his score is 64 or above (start at 64 because one cannot have more than 64 items in his hotbar), then remove 32 if he has above 32, and so on, and for each removal you remove the same amount for check2.