There is a way to do this in vanilla using commands. I got the general idea from betathedata on reddit, but the commands below are my own implementation of the technique.
This method relies on the fact, that a villager without trade offers has no trade interface, but right-clicking it will still count as "talking" to him. Therefore, having an invisible villager at your location while holding a specific item will do the trick.
Scoreboard objectives
For this to work, we need three objectives set up.
scoreboard objectives add holdingItem dummy
scoreboard objectives add holdTime stat.playOneMinute
scoreboard objectives add rightClick stat.talkedToVillager
The first two are for tracking when a player has just selected a specific item, the third one is what we want to unobtrusively set to 1.
Clock commands
First, set up a fill clock large enough to run at least 8 commands. Start off with
scoreboard players set @a holdingItem 0
scoreboard players set @a holdingItem 1 {SelectedItem:{id:minecraft:iron_ingot}}
scoreboard players reset @a[score_holdingItem=0] holdTime
Replace the SelectedItem tag with whatever you like. You can add multiple of this second command if you want it to work for multiple items.
execute @a[score_holdTime=1] ~ ~ ~ summon Villager ~ ~ ~ {CustomName:"RightClicky",Offers:{Recipes:[]},NoAI:1,Silent:1,ActiveEffects:[{Id:14,Duration:9999,ShowParticles:0b},{Id:11,Amplifier:4,Duration:9999,ShowParticles:0b}]}
execute @a[score_holdTime_min=2] ~ ~ ~ tp @e[type=Villager,name=RightClicky,r=2,c=1] @a[c=1]
execute @a[score_holdingItem=0] ~ ~ ~ tp @e[type=Villager,name=RightClicky,r=2,c=1] ~ -100 ~
The first will summon a villager called RightClicky, who does not move, does not speak and has Invisibility and Resistance V (=Invulnerability), when you swap to the item(s) you specified above.
The second and third commands will either teleport that Villager to you, or into the void (killing him without particle effects showing), depending on whether you are still holding your item or not. This teleporting causes the villager to lag behind you a little, so it does not always work when you are running.
After putting in these commands, your RightClick score will go up every time you right click while holding your item. At this point, place any commands that are supposed to be triggered by the right-click. Use @a[score_RightClick_min=1] for targetting. For example:
effect @a[score_RightClick_min=1] minecraft:jump_boost 10 5
At the very end of your clock, run
scoreboard players set @a RightClick 0
to reset the system to detect another right-click.
Detecting a Left-Click should also possible. After summoning the villager, you can detect if a player hits it, as detailed in this question. Putting the two together is left as an exercise to the reader.
Best Answer
Easiest way to accomplish a "generic boost" is through attributes, which can be set to only apply on a specific slot. This is a generator that allows adding attributes to items for specific slots. Here's an example command that gives you 10 extra health (5 hearts) when held only in the offhand:
There's this wiki page if you want to learn more about all the things attributes can do, and what do different operations are for.
Though there's a few things attributes do better than potions effects, they're a bit limited and may not be able to do what you want.
Alternatively, you could do this by detecting when the player is holding the item in their offhand slot (-106)
You can then do anything you want to the player, such as: