[SalesForce] How to change colors to lightning:buttonStateful component

I want to implement a button with states – lightning:buttonStateful seems to be the right standard component for that.

The problem is – I want to put a few buttons – more than 5 and to have for each button a different color.

lightning:buttonStateful has the variant attribute which change colors, but it has only 3!

So I want to know is there a way to change the button colors?

I have tried to put a css class and change the background color, but the button has 2 different states – selected, unselected and hover, and I want to change all states.

component markup

<aura:attribute name="buttonstate" type="Boolean" default="false"/>
<lightning:buttonStateful
    labelWhenOff="Follow"
    labelWhenOn="Following"
    labelWhenHover="Unfollow"
    iconNameWhenOff="utility:add"
    iconNameWhenOn="utility:check"
    iconNameWhenHover="utility:close"
    variant="success"
    state="{! v.buttonstate }"
    onclick="{! c.handleClick }"
    class="ColorBlue"
/>

component css

.THIS .ColorBlue {
    background-color: rgb(0, 95, 178);
}