[SalesForce] How to handle system limit exception

You are looking to test this catch block, right?

catch(InnovationService.InnovationServiceException ex) {
    ApexPages.addmessage(new ApexPages.message(ApexPages.severity.ERROR, ex.getMessage()));
}

And you are expecting that that exception to be thrown by this block, right?

try {
    insert recordsToInsert.values();
} catch(InnovationServiceException ex) {
    throw new InnovationServiceException(ex);           
}   

If my assumption is right, then I don't think you will ever be generating that error. insert <sobjects> will not throw your custom exception but System.DmlExeption (and other standard exceptions).

One way you can generate that error is to try to insert a Vote object without some required values not specified, catch System.DmlException and convert to InnovationServiceException, may be like this.

try {
    insert recordsToInsert.values();
} catch(System.DmlException ex) {
    throw new InnovationServiceException(ex);           
}   

Broadly speaking, you'll need to handle the error response in the callback of your action and then bubble it up to the UI somehow.

Here is a very rudimentary example of this for Salesforce1 Mobile:

action.setCallback(this, function(resp) {
    attachId = resp.getReturnValue();

    var showToast = $A.get('e.force:showToast');
    showToast.setParams(
        {
            'title': 'Progress: ',
            'message': resp.state
        }
    );
    showToast.fire();
}, 'ALL');

The trick is that the parameter in the anonymous callback function has the state attribute. In my example above, I'm just using it as the message I send to the user. It will either say SUCCESS or ERROR.

Not very useful.

More useful might be to use it in an if and have some code execute when an error occurs so you can write to the console, and have a more intelligent message for the user.

if (resp.state === 'ERROR') {
  //...do something here
}

In my case, I'm communicating to the user with the showToast event. This event will pop down a toast message in the Salesforce1 mobile app. This event also has a duration attribute that can be used to make the message last longer.

If you are not using S1 mobile, or don't want the notification to auto-close, there is a ui:message component that is in the docs. I've not played with it yet, but it has a promising attribute called closable in it.

One last thing...you might want to have a different callback for error versus success. Ostensibly you would do this if you want to encapsulate your error logic in one callback (in a helper, for instance) that you use in several different places in your component.

Happily, the JS api supports this in the setCallback function.

To register a callback for success-only responses:

action.setCallback(this, function(resp){
  //...my success callback code goes here. 

}, 'SUCCESS');

To register a callback for error-only conditions:

action.setCallback(this, function(resp){
  //...my error callback code goes here. 

}, 'ERROR');

These features are documented in the JS Api docs and are worth spending some time just playing with to understand better.

Long story short, you can't just do this in your Apex code. You will need to do some work in your Lightning Component Javascript in order that you communicate error conditions to users.