[SalesForce] Lightning Service Console : Issue facing on open a new subtab in Lightning service console

Did you try with this:

Service console integration toolkit has a method named setTabTitle

Ex:

<script type="text/javascript">
        sforce.console.getFocusedPrimaryTabId(function(result) {
               // Result.id is the id of tab
               sforce.console.setTabTitle('Change Title', result.id);
        });
</script>

Reference: http://www.salesforce.com/us/developer/docs/api_console/index.htm

Old thread, but posting this for others unaware of the other solutions in this use case, and some general debugging tips for this type of issue.

The method used in the question should work, but is missing some parameters like the callback function for success and the override subtabid (or null if not overriding an existing subtab). This page is also using API 20 of the console kit which is fairly old at this point (not as much at original post, but still old).

Also, you can do some debugging in the openSubtab callback with console log to double check that the result coming to the openSubtab callback is containing a primary tab id. If that Id is null, I believe you'll see a brand new primary tab opened up (as you see here).

Example with openSubtab with all necessary params, updated console integration kit (API 36), and a sample console.log statement:

<apex:includeScript value="/support/console/36.0/integration.js"/>
<script type="text/javascript">

    function testOpenSubtab() {
        //First find the ID of the primary tab to put the new subtab in
        sforce.console.getEnclosingPrimaryTabId(openSubtab);
    }

    var openSubtab = function openSubtab(result){
        var primaryTabId = result.id;

        // good place to debug to see if we received a primary tab id
        if (console && console.log) {
            console.log(result);
        }

        sforce.console.openSubtab(primaryTabId , 'https://na1.salesforce.com/apex/service_cloud_console_flow_1', true, 
            'Add New Service',null,openSuccess);
    }
    var openSuccess = function openSuccess(result) {
        //Report whether we succeeded in opening the subtab
        if (result.success == true) {
            alert('subtab successfully opened');
        } else {
            alert('subtab cannot be opened');
        }
    };

</script>
<button type="button" onclick="testOpenSubtab();">Test Subtab</button>

Additionally there are two other ways to change tabs in VF with the srcUp and srcSelf functions. srcUp is used to open a subtab and srcSelf is used to change the current tabs location. These two methods are very useful for javascript custom buttons as well.

You can find a lengthy example from one of the PMs of service cloud console Gautam Vasudev in his descriptive post, on the idea exchange for some reason...

https://success.salesforce.com/ideaView?id=08730000000YKxYAAW

Because the srcUp function is only available in the context of console you'll want to check for whether the function exists to determine if you're in that context, otherwise handle appropriately for your VF being hosted externally (call window.open for example).

Here is a brief example:

var url = 'http://someurl.com'; // or {!$Page.somevf}, or urlfor($action.... etc
if (typeof(srcUp) == 'function') {
    srcUp(url);
} else {    
    window.open(url); // open new browser tab if not in console context
}