[SalesForce] How to display more than 5000 records in pageblock table with out pagination..

I suspect using the standard controller on a custom setting, or perhaps even the recordSetVar is causing the issue.

My suggestion would be to create a simple visualforce controller and either query for the custom setting record or use the getAll() method, and then iterate over that list in your Visualforce page.

Controller:

public class MySettingsCtrl {
    public List<MySettings__c> settings {get; private set;}

    public MySettingsCtrl(){
        //You're probably going to have to query for the Custom Setting records if you want 
        //all hierarchy instances.  Otherwise, you could use the cached records with getAll() or get Instance
        //More info on custom setting methods here: http://www.salesforce.com/us/developer/docs/apexcode/Content/apex_methods_system_custom_settings.htm
        settings = [Select SetupOwnerId,Priority_1__c,Priority_2__c,Priority_3__c From UserQueueSettings__c];
    }
}

And your visualforce would be updated to look like:

<apex:page controller="MySettingsCtrl" sidebar="false">
<apex:form >
    <apex:pageBlock >
        <apex:pageMessages />
        <apex:pageBlockButtons >
            <apex:commandButton value="Save" action="{!save}"/>
        </apex:pageBlockButtons>
        <!--<apex:repeat value="{!settings}" var="block" rows="1000">-->
            <apex:datatable value="{!settings}" var="ES" id="memberTable" rows="1000">


                <apex:column headerValue="Name">
                    <apex:inputField value="{!ES.SetupOwnerId}"/>

<!--                    <apex:facet name="header">TEST</apex:facet>
                <apex:inputField value="{!ES.Priority_1__c}" 
                                 required="false"/>

                    <apex:outputField value="{!ES.Priority_1__c}" />
-->                 

                </apex:column>
                <apex:column headerValue="Priority 1">
                    <apex:inputField value="{!ES.Priority_1__c}"/>
                </apex:column>
                <apex:column headerValue="Priority 2">
                    <apex:inputField value="{!ES.Priority_2__c}"/>
                </apex:column>
                <apex:column headerValue="Priority 3">
                    <apex:inputField value="{!ES.Priority_3__c}"/>
                </apex:column>

            </apex:datatable>
        <!--</apex:repeat>-->
    </apex:pageblock>  
    </apex:form>
</apex:page>

You can either use client-side or server-side pagination. Either way, you won't be able to use standard tools like StandardSetController or OFFSET.

Server-side, you need to use the @ReadOnly annotation to enable the 1,000,000 row limit, which might look like this:

@RemoteAction @ReadOnly public static SObject[] getRecords(Integer page) {
SObject[] results = new SObject[0];
for(SObject[] records: [SELECT ... FROM SObject WHERE ... LIMIT :page*200]) {
    results = records;
}
return results;
}

From there, you just repeatedly call the value until you get all of your results:

var buffer = []
function getPage(i) {
myController.getPage(i, function(event, data) {
     if(data.length) {
         data.forEach(function(v) {buffer.push(v)})
         getPage(i+1)
     }
}
}

Client-side, it'd look more like this example from the docs:

var result = sforce.connection.query("select id, name from account"); 
var it = new sforce.QueryResultIterator(result); 
var buffer = [];
while (it.hasNext()) { 
var account = it.next(); 
buffer.push(account);
}