[SalesForce] Collection size 1,768 exceeds maximum size of 1,000.

The collection limit of 1,000 is applied per-collection by the visualforce system when rendering. In effect it means that the apex:repeat tag doesn't support more than 1,000 iterations.

However you can nest collections in apex, List<List<sObject>>, and ensure that the inner lists contain no more than 1,000 elements each.

In your visualforce you can then nest repeat tags within repeat tags (assuming there List<List<sObject>> structure is accessible as a variable named "uglyHack"):

<apex:repeat value="{!uglyHack}" var="subList">
  <apex:repeat value="{!subList}" var="record">
    {!record[someField]}
  </apex:repeat>
</apex:repeat>

Ah the joys of CollectionLimiter.

With Apex repeat we are able to export more than 10 K however if records are more than 50K it will give SOQL error

The problem is that the action function is querying data and piling the data onto the list each time it is called. This is an inherent risk factor with queries inside of getter or setter methods. It would be more appropriate to simply expose the list of records to the user directly. I've wrote an essentially same version (virtually cut-and-copy) of what you've done with the class, included below.

Revision

public with sharing class Checkbox_Class {
    // The list of assets we're viewing
    public AssetsWrapper[] assets { get; set; }
    // The total count of assets selected
    public Integer selectedAssets { get; set; }

    // Initialize our data here
    public Checkbox_Class() {
        assets = new AssetWrapper[0];
        selectedAssets = null;
        refreshAssetList();
    }

    // Get the currently selected assets and update counter
    Asset[] getSelectedAssetList() {
        Asset[] results = new Asset[0];
        for(AssetWrapper item: assets) {
            if(item.selected) {
                results.add(item.acc);
            }
        }
        selectedAssets = results.isEmpty()? null: results.size();
        return results;
    }

    // Assumption: We'll be returning a page reference somewhere later.
    public PageReference getSelected() {
        getSelectedAssetList();
        return null;
    }

    // Query the database for a list of assets meeting the criteria
    public void refreshAssetList() {
        Map<Id, Asset> selectedAssets = new Map<Id, Asset>(getSelectedAssetList());
        assets.clear();
        for(Asset record: [SELECT   Id, SerialNumber, Name, Retired__c, Returned__c, Quantity, RMAed__c
                           FROM     Asset
                           WHERE    Retired__c = false AND Returned__c = false
                           ORDER BY CreatedDate DESC    
                           LIMIT    1000]) {
            assets.add(new AssetsWrapper(
                selectedAssets.containsKey(record.Id)?selectedAssets.get(record.Id):record,
                selectedAssets.containsKey(record.Id)
            ));
        }
    }

    // Wrapper class
    public class AssetsWrapper {
        public Asset acc { get; set; }
        public Boolean selected { get; set; }
        public AssetsWrapper(Asset record, Boolean isChecked) {
            acc = record;
            selected = isChecked;
        }
    }
}

Helpful Tips

Getters and setters may be called more than once per page transaction, and in particular, the getAssets function originally posted kept piling on records to the master list 100 at a time (duplicates). This is why the error occurred at exactly 1100 records. Instead, initialize lists only once, and refer to them as necessary.

The refresh function provided here could be linked to a button to refresh the list from scratch (e.g. in the event new assets were added), while still retaining the old checkboxes, if any. This is generally useful especially if the users need updated information.

In general, avoid returning a pagereference if the only possible value is null. In longer functions, this helps visually identify the difference between an action function that will redirect to a new page versus one that will never redirect to a new page. I left that function as is under the assumption that "Create Quote" would probably go to a new page at some point.

Don't use member variables when local variables and parameters will suffice. Each member variable consumes memory, which in turn uses view state (unless it is transient, which is a useful, but often confusing language feature). You don't want your view state getting any larger than necessary, because it slows down loading times and can eventually run afoul of the governor limits.

You can make the page "Read Only" if you want to increase your limits from 1000 items in a list to 10000. This only works if you plan on not creating any data on this page.

You could have used a StandardSetController to emulate all of this functionality, with a maximum query of 2000 records deep, and offer pagination (first, next, last, previous pages), and checkbox capabilities.

Try to avoid calling a variable "x", unless it's in a local loop somewhere that won't be used outside that area. A member variable of "x" is always a bad idea, because it's not apparent what "x" is. Is it the X in an X/Y coordinate? It is a location on the screen? Is it just a simple counter? Does it contain the answer to life, the universe, and everything? Obviously, in this code, it was easy to figure out, but a 10000 line class would be a different story, or even worse, a class that is later subclassed and that subclass uses X to communicate to functions in the main class.