[SalesForce] Time limit exceeded and ‘Response size exceeded 30MB organization limit’ and Semi-join subselects not allowed

I've done jqGrid before in Apex with @RemoteActions and it's definitely the way to go. I think with Ajax toolkit and SOAP, you'll have too much XML bloat compared to JSON for large sets of records.

As for loading all 78000 records, I would say you should stick to the limits of loading 2000 records at a time and than breaking those into further subsets of 100 records / page on the client side.

To pagination with large record sets I would recommend using the standard set controller for pagination. Just make sure to include an order clause to keep the records returning in the same order.

Than load the next 2000 records in each subsequent call. And reduce the number of records you return from the server to speed things up in between 'set loads' as you get to the end of each set and need more pages.

You can also count the number of records and return it in a complex object from the @RemoteAction to keep your pagination accurate:

public class returnType{
  List<sObject> objects {get; set;}
  Integer totalRecords {get; set;}
  Integer nextPage {get; set;}
  Integer lastPage {get; set;}
}

UPDATE

Ok using the SSC to maintain pagination (at least this is the way I would do it with jqGrid:

@RemoteAction
global static returnType pagination(Integer pageNumber){
   ApexPages.StandardSetController ssc = new ApexPages.StandardSetController(Database.getQueryLocator([SELECT Id FROM sObject ORDER BY Id LIMIT 50000]);
   ssc.setPageNumber(pageNumber);
   ssc.Pagesize(2000);

   returnType retVal = new returnType();
   retVal.objects = ssc.getRecords();
   retVal.nextPage = pageNumber + 1;
   retVal.previousPage = pageNumber - 1;

   return retVal;
}

I don't have my original code in front of me, but generally, the SSC should be re-instantiated with each call. As for getting past 50k records, and to your total set, the query locator should continue to the end of the set reguardless of size taking the larger records towards the end of the set as you get to the end of [0..n-1] pages.

In practice I would use an ORDER BY clause to ensure your records are consistently returned in the same order to ensure that your getting all results.

As you state, normally you need to do the subquery separately and store the IDs in a set you can then use in the main query:

set<Id> AssetSet = new set<Id>();

for(Customer_Asset__c asset : [Select Asset__c from Customer_Asset__c where isDeleted = false and Company__c =: CompanyId and (Asset__r.Title__c like : SearchText or Asset__r.Name like : SearchText)])
{
  AssetSet.add(asset.Asset__c);
} 

List<AggregateResult> lstaggResult = [Select count(id) RecCount  FROM Asset__c where  Id IN 
: AssetSet or (Group__c IN : SearchgroupassetidSet and ((Title__c like : SearchText) or (Name like : SearchText)))];

However, if you're sure you can't do this then it would suggest that you may need to use batch apex to do the first (sub) query, leveraging database.Stateful to help with managing the data.

Alternatively look for a way to leverage other fields to reduce the scope of the query.

Important note

In your code you've got a lot of hard coded IDs, e.g. Id CompanyId = '001Z000000CF2PtIAL'; and SearchgroupassetidSet.add('a7aZ00000004CHhIAM');

You should be aware that this is considered bad practice, because once you try to move this code to a different org, say a production environment, these IDs will no longer be valid and as such your code will fail to run.