Supplementary (or non-defining, or non-restrictive) relative clauses are by convention set off by commas and integrated (or defining, or restrictive) relative clauses are not. The clause in the example, 'which I am trying to configure', is a supplementary relative clause, so the comma is appropriate.
You can generally distinguish a supplementary relative clause from an integrated one by seeing whether or not you can detach the relative clause and still be left with a meaningful sentence. With the example, you can say simply 'I have just got a new Ubuntu LAMP VPS server' without the clause 'which I am trying to configure', which is a piece of bonus information. However, if the example had been 'I'm not getting anywhere with that new Ubuntu LAMP VPS server which I am trying to configure' the relative clause is essential in identifying exactly which new Ubuntu LAMP VPS server is being discussed. 'I'm not getting anywhere with that new Ubuntu LAMP VPS server' on its own doesn't tell us.
English Teachers are like MS Word's grammar checker. They should be used but not trusted.
You are correct that the first statement needs no comma before rather. Here, the expression rather than [to] a restaurant is essential information for understanding the statement. It also describes or explains grocery store, again indicating it's importance.
Commas separate parts of sentences. Because you don't want to separate the final phrase in the first example, you don't use a comma.
In the second example, rather than going out to a restaurant, you still don't need a comma before rather. Here, the expression also provides necessary information, as in the first case. The phrase is not parenthetical, and it certainly isn't an appositive.
However, you will need to follow the expression with a comma because it is serving as an introductory dependent phrase, as in "Rather than going to the store, we went to the restaurant."
But why no comma before rather in the second example? The word that turns the following expression into a noun phrase, here to be used as the direct object of decided. If we place a comma after that, we separate the expression from the noun phrase, which is not correct because it needs to be part of the noun phrase.
Bottom Line:
First example: We decided to go to the grocery store rather than to a restaurant.
Second example: We decided that rather than going out to a restaurant, we would go to the grocery store.
You might pick up a copy of Zen Comma, which has a much more thorough discussion of comma uses.
On a side note: You seem to be confused about appositive phrases. Although appositives don't provide essential information, not every non-essential phrase is an appositive. I think you mean parenthetical expressions, of which appositives are one type, or non-restrictive phrases and clauses.
Example appositive: "This toy, a 1992 Barbie doll, is a family treasure." A 1992 Barbie doll is an appositive.
Example non-restrictive clause: "Take away my life, which is as precious to me, but don't take my dignity." Which is precious to me is the non-restrictive clause.
Best Answer
Use a comma before non-restrictive clauses, and don't use a comma before restrictive clauses. See this webpage.
How do you know whether a clause is restrictive or non-restrictive? It's restrictive if the information is essential, and limits the scope of the thing it's modifying. It's non-restrictive if the information is non-essential. For both of your sentences, the clause can be analyzed as either restrictive or non-restrictive, so in cases like this, the comma is optional.