Grammar – ‘Rather Than Do’, ‘Rather Than Doing’ or ‘Rather Than to Do’?

The "ing" gerunds are fine. http://www.eslcafe.com/grammar/verbs_other_verbs08.html

Also, prefer has a number of rules to follow. http://www.learnenglish-online.com/grammar/ratherprefer.html

Prefer and Rather are not interchangeable. Would Rather is followed by a base verb and Would Prefer is followed by an infinitive.

"I prefer ride to walk" is not correct.

Your intuition is right - "prefer to study at X to to study at Y" is no good.

Your suggestion isn't bad, but it does sound a bit unnatural. "prefer ... rather than" is sort of strange considering that "rather" itself often means "prefer".

I'd rather study in a coffee shop than study in the library.

This usually expresses a preference for a specific event though rather than your general preferences. For alternatives so you can use the infinitive in the first clause and keep the expression general, use direct expressions of comparison (all of which work with both the infinitive and the gerund):

I prefer to study/studying at a coffee shop as opposed to studying in the library.

I prefer to study/studying at a coffee shop compared with/to studying in the library.

I prefer to study/studying at a coffee shop versus studying in the library. (uncommon in writing; fairly common in speech)