JavaScript Remoting does not instantiate the controller or load the view state. For this reason, the methods are declared as static, because there is no instance associated with these calls. There is a buffering mechanism that causes calls that are called close together to be buffered together and submitted at once; this is a default behavior that can be modified per the JavaScript Remoting for Apex Controllers page. Disabling buffering may increase performance, although surely not as much as 2000 ms worth of delay that you're observing. My typical response time when using remoting with buffering is around 19 ms in demo classes I've written.
The server may choose to delay these requests until resources are available, just like any other request. They are not given any special priority queue, nor are they processed any differently than a normal request. They are generally faster, though, because they do not carry their view state with them, which tends to make them faster and more responsive. However, like all software systems, there can sometimes be unforeseen contention. Submitting a case with support may be able to help isolate this problem.
Classes are, as in Java, compiled to bytecode when they are successfully deployed. This means that your code should load in linear time for any given execution cycle. The size of the class and all the classes upon which it depends can play a factor. More importantly, the first run after a class changes may be delayed if there are inter-class relationships that haven't been compiled (e.g. a change to a utility class will cause all classes that use that utility class to have to re-compile on next execution). This delay is usually minor, and should only occur once per code change.
The system also depends on a cache, so loading a controller for the first time in a while (however long the cache keeps recently used bytecode) may result in a loading penalty, but again this is usually far less than your observation of 1800 ms. Generally this delay is less than 100 ms on a busy day, and I can't think of any documented reason why it would take as long as you've observed unless there's some other issue going on, usually internal in nature.
It might be worth the effort to submit a case to Developer Support and see if they can help isolate the issue. Occasionally, there are definitely hiccups in the system, but the engineers usually keep tabs on this and submit hotfixes for situations like this. You might try reducing the complexity of your code and/or splitting your remote functions across multiple classes (this is a great use case for extensions).
Best Answer
Since you're not required to do this, I believe people just do it because they don't know any better. They think "I'm calling this from Javascript, I better return JSON, so I need to serialize it." It's just one of many options that generate additional work, like returning
String.valueOf(123)
and sending it to a method that just doesInteger.valueOf('123')