JavaScript Remoting does not instantiate the controller or load the view state. For this reason, the methods are declared as static, because there is no instance associated with these calls. There is a buffering mechanism that causes calls that are called close together to be buffered together and submitted at once; this is a default behavior that can be modified per the JavaScript Remoting for Apex Controllers page. Disabling buffering may increase performance, although surely not as much as 2000 ms worth of delay that you're observing. My typical response time when using remoting with buffering is around 19 ms in demo classes I've written.
The server may choose to delay these requests until resources are available, just like any other request. They are not given any special priority queue, nor are they processed any differently than a normal request. They are generally faster, though, because they do not carry their view state with them, which tends to make them faster and more responsive. However, like all software systems, there can sometimes be unforeseen contention. Submitting a case with support may be able to help isolate this problem.
Classes are, as in Java, compiled to bytecode when they are successfully deployed. This means that your code should load in linear time for any given execution cycle. The size of the class and all the classes upon which it depends can play a factor. More importantly, the first run after a class changes may be delayed if there are inter-class relationships that haven't been compiled (e.g. a change to a utility class will cause all classes that use that utility class to have to re-compile on next execution). This delay is usually minor, and should only occur once per code change.
The system also depends on a cache, so loading a controller for the first time in a while (however long the cache keeps recently used bytecode) may result in a loading penalty, but again this is usually far less than your observation of 1800 ms. Generally this delay is less than 100 ms on a busy day, and I can't think of any documented reason why it would take as long as you've observed unless there's some other issue going on, usually internal in nature.
It might be worth the effort to submit a case to Developer Support and see if they can help isolate the issue. Occasionally, there are definitely hiccups in the system, but the engineers usually keep tabs on this and submit hotfixes for situations like this. You might try reducing the complexity of your code and/or splitting your remote functions across multiple classes (this is a great use case for extensions).
Best Answer
This depends on the current state of buffering. By default, methods called close together are buffered together and sent all at once. The documentation hints that the limit may be per each buffered transaction (but it isn't clear, so I'd have to test). You can disable buffering as described in JavaScript Remoting for Apex Controllers in order to prevent buffering. This may greatly increase the time it takes for the page to load data, but will reduce errors related to maximum response sizes and timeouts. The rationale behind this answer is that the timeout is controlled on the client side, not server side, meaning that each buffered request would have a single client-side timeout aggregated together.