Minecraft – Modded Minecraft Uses Almost All RAM

I am using Ubuntu too here are a couple of questions and suggestions ...

1) Are you using the open JDK? If you install sun's JDK and run the command sudo update-alternatives --config java and select Sun's JDK.

2) What speed is your processor and how many cores and how much RAM do you have? You might to specficy some command parameters when you run minecraft. Try giving minecraft more RAM by using the -Xmx and -Xms to give Minecraft more RAM at startup. You can read more about that here. For example the command I use is java -Xmx5000m -Xms5000m -jar minecraft.jar

3) Try turning down your video settings. Turn down the render distance and switch the graphics settings to Fast.

I see a number of possible problems that may help your server run with less ram.

PlayerViewDistance

On many servers, including my own, the ViewDistance parameter in your server properties file may be too high. Decreasing this value will cause less chunks to be loaded for the players, but if you set it too low, ie below 3, then you won't be able to see far enough to have a good time playing.

This is easily fixed by decreasing the "view-distance" parameter in the server.properties file.

Command line args

I see a small problem with the one you pasted in your question:

Java -Xms512M -X3584M -jar minecraft_server.jar

I could not find a java argument that is simply 'X', although I assume that you are trying to set the upper memory limit to 3.5GB. Here is what that should look like:

Java -Xms512M -Xmx3584M -jar minecraft_server.jar

Or an even simpler version:

Java -Xms512M -Xmx3.5G -jar minecraft_server.jar

If you're willing to experiment, lower the maximum limit by 512mb every restart, and measure how it performs. In some odd cases, you may see a decrease in actual used memory.

Use Java 7

From my past experience, running on an Ubuntu server, running a Craftbukkit server under Java7 instead of Java6 used a noticeably lesser amount of RAM.

Switch to Bukkit

Although this may be an unwanted option for you, it definitely works. With Craftbukit there is a multitude of Plugins available for download on http://bukkit.org/ that help to increase server performance and lower resource usage.