Tip: If you want to spare more memory, you may set the -Xms parameter even lower, say:
java -Xms32M -Xmx512M -jar minecraft_server.jar nogui
The parameter controls how much memory is reserved on startup. Your server will start with 32MB RAM and whenever it needs more memory it will allocate some until it reaches the maximum of 512M. However, this will result in a little slowdown whenever the allocation is done.
Source: http://www.minecraftwiki.net/wiki/Tutorials/Setting_up_a_server#Less_than_1GB_free_RAM
Also you can check this website : http://www.canihostaminecraftserver.com/
With this you can have a small idea on how many players you can handle :D
TL;DR It may or may not be fine to play and host a dedicated server on 8GB of ram. It also depends on wheter you have a GPU or an APU. As well as the power of your APU if you dont have a GPU. You can try it out but make sure you dont use a HD Texture Pack. And dont use any shaders. You might have to give up on fancy quality settings as well.
This is to clarify how minecraft server and client works.
RAM
As for the Server, ram is primarily used to store chunk information. Having few players does not mean low chunk information especially when you are playing a tech pack (A mod pack full of mostly tech based mods). There lies the existence of chunk loaders and quarrys that load chunks. In any case, the server should not use more than 2-3gb of ram and that is the case for my server. (SolitaryCraft, 250+ mods, 3-4 players)
As for the client. It also stores chunk information to render to the user. It is also used to store texture information for the renderer to show to the client. I am unsure if the renderer uses VRAM or RAM to store its post rendered stuff. Anyways, by my guess the client shouldnt exceed 3-4 gb of ram as well. UNLESS you use HD texture packs. A 64x texturepack can easily push the ram requirement to about 5-6gb
Thus if you are hosting and playing on the same machine, and using a dedicated server instead of "Publish to LAN" you will use more ram than playing single player. Past experience tells me that hosting a dedicated server and playing on the same machine with Less than 8 GB of ram is a bad idea for tech packs. 8 GB should be fine depending on the mod list and the prerequsite that you dont use HD texture packs.
CPU
The CPU in both cases is used to process the ticks on the server. To briefly explain ticks, it is the passage of time on the server. Each tick should take less than 50 ms to process. This is due to the fact that there 20 ticks make up a second. Having low processing power slows the tick calculation down which results in a slow passage of time. Every machine, every chunk loaded with things going on is gonna eat into your CPU requirements. I would suggest you install a lag profiler to identify if you are running low on tick rate.
The mod suggested is OPIS by ProfMobius. You would also need a dependency called MobiusCore. THe mod can be installed on the server and the client. It does not need every client to install it to connect to the server. You as the host can install it and monitor which machine is lagging the server the most and do appropriate actions to work around it.
GPU
This is a part where it gets abit complicated. I will go into 2 possible cases.
External GPU.
If you have a separate GPU it would help remove load on the ram while playing minecraft. Normally a GPU have its own ram to store information used while rendering. Minecraft should theoretically send its rendering to the GPU and free up your computer RAM of the rendering tasks.
Integrated GPU.
If you do not have a GPU and uses your CPU to do the rendering, this might pose an issue. If i recall correctly, playing minecraft on an APU takes about 3-4 GB of ram just for the client alone. As there is no dedicated ram for rendering, all rendering tasks is stored at the same location on the onboard RAM. This might also slow down tick rate as the CPU have to compensate for rendering as well.
I am abit incoherent at this last part as im not sure how to explain. You may clarify with me in the comments section.
EDIT: Adding some observations i have while running a SolitaryCraft Client with Optifine. It seems that somehow, it may be caused by Optifine or not, It have horrible garbage collection. ie. You might need to restart the client to flush your ram. If i keep my client running for like 4-5 hours, the ram usage is > 7GB on the client. Restarting the client lowers it to 4 GB again. I am using a x64 texture pack.
Best Answer
The error you are receiving means that the computer can not keep up with the processing required for the game environment. This doesn't necessarily mean that the CPU is too slow to run the Minecraft server, just that there are too many things running each tick that it can not keep up with. Minecraft has 20 ticks per second, and game events are processed each tick.
Entities, animals, players, chunk generation, flowing liquid, block entities, command blocks, redstone machines, hoppers, minecarts... all these things can contribute to the processing per tick. When there is too much, the server becomes overloaded and says "Can't keep up".
If you have been doing a lot of these things very quickly in a new server and new world, then this could be the source of your problem. But if you have a new server and new world and you haven't done anything yet, then it can mean that the Minecraft server is just not efficient enough for your computer. The Minecraft server has been noted to have become harder to process over the years.
Note, you should store and run the Minecraft server data on an SSD drive. HDDs are too slow to produce good results.
More Efficient Minecraft Server
But, you are not limited to the native Minecraft server. You can use Spigot or, better yet, Paper Spigot. These are open-source server implementations that make a more efficient Minecraft server. They also allow for plugins to be installed, but I'm not going to get into that in this response. I recommend downloading Paper Spigot.
Paper Spigot runs better than the native Minecraft server, but there are also basic optimization settings you can implement that have been documented here.
Notes On Optimization
Note that with the optimization guide by celebrimbor, the final item talks about your start file java flags. You don't need a lot of ram dedicated to the server when running a simple server. 3-4gb should be fine for you, and even a couple of friends.
The optimization guide also talks about pre-generating the map. If you are playing solo, and even with a couple friends, you may not need to pregenerate the map(s). Be aware that if you did use a plugin to help do this, the world size will get big. Depending on how much you generate, this could take several GB on your storage drive. If you do generate, the plugin listed, WorldBorder, is out of date. Use Chunky or Chunkmaster instead.
The chunks around the overworld spawn point are called spawn chunks. Normally a chunk is only loaded when you are near them, and thus things inside the chunk begin processing like hoppers, mob movement... But spawn chunks remain loaded and processing wherever you are on the server. This means that it may be best not to do a lot of tick heavy things in spawn chunks, like having a giant animal farm, hopper system, redstone machine. One thing spawn chunks are good for is if you really wanted to run command blocks that affect the entire server all the time. Aside from that use, I recommend conducting your adventures away from the spawn chunks.
Lastly, Paper Spigot (and spigot) have command tools that can help to analyze what is causing lag that results in the "Can't keep up!" error. These are called Timings. If you still have problems that can not be resolved with a more efficient server and optimization, learn how to use Timings reports. They can be hard to understand, but the Spigot and Paper Spigot communities can help you read what the reports mean. There are also alternative plugins that do this.
I hope this helps get you on a path to understanding what is going on. I know it is a lot to take in right away.