website/public/posts/_Putting hungry minecraft servers to sleep/post.md

2 Minecraft Rust Async Networking Proxy CRIU \title

\toc

Minecraft servers are HUNGRY

They hunger for your ram and your cpu. This makes it either expensive or laggy to try and host multiple servers at once.

This was something I encountered when my friend built a homeserver out of spare computer parts that was barely powerful enough to run a minecraft server.

The problem was that soon multiple people wanted a minecraft server hosted by him (which included us wanting to play modded minecraft).

It was a hassle to ssh into the server and start and stop the various servers depending on who wanted to play, especially since a lot of people only played very rarily.

I remembered that I'd seen a project that claimed to be able to hibernate a minecraft server if nobody was playing on it. The only issue was that it worked for a single server, and did so by starting and stopping the server process.

This meant that if we wanted to join a modded server the hundreds of mods could make us wait for several minutes before we could play.

Another issue was the fact that it could only host a single server. This meant that we would have to run multiple intances of the watcher, and that each server would be assigned to an arbitrary port that would be needed when connecting.

Building a reverse proxy for minecraft

Since my friends server was accessible through a domain we thought it would be cool if instead of supplying a port you could connect to a subdomain and be sent to a specific server.

The simplest way to do this would be to create a dhcp record for each subdomain to point to a server, but that would be slow and tedious to set up for every server.

We then tried nginx, as it seemingly could magically redirect traffic to an internal port based on the subdomain. I quickly found out that this did not work for minecraft servers (who would have guessed), but after doing some research I decided on creating my own reverse proxy that spoke the minecraft protocol instead of HTTP.

The Minecraft protocol

Minecraft implements its own protocol consistent of packets. My first idea was to see if anybody had created a rust library for dealing with minecraft packets.

While some did exist, most of them where unfinished or couldn't do what I wanted. One crate was useful for seeing how an implementation of parsing packets was done, but ultimately I had to write my own parser.


Detecting the subdomain

The first step in routing the traffic was determining where to send it to.

Luckily for me, the Minecraft procotol sends the full address it is trying to connect to during the handshake. This meant that I simply had to parse the handshake and extract the subdomain from the address, see if it matches any configured server, and then redirect all traffic to the internal port that server runs on.

$ vim protocol/handshake.rs

impl Handshake {
    pub async fn new(buf: &mut BufferedReader) -> Result<Self, Error> {
        // Packets start with their type id. The handshake is id 0x00
        if !check_bytes(&[0x00], buf).await? {
            return Err(Error::Err(
                "Wrong packet type for handshake. Expected {} got {}".into(),
            ));
        }
        buf.read(1).await?; // Consume handhake byte

        let protocol_version = VarInt::read_from(buf).await?;
        let address = read_string(buf).await?;
        let port = buf.read_u16().await?;
        let next_state = VarInt::read_from(buf).await?;

        return Ok(
            Self {
                protocol_version,
                address,
                port,
                next_state,
            },
        );
    }
}

The only thing I was interrested in was the address part of the packet. This is a full string containing the text the user typed in the IP Address field in their client.

Once I had this it was as simple as splitting by . and taking the first result. This "server name" would now be used to look up the corosponding servers information in a table.

    let result = Handshake::new(&mut reader).await;
    if !result.is_ok() {
        return Ok(None);
    }

    let handshake = result.ok().unwrap();
    let split = handshake.address.split(".").collect::<Vec<&str>>();
    let server_name = *split.first().unwrap();

    if !SERVERS.lock().unwrap().contains_key(server_name) {
        println!("Unknown server {}", server_name);
        write_unknown_server_status(server_name.into(), &mut reader).await?;
        return Ok(None);
    }
    let mut server_lock = SERVERS.lock().unwrap();
    let server = server_lock.get_mut(server_name).unwrap();
    status = server.status;
    port = server.port;

show images of hibernation and starting

 

I've talked a bunch about this hibernation, but how does it actually work?

cat /proc/$(ps aux | grep server.jar) > hibernation.txt

Finally being able to connect to the server it was time for the next item on the list.

✨**Hibernation**✨

Now, instead of closing the server and restarting it when a player joined I wanted to hibernate the server and unhibernate it when someone joined.

I had a feeling this would be possible as this is basically how hibernation works for your system; it saves the memory to disk and loads it into memory again upon boot.

After a bit of searching I found CRIU, a program capable of suspending another program to disk.

Not only does it save the memory, it also saves all the file descriptors the program was using (and fails with a million cryptic error messages if any of the files where changed during hibernation)

There was a Rust crate for CRIU, however it was poorly documented and didnt support all the command line arguments I needed, so I resorted to calling the binary.

This led to the program halting until the unhibernated server closed again, but it was fixed with a simple fork.

The only downside to this approach is that CRIU requires root access in order to read the memory and file descriptors of another process.
Running your minecraft server as root is probably not the smartest idea...

My solution was to not care about it and hope for the best :D

Why aren't my chunks loading?

Just as I thought I was finally done I encountered a problem. The chunks where loading really slowly. It felt like we we're back to hosting on aternos (nothing against aternos, the free servers are just a little slow).

image of chunks not loading

I couldn't figure out why it was so slow. I thought it was because all the traffic had to flow through my program, but the sockets are connected on the kernel level.

After hours of debugging I found the solution. stream.set_nodelay(true)

...

**All this time the problem was as simple as saying ** if (slow) { dont(); }!

To say I was pissed was an understatement, but at the same time I was glad it was finally working.

The reason this fixed it was because it disabled Nagle's algorithm, which bunches packets together in order to (hopefully) speed up the sending of lots of smaller packets as one big packet.

For some reason this absolutely destroyed the chunk loading speed, and disabling it led to finally having the perfect Minecraft reverse proxy hibernation system thing.