website/public/posts/_Putting hungry minecraft servers to sleep/post.md

2
Minecraft Rust Async Networking Proxy CRIU
\title

\toc

# Minecraft servers are HUNGRY

They hunger for your ram and your cpu. This makes it either expensive or
laggy to try and host multiple servers at once.

This was something I encountered when my friend built a homeserver out of
spare computer parts that was barely powerful enough to run a minecraft 
server.

The problem was that soon multiple people wanted a minecraft server 
hosted by him (which included us wanting to play modded minecraft).

It was a hassle to ssh into the server and start and stop the various 
servers depending on who wanted to play, especially since a lot of people
only played very rarily.

I remembered that I'd seen
[a project](https://github.com/gekware/minecraft-server-hibernation) 
that claimed to be able to hibernate a minecraft server if nobody was 
playing on it. The only issue was that it worked for a single server, and 
did so by starting and stopping the server process.

This meant that if we wanted to join a modded server the hundreds of mods 
could make us wait for several minutes before we could play.

Another issue was the fact that it could only host a single server. This 
meant that we would have to run multiple intances of the watcher, and that 
each server would be assigned to an arbitrary port that would be needed 
when connecting.

# Building a reverse proxy for minecraft

Since my friends server was accessible through a domain we thought it 
would be cool if instead of supplying a port you could connect to a 
subdomain and be sent to a specific server.

The simplest way to do this would be to create a dhcp record for each 
subdomain to point to a server, but that would be slow and tedious to set 
up for every server.

We then tried nginx, as it seemingly could magically redirect traffic to 
an internal port based on the subdomain. I quickly found out that this did 
*not* work for minecraft servers (who would have guessed), but after doing 
some research I decided on creating my own reverse proxy that spoke the 
minecraft protocol instead of HTTP.

## The Minecraft protocol

Minecraft implements its own protocol consistent of packets. My first idea 
was to see if anybody had created a rust library for dealing with minecraft 
packets. 

While some did exist, most of them where unfinished or couldn't do what I 
wanted. [One crate](https://www.youtube.com/watch?v=E4WlUXrJgy4) was useful 
for seeing how an implementation of parsing packets was done, but ultimately 
I had to write my own parser.

![An image of the steps taken to do a Ping using the minecraft protocol](/posts/Putting hungry minecraft servers to sleep/ping.png "Source: <a href='https://minecraft.wiki/w/Java_Edition_protocol/FAQ#What_does_the_normal_status_ping_sequence_look_like?'>minecraft.wiki</a>")

## Detecting the subdomain

The first step in routing the traffic was determining where to send it to.  

Luckily for me, the Minecraft procotol sends the full address it is trying to
connect to during the handshake. This meant that I simply had to parse the 
handshake and extract the subdomain from the address, see if it matches any
configured server, and then redirect all traffic to the internal port that 
server runs on.

`$ vim protocol/handshake.rs`
```rust
impl Handshake {
    pub async fn new(buf: &mut BufferedReader) -> Result<Self, Error> {
        // Packets start with their type id. The handshake is id 0x00
        if !check_bytes(&[0x00], buf).await? {
            return Err(Error::Err(
                "Wrong packet type for handshake. Expected {} got {}".into(),
            ));
        }
        buf.read(1).await?; // Consume handhake byte

        let protocol_version = VarInt::read_from(buf).await?;
        let address = read_string(buf).await?;
        let port = buf.read_u16().await?;
        let next_state = VarInt::read_from(buf).await?;

        return Ok(
            Self {
                protocol_version,
                address,
                port,
                next_state,
            },
        );
    }
}
```

The only thing I was interrested in was the `address` part of the packet. This is a full string
containing the text the user typed in the `IP Address` field in their client.

Once I had this it was as simple as splitting by `.` and taking the first result. This "server name" would now be used to look up the corosponding servers information in a table.

```rust

    let result = Handshake::new(&mut reader).await;
    if !result.is_ok() {
        return Ok(None);
    }

    let handshake = result.ok().unwrap();
    let split = handshake.address.split(".").collect::<Vec<&str>>();
    let server_name = *split.first().unwrap();

    if !SERVERS.lock().unwrap().contains_key(server_name) {
        println!("Unknown server {}", server_name);
        write_unknown_server_status(server_name.into(), &mut reader).await?;
        return Ok(None);
    }
    let mut server_lock = SERVERS.lock().unwrap();
    let server = server_lock.get_mut(server_name).unwrap();
    status = server.status;
    port = server.port;

```

***show images of hibernation and starting***

![A server that is starting](/posts/Putting hungry minecraft servers to sleep/starting.png "A server in the middle of starting")
![A server that is hibernating](/posts/Putting hungry minecraft servers to sleep/hibernating.png "An empty server thats been hibernated")

I've talked a bunch about this hibernation, but how does it actually work?

# cat /proc/$(ps aux | grep server.jar) > hibernation.txt

Finally being able to connect to the server it was time for the next item on the list.

<p style="text-align: center">✨**Hibernation**✨</p>

Now, instead of closing the server and restarting it when a player joined I
wanted to hibernate the server and unhibernate it when someone joined.

I had a feeling this would be possible as this is basically how hibernation
works for your system; it saves the memory to disk and loads it into memory
again upon boot.

After a bit of searching I found [CRIU](https://criu.org/Main_Page), a program
capable of suspending another program to disk.

Not only does it save the memory, it also saves all the file descriptors the
program was using (and fails with a million cryptic error messages if any of the
files where changed during hibernation)

There was a Rust crate for CRIU, however it was poorly documented and didnt support
all the command line arguments I needed, so I resorted to calling the binary.

This led to the program halting until the unhibernated server closed again, but
it was fixed with a simple fork.

The only downside to this approach is that CRIU requires root access in order
to read the memory and file descriptors of another process.  
Running your minecraft server as root is probably not the smartest idea...

My solution was to not care about it and hope for the best :D

# Why aren't my chunks loading?

Just as I thought I was finally done I encountered a problem. The chunks where loading *really* slowly. It felt like we we're back to
hosting on [aternos](aternos.org) (nothing against aternos, the **free** servers are just a little slow).

*image of chunks not loading*

I couldn't figure out why it was so slow. I thought it was because all the traffic had to flow through my program,
but the sockets are connected on the kernel level.

After hours of debugging I found the solution. `stream.set_nodelay(true)`

**...**

**All this time the problem was as simple as saying ** `if (slow) { dont(); }`!

To say I was pissed was an understatement, but at the same time I was glad it was finally working.

The reason this fixed it was because it disabled [Nagle's algorithm](https://en.wikipedia.org/wiki/Nagle%27s_algorithm), which bunches packets together
in order to (hopefully) speed up the sending of lots of smaller packets as one big packet.

For some reason this absolutely destroyed the chunk loading speed, and disabling it led to finally having the perfect Minecraft reverse proxy hibernation system thing.