Isleward, the open-source MMORPG I am working on with @bigbadwofl, is growing and with it grow the server hardware demands. This post describes the infrastructure I’ve put in place for our indie MMORPG as of the v0.9.0 release in mid-May.
I wrote about some parts of the infrastructure and release deployment here in the past. But instead of making you read the previous blog posts and pointing out changes, I’ll write it all out from scratch in this post.
Server Infrastructure
This section covers the actual servers - the hardware, how the servers are being set up and kept up to date, and what software is actually running on the servers.
Hardware
Up until v0.9.0, we used to run the entire stack for Isleward on a single VPS. This worked well, but we were starting to hit some hardware limits and while we could just rent a beefier server. It was not really a scalable or future-proof solution.
Instead, Isleward is now running on four separate VPS instances with slightly different specs. The idea is that some parts of the stack are more demanding than others and we’ll run those parts in a distributed fashion - databases on one VPS, actual Isleward on a different one, etc. Of course this does nothing for improving reliability, it is not a highly available setup, and there are multiple single points of failure. But for now, Isleward is still rather small and should the need arise, we are now prepared to spin up and provision new machines and switch to a more robust HA set up in a matter of minutes.
All VPS instances are connected together in a private network, so they can communicate with each other.
Provisioning
The VPS instances are provisioned using trusty old Ansible, using a playbook based on Michal Klempa’s ansible-docker-base. Originally I was planning to run the playbook in Gitlab CI, but due to some technical limitations I’ve put off that idea for now and I’m running it from my own machine for the time being.
Role wise, it’s a pretty light playbook, featuring only geerlingguy.docker, geerlingguy.security and a custom role for some extra configuration, like the aforementioned private network, and system packages.
Software
Containers
The heart of the stack from the software side is Docker running in swarm mode. I opted for Docker swarm mode instead of Kubernetes, even considering something like MicroK8s or K3s, because it’s lighter on resources and, at least to me personally, docker compose files are easier to understand than Kubernetes manifests.
On top of Docker swarm mode, we are running Portainer, which is an awesome utility that makes managing containers a breeze. Yes, we could still use the CLI, and sometimes I do, but Portainer allows me to get information at a glance, and has a few extra tricks up its sleeves like browsing mounted volumes. For swarm mode specifically there is a webhook you can call to update a service, enabling push-based updates rather than polling a docker registry periodically to see whether there’s a new image.
The stack
On the various servers, each dedicated to its purpose, we run the whole container stack - Multiple instances of Isleward, one for production, others for testing; RethinkDB for Isleward; Mediawiki and its database, and Caddy as a reverse proxy to point people at the right containers.
I’ve decided to replace Nginx with Caddy for the ease of use. Caddyfiles are easier to understand, and a lot of things happen out of the box in Caddy. Setting up a reverse proxy is usually a single line in a config, TLS (https) certificates are obtained and renewed automatically, etc. Thanks to Caddy I managed to cut down our long and messy nginx config file to 94 lines and that includes comments and extra lines to increase readability.
Deploying a release
Now that you know the server infrastructure side, it’s time to talk about how Isleward gets from a git repo to running on a production server. I will only cover the journey for the release build running on play.isleward.com, but it’s very similar for ptr.isleward.com as well.
Building the public image
First, we build the open source public image, that’s pushed to Docker Hub. This happens using GitLab CI, and you can see the .gitlab-ci.yml file here, but I can do a quick rundown:
- First we merge the master branch into the release branch
- Then we make a new tag on the release branch
- Tagging the release branch triggers the release build
- In the build, first, we lint and audit the code, to make sure it’s up to standard
- And then we build the docker image from the Dockerfile and push it to Docker Hub
Building the private image
After that is done, we kick off another GitLab CI build in our private repository. Because it’s private, I can’t link to it here, but I’ll sum up the process again:
- Using a variable, we set the deployment target. For release, that’s play.isleward.com
- We then clone a bunch of private content mods, again leveraging the deployment target variable and a yaml file that defines what mods go to which deployment target
- After that, we build a docker image based on the public release image built just moments ago and push it into our private repository hosted on GitLab
Deploying the image
As a part of the private image GitLab CI pipeline, we notify players that a new version will be pushed soon, and call the Portainer webhook telling the appropriate Isleward service that it should update itself. When that happens, the container restarts itself, booting people out and when they log back in, they’ll be on a new, updated container.
A word on private content mods
While Isleward is open source, it’s probably closer to open core. We do have some private content and feature mods which are not open running in the live game. That being said, we do want to make some mods open source for people to look at and use.
If you got this far, thanks for reading :) Work on this infrastructure was a long time coming and we’ve been planning it for at least a year, and it also took some time to work it all out but in the end it was a lot of fun and I am mostly happy with it.
As usual, if you have any questions, comments, etc. you can reach me through the links in the footer.
~ Vildravn
>> Home