Some experiments with Erlang and Nix

Some experiments with Erlang and Nix September 7, 2024 by Marcos Benevides

Recently I’ve been collaborating with friends on Lyceum, an MMO game with an Erlang backend and a Zig + Raylib client (as if erlang wasn’t a weird enough of a choice). Now, this is an unusual combination, but that’s the whole reason our pesky group exists in the first place (if you want to know more check my friend’s Lemos post).

There is also a couple of standards we try to follow when doing this project, all of the team works with microservices all day in their normal jobs, so whenever we want to do something we try follow some simple rules:

Can we develop all of the project parts locally? Preferably with no networking as well (besides pulling dependencies).
Can we do so by leveraging a couple handy tools to their limit?

One can imagine that setting up such a development environment might be nightmarish, but thankfully the 21st century brought us some interesting tools that make Unix less of a mess to deal with, and yes, I’m talking about Nix. My goal here is to show people what our development experience looks like and maybe convince a few souls dealing with more normal tools (brew, asdf, <insert random linux package manager>, …) to at least give Nix a try.

Devenv

We use devenv to setup our development shell, think of it as your favorite programming language’s envinroment and dependency manager (pip, poetry, nvm, rvm, …) but capable of installing anything availiable on nixpkgs and much more.

A unified development shell for Erlang and Zig

No one is expected to have Erlang, Zig and Postgres installed, nor are they expected to have any of the environment variables needed for this project to work, the development shell already does all of that boring stuff for you. Here’s a snippet of what it looks like:

  # (...)
  devShells = forAllSystems (
    system:
    let
      pkgs = nixpkgs.legacyPackages.${system};

      # Erlang shit
      erlangLatest = pkgs.erlang_27;
      erlangLibs = getErlangLibs erlangLatest;

      # Zig shit
      raylib = pkgs.raylib;
      zigLatest = pkgs.zig;

      linuxPkgs = with pkgs; [
        inotify-tools
        xorg.libX11
        xorg.libXrandr
        xorg.libXinerama
        xorg.libXcursor
        xorg.libXi
        xorg.libXi
        libGL
      ];
      darwinPkgs = with pkgs.darwin.apple_sdk.frameworks; [
        CoreFoundation
        CoreServices
      ];
    in
    {
      # (...) This will be shown in the next section

      # `nix develop`
      default = devenv.lib.mkShell {
        inherit inputs pkgs;
        modules = [
          (
            { pkgs, lib, ... }:
            {
              packages =
                with pkgs;
                [
                  erlang-ls
                  erlfmt
                  just
                  rebar3
                  dbeaver-bin
                ]
                ++ lib.optionals stdenv.isLinux (linuxPkgs)
                ++ lib.optionals stdenv.isDarwin darwinPkgs;

              languages.erlang = {
                enable = true;
                package = erlangLatest;
              };

              languages.zig = {
                enable = true;
                package = zigLatest;
              };

              env = mkEnvVars pkgs erlangLatest erlangLibs raylib;

              scripts = {
                build.exec = "just build";
                server.exec = "just server";
              };

              enterShell = ''
                echo "Starting Development Environment..."
                just deps
              '';

              services.postgres = {
                package = pkgs.postgresql_16.withPackages (p: with p; [ p.periods ]);
                enable = true;
                initialDatabases = [ { name = "mmo"; } ];
                port = 5432;
                listen_addresses = "127.0.0.1";
                initialScript = ''
                  CREATE USER admin SUPERUSER;
                  ALTER USER admin PASSWORD 'admin';
                  GRANT ALL PRIVILEGES ON DATABASE mmo to admin;
                '';
              };
            }
          )
        ];
      };

     # (...)

Let’s try building the Zig client:

  $ just client-build
  $ just client

Running Postgres

As you may have noticed, not only are we installing Erlang and Zig, some madlad even put dbeaver there for God knows what reason, but hey, that’s the dev shell, just do whatever you want. We also have a local postgres setup and the workflow mimics what you usually have with docker-compose or podman. By running:

  devenv up

inside the shell, a local Postgres 16 with custom extensions will be spinned. The list of services supported by devenv keeps growing and you can check them here.

Direnv

As if thigs weren’t awesome enough, I need to talk about direnv, a simple tool that can make wonders (and it comes with nix integrations for free), with a single .envrc in your project’s repo you can jump inside a certain development shell just by cd-ing into the directory. Here’s an example of my .envrc:

use flake . --impure

followed by a direnv allow in my shell:

  $ direnv allow
  direnv: loading ~/Code/Personal/lyceum/.envrc
  direnv: using flake . --impure
  direnv: nix-direnv: Renewed cache
  Starting Development Environment...
  rebar3 get-deps
  ===> Verifying dependencies...
  rebar3 nix lock
  ===> Verifying dependencies...
  # (...)

That’s it. Now every time I cd <lyceum-directory>, I’ll immediatly load the whole development shell and be ready to work on it. This section is optional but it really simplifies my life, as I don’t need to remember to activate/deactivate an environment.

The CI environment

Since we are already went to the trouble of setting up a whole dev environment for Erlang and Zig, we should just make another one for when we need to run builds and test suites on CI.

    # `nix develop .#ci`
    # reduce the number of packages to the bare minimum needed for CI
    ci = pkgs.mkShell {
      env = mkEnvVars pkgs erlangLatest erlangLibs raylib;
      buildInputs = with pkgs; [
        erlangLatest
        heroku
        just
        rebar3
        zigLatest
      ];
    };

If you use Github Actions, now you can leverage both the Install Nix and Magic Nix Cache actions.

The full devshell

You can check what the full devshell looks like here.

Nix Build

In the previous section I’ve showed you our impure environment, there’s no way (as of now) to make things 100% pure while developing, specially because we need to have a postgres service running to debug and test locally. However, things change when we talk about releases, we need to find a way to properly build the server.

A pure build of the Erlang server

This is the original reason I’ve decided to write this, it took me some time to go through the NixOS BEAM manual and I’ve yet to know how to properly build this project with the buildRebar3 Tools (it seems it’s used more inside Nixpkgs itself than to integrate with Erlang projects). Nevertheless, you can properly package this with the abstractions plain Nix already gives you:

  # Leverages nix to build the erlang backend release
  # nix build .#server
  server =
    let
      deps = import ./rebar-deps.nix { inherit (pkgs) fetchHex fetchFromGitHub fetchgit; };
    in
    pkgs.stdenv.mkDerivation {
      name = "server";
      version = "0.0.1";
      src = pkgs.lib.cleanSource ./.;
      buildInputs = with pkgs; [
        erlangLatest
        pkgs.stdenv.cc.cc.lib
        rebar3
        just
        gnutar
      ];
      nativeBuildInputs = with pkgs; [
        autoPatchelfHook
        coreutils
        gawk
        gnugrep
        libz
        ncurses
        openssl
        systemdLibs
      ];
      buildPhase = ''
        mkdir -p _checkouts
        # https://github.com/NixOS/nix/issues/670#issuecomment-1211700127
        export HOME=$(pwd)
        ${toString (
          pkgs.lib.mapAttrsToList (k: v: ''
            cp -R --no-preserve=mode ${v} _checkouts/${k}
          '') deps
        )}
        just release-nix
      '';
      installPhase = ''
        mkdir -p $out
        mkdir -p $out/database
        # Add migrations to the output as well, otherwise the server
        # breaks at runtime.
        cp -r database/migrations $out/database
        tar -xzf _build/prod/rel/*/*.tar.gz -C $out/
      '';
    };

This is a derivation, a meta-package, a recipe containing every step and every dependecy I need to satisfy and properly build our server. Now, as for the deps.nix file, it was auto-generated with rebar3-nix, which itself has a rebar3 plugin. So everytime someone adds a BEAM dependency in our current flow, we automatically generate a nix lockfile to match the rebar3 lockfile as well. Here’s what we needed to add in our rebar3 config to benefit from the Nix integration:

{plugins, [
    { rebar3_nix, ".*", {git, "https://github.com/erlang-nix/rebar3_nix.git", {tag, "v0.1.1"}}}
]}.

now let’s see if this really works:

  $ nix build .#server
  # (...)
  # We now have a `result` directory in the project's root...
  $ ls ./result/
  bin  database  erts-13.2.2.10  lib  releases
  # Now try running the server we've just build and...
  $ ./result/bin/server foreground
  Exec: /nix/store/cm6vsbfls41q6s5ms4y2gfnxvmx1qzfq-server/erts-13.2.2.10/bin/erlexec -noinput +Bd -boot /nix/store/cm6vsbfls41q6s5ms4y2gfnxvmx1qzfq-server/releases/0.0.1/start -mode embedded -boot_var SYSTEM_LIB_DIR /nix/store/cm6vsbfls41q6s5ms4y2gfnxvmx1qzfq-server/lib -config /nix/store/cm6vsbfls41q6s5ms4y2gfnxvmx1qzfq-server/releases/0.0.1/sys.config -args_file /nix/store/cm6vsbfls41q6s5ms4y2gfnxvmx1qzfq-server/releases/0.0.1/vm.args -- foreground
  Root: /nix/store/cm6vsbfls41q6s5ms4y2gfnxvmx1qzfq-server
  /nix/store/cm6vsbfls41q6s5ms4y2gfnxvmx1qzfq-server
  Connecting to: "127.0.0.1"
  Connected to "127.0.0.1" with USER = "admin"
  Finding migration scripts...
  Migration Path: "/nix/store/cm6vsbfls41q6s5ms4y2gfnxvmx1qzfq-server/database/migrations"
  Running DB migrations.
  Migrations completed successfully.
  # (...) it works

Containers

There is a treasure trove of examples in Nixpkgs, I’ve decided to go with the simplest one. This what a container for the backend looks like in Nix:

  # nix build .#dockerImage
  dockerImage = pkgs.dockerTools.buildLayeredImage {
    name = "lyceum";
    tag = "latest";
    created = "now";
    # This will copy the erlang release derivation from the
    # previous step into to the image
    contents = [ server pkgs.coreutils pkgs.gawk pkgs.gnugrep ];
    config = {
      Cmd = [
        "${server}/bin/server"
        "foreground"
      ];
      Env = [
        "ERL_DIST_PORT=8001"
      ];
      ExposedPorts = {
        "8080/tcp" = { };
      };
    };
  };

It doesn’t really look like most Dockerfiles you see around the net. Notice that I’m using the server derivation from the previous step, the hard work required to make it work the first time is immediatly rewarded because now we can keep composing the previous solutions into more complex flows. To test this, let’s build the image:

  $ nix build .#dockerImage
  # Now load the build image in docker (or podman)
  $ docker load < ./result
  # Make sure you have `devenv up` running
  $ docker container run --network=host --rm lyceum:latest
  Exec: /nix/store/vwnrgsah54qf9ca0ax921061b6sm1km9-server/erts-13.2.2.10/bin/erlexec -noinput +Bd -boot /nix/store/vwnrgsah54qf9ca0ax921061b6sm1km9-server/releases/0.0.1/start -mode embedded -boot_var SYSTEM_LIB_DIR /nix/store/vwnrgsah54qf9ca0ax921061b6sm1km9-server/lib -config /nix/store/vwnrgsah54qf9ca0ax921061b6sm1km9-server/releases/0.0.1/sys.config -args_file /nix/store/vwnrgsah54qf9ca0ax921061b6sm1km9-server/releases/0.0.1/vm.args -- foreground
  Root: /nix/store/vwnrgsah54qf9ca0ax921061b6sm1km9-server
  /nix/store/vwnrgsah54qf9ca0ax921061b6sm1km9-server
  server[1] Starting up
  Connecting to: "127.0.0.1"
  Connected to "127.0.0.1" with USER = "admin"
  Finding migration scripts...
  Migration Path: "/nix/store/vwnrgsah54qf9ca0ax921061b6sm1km9-server/database/migrations"
  Running DB migrations.
  Migrations completed successfully.
  # (...)

Conclusion

As I wanted to show here, we’ve used Nix all the way from defining a common development environment for the developers, reused some of the stuff in CI, to later repurpose some of the flows for pure builds, that later got shoved into our containers, all by leveraging the same tool. I wish modern devops was more about that, but it seems it’ll take time for people to realize that immutability, composition and functional programming can go hand in hand and give us a better experience than one can find in most other solutions (built by trillion dollar companies who want you to manage infra with YAML). Luckilly, Nix is gaining some traction and more people are talking about it.

I’ve been using it for the past 6 years in my workstations and don’t regret doing so, its a tool worth learning (and there’s still so much to learn about it), it makes my life dealing with Unix systems less painfull.

TODO

There is still much to do, and it can be left for a part II later.

I have yet to learn how to deploy a production-ready erlang system. Add (Cesarini and Vinoski 2016) to my readlist.
Properly build the client, it seems that non2nix breaks with the format for zon files, I’m not familiar with Zig toolig and ill take a look at this later
We are still unsure where to deploy, but I really want to move away from Heroku and check what Nix has to offer to manage a fleet of VMs.

References

Cesarini, Francesco, and Steve Vinoski. 2016. Designing for Scalability with Erlang/Otp: Implement Robust, Fault-Tolerant Systems. O’Reilly Media, Inc.

Links to this note

You have 10 seconds to nixify your .Net project