Reproducible Development Environments with Nix Flakes

I’ve recently started using Nix and I realised that it could improve developer experience (DX) for teams. I’ve observed teams working on multiple codebases with various change rates in the past couple of year. The least frequently changing codebases often cause friction when a new developer joins the team or company, or a larger scale migration project happens and everyone has already forgotten how to get a project running.

Notes:

In this post, I will primarily focus on development on macOS but principles should be applicable to other operating systems.
I’ll assume that you have Nix installed and Nix Flakes enabled.

The Problem: capturing a working development environment

Every software project has its dependencies, some might depend on Java, some on Python, some on node.js and they use different versions of each of these. Some teams can work/own multiple projects spread to multiple technologies. Setting up a project and installing operating system level dependencies are usually captured in README.md files or sometimes go beyond that with a Makefile. However, even with the best intentions, this workflow can break down easily when:

A developer forgets to add a new dependency. For example, adds the lxml package to the requirements.txt but forgets to update the README.md/Makefile/Brewfile to install libxml2.
A project changes infrequently. A lot can happen in a year and a local development environment can get easily outdate. E.g. macOS upgrades changing Python version and breaking the local environment.

In my experience, projects that are actively worked on cause less friction because of the ongoing collaboration around them keeps the muscle memory active.

Other problem that’s worth highlighting is onboarding a new engineer to a project: how long does it take them to setup their environment from scratch following the README.md?

Example: Homebrew

Developers can meticulously document how to install dependencies in README.md files, however the outcome of brew install today is different from brew install tomorrow. It can install different versions of a toolchain (e.g. default version of Python switches from 2.7 to 3.7).

Even if autoupdate is turned off and the local environment doesn’t change due to that, it’s not possible (to my knowledge) to pin the state of the package tree. This means when Homebrew is installed on a new computer, the package tree will have a different state and even if dependencies are captured in a Brewfile, the outcome of brew bundle might be different .

Example: pyenv, nvm, and others

These tools are version managers for language toolchains themselves. They allow developers to work with multiple versions of Python on node.js easily. They either download pre-built binaries or build the toolchains from source. Python and Node.js versions are built and linked agains libraries installed locally (e.g. OpenSSL). When theses libraries gets updated (e.g. security update during brew upgrade), linking might break because Homebrew automatically removes the old version of the library. This forces engineers to regularly recompile the complete toolchain and the reason why their development environment stopped working is often not so obvious.

Example: Docker

There are two ways how teams can share development environments with Docker:

codified as a Dockerfile (and Docker Compose optionally) or
published as an image (binary blob).

First has the same issue as Homebrew: even with a pinned base image (FROM python:3.9-slim), docker build today might produce a different image than tomorrow because the image was updated.

Second has issues with architecture (x86 vs ARM) and and issue with extendability. A company might have a mixture of Intel and Apple silicon Macs in use, and this makes hard to share environments as binary blobs. Plus as soon as a developer adds an additional tool in the environment, it needs to be captured as a layer and distribute. This change happens outside of version control (except if it’s done via a continuous integration pipeline).

Using Docker for local environment brings other complexities like mapping volumes (and their performance), mapping ports between the container and the host, etc. This is more pronounced systems like macOS where Docker daemon runs in a virtual machine.

I generally prefer not to develop applications that require hot-reload within a containerised environment. However I prefer running dependencies like Redis or PostgreSQL in a container as Docker makes them easy to throw away.

A solution: Nix Flakes

Nix Flakes is an experimental feature introduced in 2.4 in November 2021. It allows capturing project dependencies, defining executables, and tests via flake.nix. Dependencies are captured in time flake.lock like in package-lock.json. This allows e.g. pinning the nixpkgs repository to a specific commit and guarantee that all project developers have the same version of the dependencies installed.

See an example for a very simple Python development environment below:

{
  description = "Python 3.9 development environment";

  inputs = { nixpkgs.url = "github:nixos/nixpkgs/nixos-21.11"; };

  outputs = { self, nixpkgs }:
    let
      system = "aarch64-darwin";
      pkgs = import nixpkgs { inherit system; };
    in {
      devShells.${system}.default = pkgs.mkShell {
        buildInputs = [
          pkgs.python39
          pkgs.python39Packages.pip
        ];
      };
    };
}

Running the nix develop command creates a bash session where the Python 3.9 interpreter is available:

$ nix develop
bash-5.1$ python --version
Python 3.9.12

You might notice that the Python interpreter version is not locked to 3.9.12 in the flake.nix file. This is because Nix also creates the flake.lock if it’s not already available. This file captures the exact state of the input dependency, in this case the nixpkgs repository. The nixpkgs repository defines the patch version used for Python 3.9.

A flake.lock file looks something like this:

{
  "nodes": {
    "nixpkgs": {
      "locked": {
        "lastModified": 1651848520,
        "narHash": "sha256-KkJ28fShdd78+zal5HlHsXUICOCtO7Bdvylc5zlp5Mk=",
        "owner": "nixos",
        "repo": "nixpkgs",
        "rev": "5656d7f92aa8264eb06ba68ad94c87ac5b5312e8",
        "type": "github"
      },
      "original": {
        "owner": "nixos",
        "ref": "nixos-21.11",
        "repo": "nixpkgs",
        "type": "github"
      }
    },
    "root": {
      "inputs": {
        "nixpkgs": "nixpkgs"
      }
    }
  },
  "root": "root",
  "version": 7
}

It contains a reference to the latest commit on the nixos-21.11 branch (latest stable release).

Let’s look at what the flake.nix file actually does more in depth.

inputs = { nixpkgs.url = "github:nixos/nixpkgs/nixos-21.11"; };

inputs field defines the sources for dependencies, in this case the nixpkgs repository that contains ~80,000 packages.

outputs = { self, nixpkgs }:
let
  system = "aarch64-darwin";
  pkgs = import nixpkgs { inherit system; };
in {
  devShells.${system}.default = pkgs.mkShell {
    buildInputs = [
      pkgs.python39
      pkgs.python39Packages.pip
    ];
  };
};

outputs functions returns information to Nix to use. These can be things like development shells, packages, apps, checks (tests). I won’t cover the additional functionalities beyond development shells here, but they can further enrich DX like the scripts field in a package.json for a node.js project or checks that could capture the validation steps for a continuous integration pipeline.

Nix Flakes explicitly require you to define all supported architectures. In this example I limited support to Nix on Apple silicon (aarch64-darwin) for simplicity.

let
  system = "aarch64-darwin";
  pkgs = import nixpkgs { inherit system; };
in ...

This line scopes the nixpkgs repository to the selected architecture by importing the nixpkgs module and passing the architecture defined by the system variable (inherit takes the system variable from the surrounding scope and makes available in the local scope).

devShells.${system}.default = pkgs.mkShell {
  buildInputs = [
    pkgs.python39
    pkgs.python39Packages.pip
  ];
};

devShells.<architecture>.default defines the default shell configuration for nix develop command. (Note: it’s possible to define multiple named shell configurations via devShells.<architecture>.<name>).

Every shell definition must be a Nix derivation. mkShell is a helper function to create a shell.

buildInputs or nativeBuildInputs define dependencies and Nix will make sure that these are available. In this case, the Python 3.9 interpreter and the pip command will be available in shell environment.

From this point on, engineers can follow the standard Python practices and manage dependencies via a requirements.txt file or Python Poetry to keep using tools that are the most common in the ecosystem.

Flakes (input dependencies) can be updated with the nix flake update command. This means, changes in dependencies can be version controlled end to end and rolled back easily.

If you want to take the developer experience a step further, direnv supports automatically activating Nix flakes based development environments when you enter a project directory. Assuming the direnv 2.24.0 or later is installed and configured for your choice of shell (e.g. zsh, fish, etc.), the setup is fairly straight-forward:

$ echo "use flake" >> .envrc
$ direnv allow .

Bonus: a simple solution to support multiple architectures

nixpkgs supports multiple architectures with a tier system. This is useful because for example Homebrew and Fedora ships development headers while you might need to install an extra package on Debian. This is a problem if you try automating installing dependencies with make or Ansible.

See an example of the flake.nix with modification to support both Intel and Apple silicon Macs plus Linux on x86:

{
  description = "Python 3.9 development environment";

  inputs = { nixpkgs.url = "github:nixos/nixpkgs/nixos-21.11"; };

  outputs = { self, nixpkgs }:
    let
      inherit (nixpkgs.lib) genAttrs;
      supportedSystems = [
        "aarch64-darwin"
        "x86_64-darwin"
        "x86_64-linux"
      ];
      forAllSystems = f: genAttrs supportedSystems (system: f system);
    in {
      devShells = forAllSystems (system:
        let pkgs = import nixpkgs { inherit system; };
        in {
          default = pkgs.mkShell {
            buildInputs = [
              pkgs.python39
              pkgs.python39Packages.pip
            ];
          };
        });
    };
}

nix flake show should list the supported architectures:

$ nix flake show
path:/Users/.../nix-flakes-python-example?lastModified=1652384685&narHash=sha256-mBJJTNPZqr5d9VoQyDxOE+V2LpIm3QMldDzX1mdmCsU=
└───devShells
    ├───aarch64-darwin
    │   └───default: development environment 'nix-shell'
    ├───x86_64-darwin
    │   └───default: development environment 'nix-shell'
    └───x86_64-linux
        └───default: development environment 'nix-shell'

There are utilities like flake-utils that can provide these functions and help avoid copy pasting between flake.nix files.

The Problem: capturing a working development environment#

Example: Homebrew#

Example: pyenv, nvm, and others#

Example: Docker#

A solution: Nix Flakes#

Bonus: a simple solution to support multiple architectures#