In the beginning… there was a software development project…

As with other software development project, we start with defining project dependencies that is required to run the project. Let’s say, for a typical Python project, you defined a REQUIREMENTS.txt file that contains all the Python packages needed to run your project. This is in itself fine and all, problem solved. However, things began to change when you want to hand the project over to someone unfamiliar with Python.

For Python beginner who doesn’t know yet about Python (or any language, but now we just assume it is Python), you don’t know how to start the project itself. You usually need a README.md file in the project repo that describes how to setup such project. But things don’t usually work well because you don’t know how to start.

Let’s just say that the project needs Python and Pip. So you Google’d a way to install this dependencies. You found an article to install Python in Linux. So you try it. But it doesn’t work, because you are on MacOS. So you find another article that describes how to install Python using brew. But it doesn’t work, because you are on MacOS M1 chip, and you needed a specific version of Python, which is Python 3.7 because Python 3.8 can’t run Tensorflow, for example. In which case, you needed Pyenv… The lists can goes on until finally you resolved the issues… in your own machine. You finally able to install the required Python until you realized that running pip install -r REQUIREMENTS.txt failed because it needs some Linux shared library that is not available unless you build it yourself.

Imagine that if you have a team of 5 people, each of them have different machines. They had to resolve it by themselves. Well, tough luck.

The above scenario is my real example in the past projects and my main motivation to learn Nix Package Manager.

What is Nix Package Manager?

One thing that I regret is that Nix’s name is so generic that if you Google’d it, you will found dozens of articles that refers to Unix/Linux instead of the package manager called Nix. I can probably rambling around behind the motivation on why people eventually make Nix. However, for now I will just introduce it.

Nix is:

• A package manager
• A build system
• Designed to be used in Functional Programming paradigm
• Declarative
• Intended to be used as a pure function

If you are not familiar with functional programming, this can be a daunting concept to understand. Your mind will constantly fighting to understand why people need this. I understand and I can symphatize. Good news is that you can use it without having to be completely understand the mechanics behind the scene. But do try to understand it bit by bit, because the concept behind it is universal and not strictly restricted for any such domain language.

Now that I’ve said it, you just need to understand these vision that Nix want to achieve/solve:

Reproducible

Running Nix against a nix recipe (input) will produce the same result (output), wherever and whenever it was run.

Declarative

When you want to have a setup works in other machine, you only need to share the nix recipe and the input of your build. The input can be a source code, binary, another nix recipe, etc. Whatever the source code you are using, the language, and the build input, you only need nix to completely generate the output from the recipe. So, it’s very portable.

Reliable

Nix is designed to not break your existing system. If you install Python from Nix, it’s not going to delete or mess up your system packaged Python installed from MacOS or debian. You can have multiple Python instance and it will not tamper each other.

How it can be used in a project?

Before I learned about Nix, I am using Docker to build my project dependencies, and then use containers to run my project. This guarantees project isolation too. Nix does the same thing but with a different (radical) approach. Any dependencies that you declare with Nix is going to be cached and isolated in a certain directory. It then be composed so that you can use it. When you rerun a Nix build, if it uses the same input, the output is going to be the same, so it will only get the result from the cache. If you change the input, it will intelligently process it and determine if the components need to be rebuilt or not.

Nix caches are universal. The package manager ecosystem of Nix even relies on this. The nixos.org maintains binary caches that nix ever built for it’s packages. Whenever you use the same package, it will fetch the result from this cache over the internet. This cache can even be packaged and transferred to your friend’s machine, so that when your friend build the same recipe, it will use their local cache.

In summary, these are the benefit of using Nix compared with a container based approach for dependency management.

• More reproducible than Docker recipe
• It can rebuild the dependency tree from source code if you want to
• Recipes can be generic. You can customize build parameters (like source version) with the same recipe
• Lightweight, because it doesn’t need a VM

It is also important to understand that both Nix and Containers based approach is intended to solve a different thing. Nix wants to solve dependency management and build system, while Containers are used for process isolations. One doesn’t have to replace the other. However one is more suitable than the other in a certain tasks. Using Dockerfile recipe to declare package dependency is almost an overkill to some extent, while sometimes poorly addresses the issue.

That being said, since Nix is also a build system, it is possible to create an OCI image from a Nix recipe, and then run it in a container runtime such as Docker. So both can complements each other.

Tutorial on setting up Nix the first time

There should be no issue on running this setup in Linux, be it using x86_64 architecture or ARM (if it is supported). For demonstration purposes, we are going to demonstrate the setup in MacOS.

The Goal

We are going to setup a Nix-based environment for an ops based repo. We want to achieve these:

• Repo can be checked out easily or copied over anywhere
• Any executable binary needed to run the script inside the repo has to be the same for anyone who have the repo.
• It should ease up beginners to quickly setup their development environment
• It can be used in an automation tools to quickly fetch the dependencies to run the scripts inside the repo

Due to how reproducible it is, this project setup can be used for GitOps. They even have a specific terms for it called NixOps. But, of course the main motivation for us right now is to setup development environment in a specific programming language.

Installing Nix

The system-wide binary we need is Nix itself. It can be installed by following the guide here

There are two flavors of installing Nix. Single user install and Multi User install. The main difference is that for security reasons, multi-user install is recommended if your system have several users working in the machine. Otherwise, just go with the single user install.

For MacOS, there are certain quirks due to how the root filesystem is readonly even for the administrator, so you have to create a separate volume for the Nix caches. Just follow the instruction here

For Linux, you just need to execute this (single user install):

curl -L https://nixos.org/nix/install | sh -s

In the case of MacOS above, the options is tweaked a little bit.

curl -L https://nixos.org/nix/install | sh -s -- --darwin-use-unencrypted-nix-store-volume

To verify that it is running, you need to reexecute your shell initialization script, because nix needs to be hooked into your shell. I’m using zsh so I’m doing something like this:

# ZSH
source ~/.zshrc
# BASH
source ~/.bashrc

Then run a version checked

nix-env --version

Installing Nox

While Nix itself is a build system and package manager, it might be difficult for you to straight using it. Especially if you usually do something like apt -y update; apt -y install <packagename>. This is because not only the package name convention might be different, how to search the package is different too. Most nix packages are described as modules. When installing a specific modules you are actually evaluating a Nix recipe/expression in Nix distribution channel.

You can search available Nix packages in NixOS search URL. However, there is this little helper tools that allows you to search right from the command line called Nox.

We install Nox like this:

nix-env -iA nixpkgs.nox

Once you have nox, you can simply search any packages like this

nox <regular expression query>

Using nox is optional, but might help first-timer to search for packages. To demonstrate the case, installing GDAL library in debian usually will results in only one possible candidate. If you need GDAL 3 while your debian OS only supports GDAL 2 binaries, then you need to compile GDAL 3 yourself.

In Nix, it’s quite common that the distribution channel stores multiples recipes for libraries with different major version. If you search for GDAL using Nox

nox GDAL

You will be presented several options available in the channel:

1 gdal-3.2.2 (nixpkgs.gdal)
    Translator library for raster geospatial data formats
2 gdal-2.4.4 (nixpkgs.gdal_2)
    Translator library for raster geospatial data formats
3 pdal-2.2.0 (nixpkgs.pdal)
    PDAL is Point Data Abstraction Library. GDAL for point cloud data
4 gdal-3.2.2 (nixpkgs.python38Packages.gdal)
    Translator library for raster geospatial data formats
5 gdal-3.2.2 (nixpkgs.python39Packages.gdal)
    Translator library for raster geospatial data formats

Notice that option 1 and 2 refer to the C/C++ based native libraries. While option 4 and 5, is for GDAL library used as Python libraries, each corresponds for Python 3.8 and Python 3.9.

You don’t see what you want here? No worries, you can always customized the module/build with your intended version. But this is a separate topic later. For now, you just want to familiar yourself on how to use Nix.

Installing packages using Nix (optionally with Nox)

After you find your package name you can install it using:

nix-env -i <package name>

The above command will try to find the package name first. A slightly faster command, since you know the exact module name:

nix-env -iA <channel-name>.<package-name>

For example, if you want to install gdal package, you do it like this (nixpkgs is the default channel):

nix-env -iA nixpkgs.gdal

If you search the package using Nox, it also offer you to install it by specifying the number from the list.

Installing Direnv

Direnv is a powerful tools to setup a context in a directory. It’s a directory-based profile. When you change directory in your shell, it will execute the profile associated in that directory.

A “profile” can be used for many things. A simple thing is to set environment variable whenever your shell is inside that directory. A more sophisticated thing is to combine it with Nix to set up a developer environment for any shell that enters that directory.

Imagine that you have 3 separate Git repo in your local machine that all uses different Python version needed by the project. With Direnv and nix, you can set that whenever your shell enters a repo, it will use the correct Python version.

This concept is not limited to just the Python version (which is what usually Pyenv is used for), but for any packages. If you have two project that uses GDAL 3 and GDAL 2 (different binary dependencies), you can easily switch between profile just as easy as you cd into that directory.

To install Direnv using Nix:

nix-env -iA nixpkgs.direnv

Direnv also needs to be hooked to your favorite shell. Just follow it’s guide for your respective shell

Reload your shell and make sure direnv works:

direnv version

Installing Niv

Niv is a tool used to pin Nix-based dependencies. If we are to make comparison, it is analoguous with Javascript’s yarn.lock, Python’s REQUIREMENTS.txt, or Ruby’s Gemfile. The difference is that, Nix is also a build system. Thus, Niv also need to pin the source code used to produce the build. It is achieved by also making a hash of the input.

To install Niv using Nix:

nix-env -iA nixpkgs.niv

Wrapping it up together

Let us compose a small demo.

We are creating a directory called project. The goal is to have a working shell with several CLI tools.

mkdir -p /tmp/project
cd /tmp/project

First, we initialize Niv, to make sure we have a reproducible dependencies.

niv init

It will create a directory called nix with sources.json and sources.nix inside of it.

Now, let’s say that I know this setup works only if we are using Nixpkgs recipes from release 21.05 distribution channel. I want this to work on your machine too, so I explicitly said that we need Nixpkgs 21.05.

niv update nixpkgs -b release-21.05

Where release-21.05 corresponds to the name of the git branch of Nixpkgs. For practical purposes, most likely you don’t have time to completely build the dependencies from source. In 6 months time or the next couple of years, most of the caches probably been purged. Because our dependencies are not too strict, let’s switch the channel to the nixpkgs-unstable branch where the current public caches are tracked. If you want to switch back to release-21.05 branch and try to completely build it, you can reexecute the previous command.

niv update nixpkgs -b nixpkgs-unstable

The next step is to define a file called shell.nix. This file is used to instantiate a shell where our dependencies is going to be used.

Create this file with initial content like this:

{ }:
let
  sources = import ./nix/sources.nix ;
  pkgs = import sources.nixpkgs { };
in
pkgs.mkShell {
  buildInputs = [
    pkgs.python37
  ];
}

In line 8, in buildInputs array, we declared that we want python37 provided by nixpkgs.

Run nix-shell and wait for it to complete. After that, check the python version. It will output Python 3.7. To exit nix-shell, go back by typing exit.

Most programming language have Nix packages for different major version. Let’s try to change from Python 3.7 to Python 3.8.

...
buildInputs = [
    pkgs.python38
  ];
...

You will notice that if you use nix-shell and execute python --version, it will output Python 3.8.

Most probably you know about Python before you even know about Nix. In this situation, a Python project usually have a file called requirements.txt to store it’s Python dependencies. In normal situation, you would use pip to build and/or fetch these python packages from python package registry.

However, let’s just say upfront that the way Nix build a package is often philosophically different on how packages are built in a specific programming language. There’s usually a guide on how to adapt and let Nix build from existing language builder such as pip and setuptools in Python. We’re not going to cover this now (it’s quite advanced topic). Instead, we are going to use Nix to make some CLI tools available (which is usually lightweight and prebuilt).

We want to include a couple of CLI:

• Kubectl, to operate against a kubernetes cluster
• Docker CLI, (only the client) to operate against a Docker daemon
• jq, to operate on a JSON output, right from the CLI
• curl, a CLI to interact with URL
• Helm, a CLI to install Helm Apps into a kubernetes cluster

We can search each package using Nox, just to check if it is indeed supported in Nix. Then we can put the package name in the buildInputs declaration.

...
buildInputs = with pkgs; [
    python38
    kubectl
    docker-client
    jq
    curl
    kubernetes-helm
  ];
...

Now, in addition to the previous Python 3.8, you now have these CLI installed and available in nix-shell.

This seems to be trivial and unimportant. However, what Nix does right here is providing a “meta” way to manage your native dependencies, so that you can work on top of them. You don’t need to include a specific instruction to tell your collaborator that you need Kubectl and told them to download the binaries. You just told them to install Nix and run nix-shell, and these CLI tools will be ready.

Imagine if all the repo in your organization works like this. There’s no need to ask or confirm if a certain repo needs make or cmake or any combination of build tools. You just need to declare that such build tools or CLI exists by describing it in Nix. It is saving them time to setup their projects.

I personally use this approach to handle cloud-based deployment. I put the Nix recipes so that any devops or developer can immediately interact with a cloud instance, because the CLI is available right after they cd into the repo directory. This is where direnv comes into play.

Create a .envrc file containing just this line (in /tmp/project):

use_nix

If you press enter in the current directory /tmp/project, you will get a notice that Direnv recognize a setup script and asked you to approve it if you want to, using direnv allow. Simply execute that. Direnv now will make your current shell becomes a nix-shell. You will have all the above tools ready (kubectl, helm, jq).

You can check the version of above CLI. The magic is that when you cd out of the directory, Direnv will unload the setup and your shell becomes your regular shell again, where those CLI is not available anymore.

cd /
# check those CLI, you will get error if you don't have the tools in system-wide
# or, if you have it system-wide, note the version
kubectl version
helm version
jq --version
cd /tmp/project
# Direnv is activated again
# You can check now if the version is different
kubectl version
helm version
jq --version

Possible usage beyond setting up development environment

Due to it’s portability, this approach works beyond setting up local development environment. Basically any work that needs a first time setup might benefit from this approach:

• Setting up build agent in CI
• Setting up a cloud server
• Setting up an on-prem deployment
• Setting up an environment for a tutorial
• Creating a sandboxed dependencies

Despite it’s promise, the huge problem of adopting Nix is it’s far departure from conventional package manager. Running an existing recipes is straight-forward. However, creating those recipes is a huge investment because Nix is in itself a programming language. This is unavoidable if you are learning a new build system.

I think the analogy works the same. When you first use a Makefile, you only need to learn commands to run a Make target. You don’t have to understand how to create the Makefile itself. Unless you actually compose the Make project from scratch.