Chapter 13. Callpackage Design Pattern

In the previous pill, we demonstrated how the inputs pattern decouples packages from the repository. This allowed us to manually pass the inputs to the derivation; the derivation declares its inputs, and the caller passes the arguments.

However, as with usual programming languages, there is some duplication of work: we declare parameter names and then we pass arguments, typically with the same name. For example, if we define a package derivation using the inputs pattern such as:

{ input1, input2, ... }:
...

we would likely want to bundle that package derivation into a repository via a an attribute set defined as something like:

rec {
  lib1 = import package1.nix { inherit input1 input2 ...; };
  program2 = import package2.nix { inherit inputX inputY lib1 ...; };
}

There are two things to note. First, that inputs often have the same name as attributes in the repository itself. Second, that (due to the rec keyword), the inputs to a package derivation may be other packages in the repository itself.

Rather than passing the inputs twice, we would prefer to pass those inputs from the repository automatically and allow for manually overriding defaults.

To achieve this, we will define a callPackage function with the following calling convention:

{
  lib1 = callPackage package1.nix { };
  program2 = callPackage package2.nix { someoverride = overriddenDerivation; };
}

We want callPackage to be a function of two arguments, with the following behavior:

In this section, we will build up the callPackages pattern from scratch. To start, we need a way to obtain the argument names of a function (in this case, the function that takes "inputs" and produces a package derivation) at runtime. This is because we want to automatically pass such arguments.

Nix provides a builtin function to do this:

nix-repl> add = { a ? 3, b }: a+b
nix-repl> builtins.functionArgs add
{ a = true; b = false; }

In addition to returning the argument names, the attribute set returned by functionArgs indicates whether or not the argument has a default value. For our purposes, we are only interested in the argument names; we do not care about the default values right now.

The next step is to make callPackage automatically pass inputs to our package derivations based on the argument names we've just obtained with functionArgs.

To do this, we need two things:

The former is easy: we just have to set our package derivation's inputs to be package names in a repository, such as nixpkgs. For the latter, Nix provides another builtin function:

nix-repl> values = { a = 3; b = 5; c = 10; }
nix-repl> builtins.intersectAttrs values (builtins.functionArgs add)
{ a = true; b = false; }
nix-repl> builtins.intersectAttrs (builtins.functionArgs add) values
{ a = 3; b = 5; }

The intersectAttrs returns an attribute set whose names are the intersection of both arguments' attribute names, with the attribute values taken from the second argument.

This is all we need to do: we have obtained the argument names from a function, and populated these with an existing set of attributes. This is our simple implementation of callPackage:

nix-repl> callPackage = set: f: f (builtins.intersectAttrs (builtins.functionArgs f) set)
nix-repl> callPackage values add
8
nix-repl> with values; add { inherit a b; }
8

Let's dissect the above snippet:

In the snippet above, we've also demonstrated that the callPackage call is equivalent to directly calling add a b.

We achieved most of what we wanted: to automatically call functions given a set of possible arguments. If an argument is not found within the set we used to call the function, then we receive an error (unless the function has variadic arguments denoted with ..., as explained in the 5th pill).

The last missing piece is allowing users to override some of the parameters. We may not want to always call functions with values taken from the big set. Thus, we add a third parameter which takes a set of overrides:

nix-repl> callPackage = set: f: overrides: f ((builtins.intersectAttrs (builtins.functionArgs f) set) // overrides)
nix-repl> callPackage values add { }
8
nix-repl> callPackage values add { b = 12; }
15

Apart from the increasing number of parentheses, it should be clear that we simply take a set union between the default arguments and the overriding set.

Given our callPackages, we can simplify the repository expression in default.nix:

let
  nixpkgs = import <nixpkgs> {};
  allPkgs = nixpkgs // pkgs;
  callPackage = path: overrides:
    let f = import path;
    in f ((builtins.intersectAttrs (builtins.functionArgs f) allPkgs) // overrides);
  pkgs = with nixpkgs; {
    mkDerivation = import ./autotools.nix nixpkgs;
    hello = callPackage ./hello.nix { };
    graphviz = callPackage ./graphviz.nix { };
    graphvizCore = callPackage ./graphviz.nix { gdSupport = false; };
  };
in pkgs

Let's examine this in detail:

Note how easily we overrode arguments in the case of graphviz without gd. In addition, note how easy it was to merge two repositories: nixpkgs and our pkgs!

The reader should notice a magic thing happening. We're defining pkgs in terms of callPackage, and callPackage in terms of pkgs. That magic is possible thanks to lazy evaluation: builtins.intersectAttrs doesn't need to know the values in allPkgs in order to perform intersection, only the keys that do not require callPackage evaluation.

The "callPackage" pattern has simplified our repository considerably. We were able to import packages that require named arguments and call them automatically, given the set of all packages sourced from nixpkgs.

We've also introduced some useful builtin functions that allows us to introspect Nix functions and manipulate attributes. These builtin functions are not usually used when packaging software, but rather act as tools for packaging. They are documented in the Nix manual.

Writing a repository in Nix is an evolution of writing convenient functions for combining the packages. This pill demonstrates how Nix can be a generic tool to build and deploy software, and how suitable it is to create software repositories with our own conventions.

In the next pill, we will talk about the "override" design pattern. The graphvizCore seems straightforward. It starts from graphviz.nix and builds it without gd. In the next pill, we will consider another point of view: starting from pkgs.graphviz and disabling gd?