nixpkgs/doc/manual/development.xml

<chapter xmlns="http://docbook.org/ns/docbook"
         xmlns:xlink="http://www.w3.org/1999/xlink">

<title>Development</title>

<para>This chapter has some random notes on hacking on
NixOS.</para>


<section>

<title>Extending NixOS</title>

<para>A unique syntax is used to express all system, hardware, computer and
  service configurations.  This syntax helps for reading and writing new
  configuration files.  It is coming with some extra strategies defined in
  NixPkgs which are used to merge and evaluate all configuration files.</para>

  <para>A configuration file is the same as your own computer
  configuration.</para>

<example xml:id='conf-syntax'><title>Usual configuration file</title>
<programlisting>
{config, modulesPath, pkgs, ...}: <co xml:id='conf-syntax-1' />

let
  inherit (pkgs.lib) mkOption mkIf mkThenElse types; <co xml:id='conf-syntax-2' />

  cfg = config.services.locate; <co xml:id='conf-syntax-4' />
  locatedb = "/var/cache/locatedb";
  logfile = "/var/log/updatedb";
  cmd = "root  updatedb --localuser=nobody --output=${locatedb} > ${logfile}";
in

{
  imports = [ <co xml:id='conf-syntax-6' />
    (modulesPath + /services/scheduling/cron.nix)
  ];

  options = { <co xml:id='conf-syntax-3' />
    services.locate = {
      enable = mkOption {
        default = false;
        example = true;
        type = types.bool;
        description = ''
          If enabled, NixOS will periodically update the database of
          files used by the <command>locate</command> command.
        '';
      };

      period = mkOption {
        default = "15 02 * * *";
        type = types.uniq type.string;
        description = ''
          This option defines (in the format used by cron) when the
          locate database is updated.
          The default is to update at 02:15 (at night) every day.
        '';
      };
    };
  };

  config = mkIf cfg.enable { <co xml:id='conf-syntax-5' />
    services.cron = {
      systemCronJobs = mkThenElse { <co xml:id='conf-syntax-7' />
        thenPart = "${cfg.period}  root ${cmd}";
        elsePart = "";
      };
    };
  };
}</programlisting>
</example>

<para><xref linkend='conf-syntax' /> shows the <emphasis>configuration
  file</emphasis> for the locate service which uses cron to update the
  database at some dates which can be defined by the user.  This nix
  expression is coming
  from <filename>upstart-jobs/cron/locate.nix</filename>.  It shows a simple
  example of a service that can be either distributed on many computer that
  are using the same configuration or to shared with the community.  This
  file is divided in two with the interface and the implementation.  Both
  the interface and the implementation declare a <emphasis>configuration
  set</emphasis>.

<calloutlist>

  <callout arearefs='conf-syntax-1'>

    <para>This line declares the arguments of the configuration file.  You
    can omit this line if there is no reference to <varname>pkgs</varname>
    and <varname>config</varname> inside the configuration file.</para>

    <para>The argument <varname>pkgs</varname> refers to NixPkgs and allow
    you to access all attributes contained inside it.  In this
    example <varname>pkgs</varname> is used to retrieve common functions to
    ease the writing of configuration files
    like <varname>mkOption</varname>, <varname>mkIf</varname>
    and <varname>mkThenElse</varname>.</para>

    <para>The argument <varname>config</varname> corresponds to the whole
    NixOS configuration.  This is a set which is build by merging all
    configuration files imported to set up the system.  Thus all options
    declared are contained inside this variable.  In this
    example <varname>config</varname> is used to retrieve the status of
    the <varname>enable</varname> flag.  The important point of this
    argument is that it contains either the result of the merge of different
    settings or the default value, therefore you cannot assume
    that <option>config.services.locate.enable</option> is always false
    because it may have been defined in another configuration file.</para>

  </callout>

  <callout arearefs='conf-syntax-2'>

    <para>This line is used to import a functions that are useful for
    writing this configuration file.</para>

  </callout>

  <callout arearefs='conf-syntax-3'>

    <para>The variable <varname>options</varname> is
    a <emphasis>configuration set</emphasis> which is only used to declare
    options with the function <varname>mkOption</varname> imported
    from <filename>pkgs/lib/default.nix</filename>.  Options may contained
    any attribute but only the following have a special
    meaning: <varname>default</varname>, <varname>example</varname>,
    <varname>description</varname>, <varname>merge</varname>
    and <varname>apply</varname>.</para>

    <para>The <varname>merge</varname> attribute is used to merge all values
    defined in all configuration files and this function return a value
    which has the same type as the default value.  If the merge function is
    not defined, then a default function
    (<varname>pkgs.lib.mergeDefaultOption</varname>) is used to merge
    values.  The <varname>merge</varname> attribute is a function which
    expect two arguments: the location of the option and the list of values
    which have to be merged.</para>

    <para>The <varname>apply</varname> attribute is a function used to
    process the option.  Thus the value return
    by <option>config.<replaceable>option</replaceable></option> would be
    the result of the <varname>apply</varname> function called with either
    the <varname>default</varname> value or the result of
    the <varname>merge</varname> function.</para>

  </callout>

  <callout arearefs='conf-syntax-4'>

    <para>This line is a common trick used to reduce the amount of
    writing.  In this case <varname>cfg</varname> is just a sugar over
    <option>config.services.locate</option></para>

  </callout>

  <callout arearefs='conf-syntax-5'>

    <para>This line is used to declare a special <emphasis>IF</emphasis>
    statement.  If you had put a usual <emphasis>IF</emphasis> statement
    here, with the same condition, then you will get an infinite loop.  The
    reason is that your condition ask for the value of the
    option <option>config.services.locate.enable</option> but in order to
    get this value you have to evaluate all configuration sets including the
    configuration set contained inside your file.</para>

    <para>To remove this extra complexity, <varname>mkIf</varname> has been
    introduced to get rid of possible infinite loop and to factor your
    writing.</para>

  </callout>

  <callout arearefs='conf-syntax-6'>

    <para>The attribute <varname>imports</varname> contains a list of other
    module location.  These modules should provide option declarations for
    the current module in order to be evaluated safely.</para>

    <para>When a dependence on a NixOS module has to be made, then you
    should use the argument <varname>modulesPath</varname> to prefix the
    location of your NixOS repository.</para>

  </callout>

  <callout arearefs='conf-syntax-7'>

    <para>The attribute <varname>config</varname>, which should not be
    confused with the argument of the same name, contains a set of option
    definitions defined by this module. When there is
    neither <varname>imports</varname> nor <varname>options</varname>, then
    this attribute set can be return without any enclosing.  This feature
    allow you to write your <filename>configuration.nix</filename> as a
    module which does not contains any option declarations.</para>

    <para>As <varname>mkIf</varname> does not need
    any <emphasis>then</emphasis> part or <emphasis>else</emphasis> part,
    then you can specify one on each option definition with the
    function <varname>mkThenElse</varname>.</para>

    <para>To avoid a lot of <varname>mkThenElse</varname> with empty
    <emphasis>else</emphasis> part, a sugar has been added to infer the
    corresponding <emphasis>empty</emphasis>-value of your option when the
    function <varname>mkThenElse</varname> is not used.</para>

    <para>If your <emphasis>then</emphasis> part
    and <emphasis>else</emphasis> part are identical, then you should use
    the function <varname>mkAlways</varname> to ignore the condition.</para>

    <para>If you need to add another condition, then you can add <varname>mkIf</varname> to on
    the appropriate location.  Thus the <emphasis>then</emphasis> part will
    only be used if all conditions declared with <varname>mkIf</varname>
    are satisfied.</para>

  </callout>

</calloutlist>

</para>

</section>

<section>

<title>Building specific parts of NixOS</title>

<para>

<screen>
$ nix-build /etc/nixos/nixos -A <replaceable>attr</replaceable></screen>

where <replaceable>attr</replaceable> is an attribute in
<filename>/etc/nixos/nixos/default.nix</filename>.  Attributes of interest include:

<variablelist>

  <varlistentry>
    <term><varname>config</varname></term>
    <listitem><para>The computer configuration generated from
    the <envar>NIXOS_CONFIG</envar> environment variable (default
    is <filename>/etc/nixos/configuration.nix</filename>) with the NixOS
    default set of modules.</para></listitem>
  </varlistentry>

  <varlistentry>
    <term><varname>system</varname></term>
    <listitem><para>The derivation which build your computer system.  It is
    built by the command <command>nixos-rebuild
    build</command></para></listitem>
  </varlistentry>

  <varlistentry>
    <term><varname>vm</varname></term>
    <listitem><para>The derivation which build your computer system inside a
    virtual machine.  It is built by the command <command>nixos-rebuild
    build-vm</command></para></listitem>
  </varlistentry>
</variablelist>

</para>

<para>
Most parts of NixOS can be build through the <varname>config</varname>
attribute set.  This attribute set allows you to have a view of the merged
option definitions and all its derivations.  Important derivations are store
inside the option <option>system.build</option> and can be listed with the
command <command>nix-instantiate --xml --eval-only /etc/nixos/nixos -A
config.system.build</command>
</para>

</section>


<section>

<title>Building your own NixOS CD</title>

<para>Building a NixOS CD is as easy as configuring your own computer. The
idea is to use another module which will replace
your <filename>configuration.nix</filename> to configure the system that
would be install on the CD.</para>

<para>Default CD/DVD configurations are available
inside <filename>nixos/modules/installer/cd-dvd</filename>.  To build them
you have to set <envar>NIXOS_CONFIG</envar> before
running <command>nix-build</command> to build the ISO.

<screen>
$ export NIXOS_CONFIG=/etc/nixos/nixos/modules/installer/cd-dvd/installation-cd-minimal.nix
$ nix-build /etc/nixos/nixos -A config.system.build.isoImage</screen>

</para>

<para>Before burning your CD/DVD, you can check the content of the image by mounting anywhere like
suggested by the following command:

<screen>
$ mount -o loop -t iso9660 ./result/iso/cd.iso /mnt/iso</screen>

</para>

</section>

<section>

<title>Testing the installer</title>


<para>Building, burning, and booting from an installation CD is rather
tedious, so here is a quick way to see if the installer works
properly:

<screen>
$ export NIXOS_CONFIG=/etc/nixos/nixos/modules/installer/cd-dvd/installation-cd-minimal.nix
$ nix-build /etc/nixos/nixos -A config.system.build.nixosInstall
$ dd if=/dev/zero of=diskimage seek=2G count=0 bs=1
$ yes | mke2fs -j diskimage
$ mount -o loop diskimage /mnt
$ ./result/bin/nixos-install</screen>

</para>

</section>


<section>

<title>Testing the <literal>initrd</literal></title>

<para>A quick way to test whether the kernel and the initial ramdisk
boot correctly is to use QEMU’s <option>-kernel</option> and
<option>-initrd</option> options:

<screen>
$ nix-build /etc/nixos/nixos -A config.system.build.initialRamdisk -o initrd
$ nix-build /etc/nixos/nixos -A config.system.build.kernel -o kernel
$ qemu-system-x86_64 -kernel ./kernel/bzImage -initrd ./initrd/initrd -hda /dev/null
</screen>

</para>

</section>

<section>

  <title>Whole-system testing using virtual machines</title>

  <para>
    Complete NixOS GNU/Linux systems can be tested in virtual machines
    (VMs).  This makes it possible to test a system upgrade or
    configuration change before rebooting into it, using the
    <command>nixos-rebuild build-vm</command> or
    <command>nixos-rebuild build-vm-with-bootloader</command> command.
  </para>

  <para>
    <!-- The following is adapted from
         http://wiki.nixos.org/wiki/NixOS_VM_tests, by Eelco Dolstra. -->

    The <filename>tests/</filename> directory in the NixOS source tree
    contains several <emphasis>whole-system unit tests</emphasis>.
    These tests can be run<note><para>NixOS tests can be run both from
    NixOS and from a non-NixOS GNU/Linux distribution, provided the
    Nix package manager is installed.</para></note> from the NixOS
    source tree as follows:

    <screen>
      $ nix-build tests/ -A nfs.test
    </screen>

    This performs an automated test of the NFS client and server
    functionality in the Linux kernel, including file locking
    semantics (e.g., whether locks are maintained across server
    crashes).  It will first build or download all the dependencies of
    the test (e.g., all packages needed to run a NixOS VM). The test
    is defined in <link
    xlink:href="https://svn.nixos.org/repos/nix/nixos/trunk/tests/nfs.nix">
    <filename>tests/nfs.nix</filename></link>.  If the test succeeds,
    <command>nix-build</command> will place a symlink
    <filename>./result</filename> in the current directory pointing at
    the location in the Nix store of the test results (e.g.,
    screenshots, test reports, and so on).  In particular, a
    pretty-printed log of the test is written to
    <filename>log.html</filename>, which can be viewed using something
    like:

    <screen>
      $ icecat result/log.html
    </screen>
  </para>

  <para>
    It is also possible to run the test environment interactively,
    allowing you to experiment with the VMs.  For example:

    <screen>
      $ nix-build tests/ -A nfs.driver
      $ ./result/bin/nixos-run-vms
    </screen>

    The script <command>nixos-run-vms</command> starts the three
    virtual machines defined in the NFS test using QEMU/KVM.  The root
    file system of the VMs is created on the fly and kept across VM
    restarts in
    <filename>./<varname>hostname</varname>.qcow2</filename>.
  </para>

  <para>
    Finally, the test itself can be run interactively.  This is
    particularly useful when developing or debugging a test:

    <screen>
      $ nix-build tests/ -A nfs.driver
      $ ./result/bin/nixos-test-driver
      starting VDE switch for network 1
      >
    </screen>

    Perl statements can now be typed in to start or manipulate the
    VMs:

    <screen>
      > startAll;
      (the VMs start booting)
      > $server->waitForJob("nfs-kernel-nfsd");
      > $client1->succeed("flock -x /data/lock -c 'sleep 100000' &");
      > $client2->fail("flock -n -s /data/lock true");
      > $client1->shutdown;
      (this releases client1's lock)
      > $client2->succeed("flock -n -s /data/lock true");
    </screen>

    The function <command>testScript</command> executes the entire
    test script and drops you back into the test driver command line
    upon its completion.  This allows you to inspect the state of the
    VMs after the test (e.g. to debug the test script).
  </para>

  <para>
    This and other tests are continuously run on <link
    xlink:href="http://hydra.nixos.org/jobset/nixos/trunk/with-status">the
    Hydra instance at <literal>nixos.org</literal></link>, which
    allows developers to be notified of any regressions introduced by
    a NixOS or Nixpkgs change.
  </para>

</section>

</chapter>