[Haskell-cafe] ANNOUNCE managing cabal dependencies using nix and hack-nix

Marc Weber marco-oweber at gmx.de
Tue Nov 24 23:41:30 EST 2009


First of all: This is a release (very) early and often release
announcment. So expect that there are some minor glitches [5].
Linux is only supported at this moment. [1]

summary: 
========

  main goal: provide an environment (= list dependencies) to build a
       haskell package with minimal efforts.

  The two repositories

    git://github.com/MarcWeber/haskell-nix-overlay.git
    git://github.com/MarcWeber/hack-nix.git

  enable you to build dependencies required to build a specific cabal
  project automatically. In some way its similar to what cabal-install
  provides. However haskell packages are installed using the nix package
  manager which allows you installing arbitrary versions of the same
  packages at the same time on the same system. 

  Follow their READMEs to get latest updates.

what is nix?
============
See http://nixos.org. Brief: its a package manager utilizing a lazy,
interpreted language called nix to describe package dependencies.
Packages can be installed into user environments. Atomic updates and
rollbacks are some of its key features as well as allowing the user /
or system to install additional versions of a package without breaking
previously installed software. [2] I won't go into details here.

repo haskell-nix-overlay:
=====================
This contains a sketchy implementation of a dependency solver for cabal
packages using a brute force algorithm [6].
It also contains a set of patches to make hackage packages compile.

hack-nix creates a copy of the hackage index file 00-index.tar called
hackage/hackage-nix-db.nix which can be read by the nix language
applying some optimizations.
You run the solver by telling it about your target
packages and their cabal flags. [3]

      globalFlags =  {
        base4 = true;
      }
      packageFlags = {
        Cabal-1.4.4.splitBase = true;
      };
      targetPackages = [
        "happy"
        "HAppS-Server"
        "haskell-src"
      ];

  I only pasted the most important parameters here.
  

So what is this all for?
Many existing package managers assume that there is only one true way to
install packages - using the versions which got packaged.
You want to switch from base-3 to base-4 ? You want to use both?
Some package build with parsec-2 others require parsec-3?
Out of luck(?) Maybe you end up using different users, installing all
packages manually into local (user) databases. Also determining which
packages must be recompiled is a tedious job. So you updated a package
and you have recompiled depending packages. Then you notice that you
want to fix a small bug in an app you didn't update for ages.. however
the old versions of your libraries have been replaced with newer ones..

usage example
==============
Let me introduce you to a small hack-nix workflow showing you how I
think you should use it:

# get a cabal package.
vcs clone xxxx

# tell hack-nix about package profiles. A profile is a name associated
# with cabal flags:
echo "default:-split-base" > .hack-nix-cabal-config

# run hack-nix to build all dependencies required by xxxx [3]
# and create a file exporting PATH and GHC_PACKAGE_PATH
# you should source that in your bash or zsh session
hack-nix --build-env default
source hack-nix-envs/default/source-me/haskell-env
ghc --make Setup.hs
./Setup configure # optionally repeat the cabal flags here [4]
./Setup build

# the package bulids? fine. Now you can register it at your nix
# configuration file ~/.nixpkgs/config.nix after creating a dist file
# and an unmodified representation of the cabal file:
hack-nix --to-nix
cat >> ~/.nixpkgs/config.nix << EOF
    {pkgs, ... }: {
      hackNix.additionalPackages = [
        (import $PATH_TO_CABAL_REPO/dist/$CABAL_PROJECT_NAME.nix)
        # repeat the line above for each package
      ];
    }
EOF

>From now on nix will know about your (dev version?) of the pkg xxxx and it can be
used when running hack-nix --build-env in other cabal directories.
Let's compare it to cabal install --user here:
  base -> lib -> otherlib -> target

you build a dev version of lib.
When running hack-nix --build-env default in the target cabal directory
otherlib will be rebuild automatically.

nix can also create tag files for you. This is implemented but it's not
supported by hack-nix --build-env yet. (TODO)

Now you want to clean those many builds get rid of them and tidy up.
Just run nix-collect-garbage to remove all packages managed by nix
which are no longer referenced by any environment or their previous
generations. Generations? This means that if updating an environment
causes more trouble to than it solves you can temporarily switch back to
the environment you were using before.

How to get started?
===================
See [5], read those READMEs. Tell me about your problems.
Right now HAppS-Server and hack-nix can be built easily.
More work has to be done to make all packages compile.
This usally means replacing "base < 4" by "base < 5" or such.
Another cause of failure is that C-lib dependencies such as
gtk, zlib or happ & alex  have to be added manually once only.

issues:

  nix will install its own glibc which dosen't cope with all kernels.
  You an use kind of native builds. I don't have experiences using them.
  I'm using nixos so I don't care anyway.

  speed: for speed reasons I propase only adding latest hackage
  packages. However you can have different databases or
  a filter can be implemented to disable certain packages
  to speed up calculations. Last resort: reimplement it in C
  putting code aside nix interpreter code.

random notes:
  you can build many packages at the same time utilizing multiple cores
  using the nix-env -j flag

  of course many processes are started to build each package.
  So it definitely is a lot slower than using cabal-install which
  doesn't spawn processes for each cabal built.

  Maybe cabal-install and haskell-nix-overlay can be made sharing some
  patches.

  [..]

If you're interested reply or contact me on irc to make me post about
updates and major changes.


Thanks for listening and for any feedback in advance.
Marc Weber

NOTES:

[1]: Of course nix runs on cygwin as well. However nothing has been
tested so quite some effort has to be spent making this work on Windows
as well.

[2]: If you have a dependency chain
glibc -> lib -> executable
           ` -> executable2
  
and lib got some security reinstalling executable will yield:

glibc -> lib   -> executable 
     `-> lib'  -> executable2

so you have to update executable to make it use executable2 as well.

This all is done by putting each packgae into its own
/nix/store/hash-name/ directory where hash is based on all build inputs.
You can think about those hashes similar you do when thinking about git
hashes. If a patch / package changes all hashes of depending packages /
patches will change. If you want to have more details read about nix on
the official homepage.G

[3]
hack-nix --build-env does a lot of stuff:
- it reads the cabal file and writes a nix file containing the cabal
  contents
- it creates more nix files in hack-nix-envs/nix
  Example: http://github.com/MarcWeber/hack-nix/tree/master/hack-nix-envs/nix/
  I committed those so that you can bootstrap hack-nix.
  default9.nix is the nix representation of the cabal package.
  However the version has been changed to 9999. This is done
  to identify your development version of this package.
  Thus default.nix wants to build this version:
      targetPackages = [{ n = "hack-nix"; v = "99999"; }];
  hack-nix itself isn't built. But it's used to gather its
  dependencies.
- then it realises that env derivation. Right now the solver will
  output many debugging lines to help you find out why it failed finding
  a solution. You can this off by debugS = false;

[4] this can be done more convenient using scion.

[5] help about nix: try #nixos on irc.freenode.net or its mailinglist.
    help about hack-nix: contact me on irc or by mail. I will try to
    keep the READMEs up to date.

[6] At the first glance this sounds bad because:
    Each package can have different flag settings defining a different
    sets of dependencies. Each dependency declaration be satisfied by
    different versions of a package. So the amount of solutions to be
    inspected explodes. Nix isn't the best fit to write such a solver
    so it's quite slow. However limiting the ways a package can be built
    by
    - only adding latest package versions from hackage
    - setting some global flags (such as use base-4.*)
    seems to make the system fast enough for everydays use.

    I'm sorry that I didn't manage to find a better algorithm in
    reasonable time.

If I had to say thank you to some people this list would include dccouts
(for his support on Cabal), Saizan (who helped me by talking to me), the
nix(os) dev team, nominolo (for his work on scion), and many others I
don't even know about.


More information about the Haskell-Cafe mailing list