What's next?

[Johan Tibell]

Hi Simon,

(The wiki is currently down so I have to try to remember what was on that page.)

Let me start by saying that I'm in principle for moving to a Nix-style
package DB (i.e. multiple instances of the same package and version).
The Cabal UI is designed to support hermetic builds without exposing
how it's implemented (sandboxes or Nix style package DB) to the user
too much.

The problem is at least harder to solve than sandboxing. The reason I
had Mikhail implement sandboxes during GSoC is that I was skeptical
that we could make the Nix-solution work during a GSoC project, not
because the student didn't do a good job but because it's hard and
requires lots of expertise. If we magically could find for Duncan and
Simon M to work on it, I think we could do it. I personally don't have
time to do any cabal work except for managing the project.

The main downside with sandboxes compared to Nix-style package DBs is
that every new sandbox will induce one long build when all
dependencies are built. This takes a couple of minutes. After that
build, sandboxes and Nix packages should behave about the same
(correct me if I'm wrong). At that point incremental build times
matter more, as they make the edit-compile-test cycle longer. Avoiding
extra work (unnecessary linking) and doing things faster (more
parallelism) help here.

Cheers,
Johan

On Thu, Sep 5, 2013 at 12:15 AM, Simon Peyton-Jones
<simonpj at microsoft.com> wrote:
> Can I ask what the Cabal team's position is with respect to the question of allowing the same package to be installed several times, each compiled against a different collection of dependencies?
>
> This is the problem that you built sandboxes to work around; the "What are sandboxes and why are they needed" section of http://coldwa.st/e/blog/2013-08-20-Cabal-sandbox.html describes the problem well.
>
> It would be possible for Cabal to continue to work around deficiencies in GHC, but wouldn't it be better for us to work together to fix the underlying problem once and for all?
>
> Simon wrote a wiki page about this. I think this is it
>         http://ghc.haskell.org/trac/ghc/wiki/Commentary/Packages/MultiInstances
> though it's in a bit less detail than I expected, so there may be something else.
>
> In my limited understanding, this single change would do more to alleviate "cabal hell" so extensively written about than anything else we can do.  It would require changes in both Cabal and GHC.  It can't be that hard... shall we just do it?  Isn't it more important, for our users, than the other things Johan lists below?
>
> Simon
>
>
>
> | -----Original Message-----
> | From: cabal-devel [mailto:cabal-devel-bounces at haskell.org] On Behalf Of
> | Johan Tibell
> | Sent: 05 September 2013 05:14
> | To: cabal-devel at haskell.org
> | Subject: What's next?
> |
> | Hi all,
> |
> | With 1.18 out the door it's time to look towards the future. Here are
> | the major themes I'd like to see us work on next:
> |
> | ## Faster builds
> |
> | There are several interesting things we could do and are doing here.
> |
> |  * Avoid relinking if possible. This reduces incremental build times
> | (i.e. when you run cabal build after making some change) by avoiding
> | relinking e.g. all the test suites and/or benchmarks. See
> | https://github.com/haskell/cabal/pull/1410 for some in-progress work.
> |
> |  * Build components and different ways (e.g. profiling) in parallel.
> | We could build both profiling and non-profiling versions in parallel.
> | We could also build e.g. all test suites in parallel. The key
> | challenge here is to coordinate all parallel jobs so we don't spawn
> | too many. See https://github.com/haskell/cabal/pull/1413
> |
> |  * Build modules in parallel. This fine granularity would let us
> | making building a single package faster, which is the most common case
> | after all. There has been some GSoC work here e.g.
> | http://hackage.haskell.org/trac/ghc/ticket/910 and some work by
> | Mikhail.
> |
> | ## Do the right thing automatically
> |
> | The focus here should be on avoiding manual steps the cabal could do
> | for the user.
> |
> |  * Automatically install dependencies when needed. When `cabal build`
> | would fail due to a missing dependency, just install this dependency
> | instead of bugging the user to do it. This will probably have to be
> | limited to sandboxes where we can't break the user's system
> |
> |  * GHCi support could be improved by rebinding :reload to rerun e.g.
> | preprocessors automatically. This would enable the users to develop
> | completely from within ghci (i.e. faster edit-save-type-error cycle).
> | We have most of what we need here (i.e. GHC macro support) but someone
> | needs to make the final change to generate a .ghci file to pass in the
> | ghci invocation.
> |
> | ## Support large projects
> |
> | We need to better support projects with tens or hundreds of packages.
> | As projects grow it's naturally to split them up into a bunch of
> | packages (libraries and executables). Some changes to the project
> | might need changes across a couple of packages (e.g. to a supporting
> | library and to an executable) in a single change and developers need
> | to be able to
> |
> |  * build several packages that have local changes, and
> |  * share changes to packages with other developers without making a
> | release to some local Hackage server every time.
> |
> | Both can be done by having a single source repo with all the packages
> | and using `cabal sandbox add-source` to make sure they get built when
> | needed. However, that method only scales up to a handful of packages.
> |
> | I think we want to think about moving away from a world where all
> | cabal commands are run in the context of some "current" package (i.e.
> | in the current working directory). I think we want to be able to stand
> | in the root directory of some source repo and build things from there.
> | Example:
> |
> |     $ git clone git://my-company/repo my-project
> |     $ cd my-project
> |     $ ls
> |     lib-pkg1 lib-pkg2 exe-pkg1 exe-pkg2 ...
> |     $ cabal sandbox init  # or something similar
> |     $ edit lib-pkg1/Lib.hs
> |     $ edit exe-pkg1/Exe.hs
> |     $ cabal build exe-pkg1  # picks up changes to lib-pkg1
> |
> | This has implication for many things e.g. where the .cabal-sandbo and
> | the dist directories are kept. Perhaps dist would have a subdirectory
> | per package (right now we do something similar for sandbox
> | dependencies).
> |
> | I imagine that the syntax for e.g. cabal build would have to extended to
> |
> |     cabal build [[DIR':']COMPONENT]
> |
> | Example:
> |
> |     cabal build lib-pkg1:some-component
> |
> | Similar for `cabal test` and `cabal bench`.
> |
> | Cheers,
> | Johan
> |
> | _______________________________________________
> | cabal-devel mailing list
> | cabal-devel at haskell.org
> | http://www.haskell.org/mailman/listinfo/cabal-devel