[Haskell-cafe] Is there some tool for inferring wide dependency ranges?

Thu May 27 22:25:39 UTC 2021

For small packages, which depend on boot libraries only (i.e. the ones which
come with GHC), I suggest you take a look at
https://gitlab.haskell.org/ghc/ghc/-/wikis/commentary/libraries/version-history
(there is link to it from https://matrix.hackage.haskell.org/).
Running `ghc-pkg-<version> list` on command line also works.

I'll use `containers` as an example of a boot library.

It is wise to allow the version of containers which is bundled with the
oldest
GHC you plan to support, so people don't need to reinstall that dependency.

Further, let us assume that GHC-8.0 is the oldest compiler version you
want to
support.  then you look at the table (or `ghc-pkg list` output) and
figure out
that the containers-0.5.7.1 is bundled with GHC-8.0.2.  Therefore
`containers
>=0.5.7.1` is the perfect lower bound.

There is no good reason to allow older versions,
using such configurations is most likely a mistake.

For your second need, I'd suggest that you indeed just repeat yourself.

    constraints: dependency-one ==0.0.1
    constraints: dependency-two ==0.1.2
    ...

and rebuild.

> As I can see both cabal.haskell-ci and quickcheck-instances.cabal specify
> versions, so you got that DRY violation, too.

I don't think so. They are redundant, but if I (or a contributor)
bumps a lower bound in `quickcheck-instances.cabal` that will most likely
result into a build failure.

Also dependencies depend on each other, e.g. changing unordered-containers
bounds most likely may require adjusting hashable bounds: I like to keep my
lower bounds tight, so I'd need to rerun my tool to figure out.
(unordered-containers and hashable is not a good example though as
 unordered-containers lower bounds are very loose)

I may forget, CI catches it. Here redundancy is not bad thing.

Most importantly, lower bounds are quite static.
Just repeat them.

We should note that dependency solving solutions are not "convex", i.e. a
package may have lower bounds bounds

    pkg-a >=1.2,
    pkg-b >=0.4

and while there are install plans with individual dependency at the lower
bound, the `--constraint=pkg-a==1.2 --constraint=pkg-b==0.4` requirement
is not
satisifiable.

Thus in general this problem is hard, and cannot be solved by just
looking at
`.cabal` file. Of course /you/ are free to make your lower bounds "convex",
so such tool would work, but that is not general solution.

Even figuring out the convex solution is hard.
To make all lower bounds satisfiable simultaneously, that tool should
somehow
figure out find out e.g. `pkg-a ==1.2.3` and `pkg ==0.4.5` configuration.
skipping few releases of both packages.

Note: the same problem happens with the upper bounds,
and there cabal solver makes an arbitrary choice (usually picking the
later version of a dependency it considered first - an implementation
detail).
You may think that changing the solver to traverse the dependency
tree in reverse order would work (favouring old versions), but when I
tried that, the run times were miserable, solver starts to backtrack a lot
more.

As I said in my first email, just pick the current package versions you
use as
the lower bounds, and stick to them for a while. Do not overthink. I
made that
mistake myself, you end up doing work no-one benefitted from.
Wider dependency ranges: YAGNI.

One more important note is that to be 100% sure you must test the
intermediate
versions. This is especially true if the version range spans many major
versions, e.g. `lens >=4.10 && <5.1`. Technically, you need to test
4.10, 4.11,
4.12, .. 4.19 and 5. Probably also lowest and highest point release. Luckily
most maintainers are sane and don't do back-and-forth changes, but it did
happen with aeson (0.9, 0.10, 0.11 IIRC, (.:?) behaviour was changed and
reverted), so it can happen to anyone.

(I also remember a case where 0.x.0.0 release accidentally removed some
functionality, an accidental mistake happened while moving stuff around.
That
is fine as it was major release, but as the functionality was then
restored in
0.x.1.0, having continuous version range allowing 0.x.0.0 point is
incorrect).

So you may want to test with multiple constraint sets, complete or not.
One use case if your package has (manual) flags, there you'd have

    constraints: my-package +a-flag

or

    constraints: my-package -a-flag

I need a tool because I maintain dozens of packages, and my workflow is
quite
uniform, but also unique, so I automated parts of it.  I suggest some simple
and flexible "DIY" solutions to you instead, figuring out how to
configure and
use my tool will take you more time. `cabal gen-bounds` using the oldest
GHC you use, and then relaxing upper bounds is a simple way.

- Oleg

On 27.5.2021 23.28, Askar Safin wrote:
> Oleg Grenrus, thanks a lot for big answer! Please, give a link to your tool. Ideally with some free software license.
>
> Let me repeat my needs.
>
> My packages are smaller than lens. Currently I have very small package https://hackage.haskell.org/package/check-cfg-ambiguity . It has two dependencies only: base and containers. And I plan to publish 1-2 packages more, they will be small, too.
>
> I use SourceHut as code hosting and continuous integration platform. CI config is specified in usual "shell script in YAML" form ( https://lobste.rs/s/v4crap/crustaceans_2021_will_be_year_technology#c_ker4jn ) in file ".build.yml". See https://git.sr.ht/~safinaskar/check-cfg-ambiguity/tree/a9fc453eb73250650f3458659344e1b902538027/item/.build.yml for an example.
>
> Ideally, I want two different tools.
>
> First tool should actually compute dependency ranges. It will rebuild my package a lot of times during this computation. I don't plan to run this tool often. I think I will run it one time before each release. So, it is not necessary to put it into continuous integration.
>
> The second tool should simply check that package is compatible with specified range. To do this, the tool should build package exactly two times: one with all specified lower bounds and one with all specified upper bounds. So, there is no any kind of combinatoric explosion. And it is perfectly okay to run such tool in continuous integration at every commit.
>
> Currently I have neither tool.
>
> The second tool is very simple. Well, in some sense I don't need such tool at all, because I can just put to my .build.yml shell commands for building my package with two version sets. But then I have to specify versions two times: one in *.cabal and one in .build.yml. I. e. we got DRY violation. So here is my question: is there some existing tool, which extracts bounds from *.cabal and run that two builds?
>
>> For example haskell-ci allows to specify constraint sets,
>> and I have one for example in quickcheck-instances,
>> https://github.com/haskellari/qc-instances/blob/master/cabal.haskell-ci
> As I can see both cabal.haskell-ci and quickcheck-instances.cabal specify versions, so you got that DRY violation, too.
>
> ==
> Askar Safin
> http://safinaskar.com
> https://sr.ht/~safinaskar
> https://github.com/safinaskar