<div dir="ltr"><br><br><div class="gmail_quote">On Thu, Apr 16, 2015 at 1:57 PM Duncan Coutts <<a href="mailto:duncan@well-typed.com">duncan@well-typed.com</a>> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">On Thu, 2015-04-16 at 10:32 +0000, Michael Snoyman wrote:<br>
> On Thu, Apr 16, 2015 at 1:12 PM Duncan Coutts <<a href="mailto:duncan@well-typed.com" target="_blank">duncan@well-typed.com</a>> wrote:<br>
><br>
> > On Thu, 2015-04-16 at 09:52 +0000, Michael Snoyman wrote:<br>
> > > Thanks for responding, I intend to go read up on TUF and your blog post<br>
> > > now. One question:<br>
> > ><br>
> > > * We're incorporating an existing design for incremental updates<br>
> > > of the package index to significantly improve "cabal update"<br>
> > > times.<br>
> > ><br>
> > > Can you give any details about what you're planning here?<br>
> ><br>
> > Sure, it's partially explained in the blog post.<br>
> ><br>
> > > I put together a<br>
> > > Git repo already that has all of the cabal files from Hackage and which<br>
> > > updates every 30 minutes, and it seems that, instead of reinventing<br>
> > > anything, simply using `git pull` would be the right solution here:<br>
> > ><br>
> > > <a href="https://github.com/commercialhaskell/all-cabal-files" target="_blank">https://github.com/commercialhaskell/all-cabal-files</a><br>
> ><br>
> > It's great that we can mirror to lots of different formats so<br>
> > easily :-).<br>
> ><br>
> > I see that we now have two hackage mirror tools, one for mirroring to a<br>
> > hackage-server instance and one for S3. The bit I think is missing is<br>
> > mirroring to a simple directory based archive, e.g. to be served by a<br>
> > normal http server.<br>
> ><br>
> > From the blog post:<br>
> ><br>
> > The trick is that the tar format was originally designed to be<br>
> > append only (for tape drives) and so if the server simply<br>
> > updates the index in an append only way then the clients only<br>
> > need to download the tail (with appropriate checks and fallback<br>
> > to a full update). Effectively the index becomes an append only<br>
> > transaction log of all the package metadata changes. This is<br>
> > also fully backwards compatible.<br>
> ><br>
> > The extra detail is that we can use HTTP range requests. These are<br>
> > supported on pretty much all dumb/passive http servers, so it's still<br>
> > possible to host a hackage archive on a filesystem or ordinary web<br>
> > server (this has always been a design goal of the repository format).<br>
> ><br>
> > We use a HTTP range request to get the tail of the tarball, so we only<br>
> > have to download the data that has been added since the client last<br>
> > fetched the index. This is obviously much much smaller than the whole<br>
> > index. For safety (and indeed security) the final tarball content is<br>
> > checked to make sure it matches up with what is expected. Resetting and<br>
> > changing files earlier in the tarball is still possible: if the content<br>
> > check fails then we have to revert to downloading the whole index from<br>
> > scratch. In practice we would not expect this to happen except when<br>
> > completely blowing away a repository and starting again.<br>
> ><br>
> > The advantage of this approach compared to others like rsync or git is<br>
> > that it's fully compatible with the existing format and existing<br>
> > clients. It's also in the typical case a smaller download than rsync and<br>
> > probably similar or smaller than git. It also doesn't need much new from<br>
> > the clients, they just need the same tar, zlib and HTTP features as they<br>
> > have now (e.g. in cabal-install) and don't have to distribute<br>
> > rsync/git/etc binaries on other platforms (e.g. windows).<br>
> ><br>
> > That said, I have no problem whatsoever with there being git or rsync<br>
> > based mirrors. Indeed the central hackage server could provide an rsync<br>
> > point for easy setup for public mirrors (including the package files).<br>
> ><br>
> ><br>
> ><br>
> I don't like this approach at all. There are many tools out there that do a<br>
> good job of dealing with incremental updates. Instead of using any of<br>
> those, the idea is to create a brand new approach, implement it in both<br>
> Hackage Server and cabal-install (two projects that already have a massive<br>
> bug deficit), and roll it out hoping for the best.<br>
<br>
I looked at other incremental HTTP update approaches that would be<br>
compatible with the existing format and work with passive http servers.<br>
There's one rsync-like thing over http but the update sizes for our case<br>
would be considerably larger than this very simple "get the tail, check<br>
the secure hash is still right". This approach is minimally disruptive,<br>
compatible with the existing format and clients.<br>
<br>
> There's no explanation here as to how you'll deal with things like<br>
> cabal file revisions, which are very common these days and seem to<br>
> necessitate redownloading the entire database in your proposal.<br>
<br>
The tarball becomes append only. The tar format works in this way;<br>
updated files are simply appended. (This is how incremental backups to<br>
tape drives worked in the old days, using the tar format). So no, cabal<br>
file revisions will be handled just fine, as will other updates to other<br>
metadata. Indeed we get the full transaction history.<br>
<br>
> Here's my proposal: use Git. If Git isn't available on the host, then<br>
> revert to the current codepath and download the index. We can roll that out<br>
> in an hour of work and everyone gets the benefits, without the detriments<br>
> of creating a new incremental update framework.<br>
<br>
I was not proposing to change the repository format significantly (and<br>
only in a backwards compatible way). The existing format is pretty<br>
simple, using standard old well understood formats and protocols with<br>
wide tool support.<br>
<br>
The incremental update is fairly unobtrusive. Passive http servers don't<br>
need to know about it, and clients that don't know about it can just<br>
download the whole index as they do now.<br>
<br>
The security extensions for TUF are also compatible with the existing<br>
format and clients.<br>
<br><br></blockquote><div><br></div><div>The theme you seem to be creating here is "compatible with current format." You didn't say it directly, but you've strongly implied that, somehow, Git isn't compatible with existing tooling. Let me make clear that that is, in fact, false[1]:</div><div><br></div><div>```</div><div><div>#!/bin/bash</div><div><br></div><div>set -e</div><div>set -x</div><div><br></div><div>DIR=$HOME/.cabal/packages/<a href="http://hackage.haskell.org">hackage.haskell.org</a></div><div>TAR=$DIR/00-index.tar</div><div>TARGZ=$TAR.gz</div><div><br></div><div>git pull</div><div>mkdir -p "$DIR"</div><div><br></div><div>rm -f $TAR $TARGZ</div><div><br></div><div>git archive --format=tar -o "$TAR" master</div><div>gzip -k "$TAR"</div></div><div>```</div><div><br></div><div>I wrote this in 5 minutes. My official proposal is to add code to `cabal` which does the following:</div><div><br></div><div>1. Check for the presence of the `git` executable. If not present, download the current tarball</div><div>2. Check for existence of ~/.cabal/all-cabal-files (or similar). If present, run `git pull` inside of it. If absent, clone it</div><div>3. Run the equivalent of the above shell script to produce the 00-index.tar file (not sure if the .gz is also used by cabal)</div><div><br></div><div>This seems like such a drastically simpler solution than using byte ranges, modifying Hackage to produce tarballs in an append-only manner, and setting up cabal-install to stitch together and check various pieces of a downloaded file.<br><br>I was actually planning on proposing this some time next week. Can you tell me the downsides of using Git here, which seems to fit all the benefits you touted of:<br><br>> pretty simple, using standard old well understood formats and protocols with wide tool support.<br><br>Unless Git at 10 years old isn't old enough yet.</div><div><br></div><div>Michael<br><br>[1] <a href="https://github.com/commercialhaskell/all-cabal-files/commit/133cd026f8a1f99d719d97fcf884372ded173655">https://github.com/commercialhaskell/all-cabal-files/commit/133cd026f8a1f99d719d97fcf884372ded173655</a> </div></div></div>