[web-devel] Overhaul of wai-handler-devel

[Michael Snoyman]

Hi all,

I think it's no secret that wai-handler-devel is not living up to its full
potential. Here's a few of the issues that I know about:

* Doesn't integrate well with cabal-dev. (Fair warning: I haven't yet
started using cabal-dev.)
* It reinterprets the entire application, instead of just the files that
changed and their dependencies.
* It has no knowledge of your cabal file, and so simply tries to use the
most recent version of each package available. A few people this week
noticed problems running wai-handler-devel since WAI 0.4 was released but is
not yet supported by Yesod.

Some of these issues can be solved while still sticking with Hint. However,
that will require reimplementing a significant amount of work that already
exists in other places. I think our best bet is to re-evaluate entirely what
we need, and think about the best way to get there. Let's start by pointing
out why "cabal build && ./dist/build/myapp/myapp" isn't good enough:

* Linking an executable is *very* slow.
* You have to manually kill your application and rerun the above command
each time you change a file.
* If an external file included by Template Haskell, such as a Hamlet
template, is modified, cabal does not know to recompile that module.

But the fact is that the simple cabal build approach *does* solve all of the
issues we currently have with wai-handler-devel. (Well, at least the three I
mention above, feel free to chime in here.) So here's an idea: let's try to
build on top of cabal so we can take advantage of its features. As for the
points listed above, I'll address them in reverse order.

wai-handler-devel already contains code that determines which "extra" files
are actually dependencies for a module. For example, it looks for hamletFile
through your source code and, when it finds it, notes that the specified
file is a dependency. We can easily write a tool that checks the timestamps
on all the files, determines if a file needs to be updated, and modifies
timestamps accordingly. For example, if Handler/Root.hs uses
hamlet/root.hamlet, and the latter has a newer modification date than the
former, the tool would change the former's modification date to the same as
the latter.

As far as manually killing applications and rerunning the compile steps,
that's also very simple: we simply have a monitor program that sees if
source files have been modified since the last build. wai-handler-devel
already does this. We can optimize this on Linux with inotify, and the
equivalent on other operating systems, but I think it's fair to say that
this is a solved problem.

The big question is how to avoid long links. The simple answer is to skip
them: we don't actually need an executable, just the object files. We'll
then use plugins[1] to hot-load the code... except that plugins seems to be
(1) buggy and (2) not cross-platform. We can try out direct-plugins[2]
instead, but it seems that it will require us to install the package into
GHC's database, which isn't necessarily a good idea. (Perhaps we can ask Dan
very nicely to make some API changes to help us here?) The third route is
writing the hot-loading code ourselves. Andy Stewart pointed out to me that
manatee[3] does this directly against the GHC API.

My guess: our best bet is to ask Dan (direct-plugins's author) to work with
us on augmenting his package with the features we need. This can have
ramifications outside the realm of wai-handler-devel, such as having a
server that hot-loads code for individual applications. (I was thinking of
this for Lambda Engine actually.) But before starting down that route, I'd
love to hear if others have any opinions on the subject.

Sorry for the long email :)

Michael

[1] http://hackage.haskell.org/package/plugins
[2] http://hackage.haskell.org/package/direct-plugins
[3] http://hackage.haskell.org/package/manatee
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://www.haskell.org/pipermail/web-devel/attachments/20110408/5344e250/attachment.htm>