[Haskell-cafe] Re: RFC: A standardized interface between web servers and applications or frameworks (ala WSGI)

[Johan Tibell]

First, apologies for not responding earlier.  I spent my week at a
conference in Austria.  Second, thanks for all the feedback!

I thought I go through some of my thoughts on the issues raised.  Just
to try to reiterate the goals of this effort:

* To provide a common, no frills interface between web servers and
  applications or frameworks to increase choice for application
  developers.

* To make that interface easy enough to implement so current web
  servers and frameworks will implement it.  This is crucial for it
  being adopted.

* Avoid design decisions that would limit the number of frameworks
  that can use the interface.  One example of a limiting decisions
  would be one that limits the maximal possible performance by using
  e.g. inefficient data types.

I'll try to start with what seems to be the easier issues.

sendfile(2) support
===================

I would like see this supported in the interface.  I didn't include it
in the first draft as I didn't have a good idea of where to put it.
One idea would be to add the following field to the Environment
record:

sendfile :: Maybe (FD -> IO ())

Possibly with additional parameters as needed.  The reason that
sendfile needs to be included in the environment instead of just a
binding to the C function is that the Socket used for the connection
is hidden from the application side and its use is abstracted by the
input and output enumerators.

The other suggested solution (to return either an Enumerator or a file
descriptor) might work better.  I just wanted to communicate that I
think it should be included.

Extension HTTP methods
======================

I did have extension methods in mind when I wrote the draft but didn't
include it.  I see two possible options.

1. Change the HTTP method enumeration to:

data Method = Get | ... | ExtensionMethod ByteString

2. Treat all methods as bytestrings:

type Method = ByteString

This treatment touches on the discussion on typing further down in
this email.  I still haven't thought enough about the consequences (if
indeed there are any of any importance) of the two approaches.

The Enumerator type
===================

To recap, I proposed the following type for the Enumerator abstraction:

type Enumerator = forall a. (a -> ByteString -> IO (Either a a)) -> a -> IO a

The IO monad is a must both in the return type of the Enumerator
function and in the iteratee function (i.e. the first parameter of the
enumartor).  IO in the return type of the enumerator is a must since
the server must perform I/O (i.e. reading from the client socket) to
provide the input and the application might need to perform I/O to
create the response.  The appearance of the IO monad in the iteratee
functions is an optimization.  It makes it possible for the server or
application to act immediately when a chunk of data is received.  This
saves memory when large files are being sent as they can be written to
disk/network immediately instead of being cached in memory.

There are some different design (and possibly performance trade-offs)
that could be made.  The current enumerator type can be viewed as an
unrolled State monad suggesting that it would be possible to change
the type to:

type Enumerator = forall t. MonadTrans t => (ByteString -> t IO
(Either a a)) -> t IO a

which is a more general type allowing for an arbitrary monad stack.
Some arguments against doing this:

* The unrolled state version is analogous to a left fold (and can
  indeed be seen as one) and should thus be familiar to all Haskell
  programmers.

* A, possibly unfounded, worry I have is that it might be hard to
  optimize way the extra abstraction layer putting a performance tax
  on all applications, whether they use the extra flexibility or not.

It would be great if any of the Takusen authors (or Oleg since he
wrote the enumerator paper) could comment on this.

Note: I haven't thought this one through.  It was
suggested to me on #haskell and I thought I should at least bring it
up.

Extra environment variables
===========================

I've intended all along to include a field for remaining, optional
extra pieces of information taken from e.g. the web server, the shell,
etc.  I haven't come up with an good name for this field by the idea
is to add another field to the Environment:

data Environment = Environment
    { ...
    , extraEnvironment :: [(ByteString, ByteString)]
    }

Typing and data types
=====================

Most discussions seem to, perhaps unsurprisingly, have centered around
the use of data types and typing in general.  Let me start by giving
an assumptions I've used when writing this draft:

Existing frameworks already have internal representations of the
request URL, headers, etc.  Changing these would be costly.  Even if
this was done I don't think it is possible to pick any one type that
all frameworks could use to represent an HTTP requests or even parts
of a request.  Different frameworks need different types.  Let me as
an example use the Last-Modified header field.  Assume we used named
record fields for all headers:

data Headers = Headers
    { ...
    , lastModified :: Maybe ???
    }

There is no type we could use for ??? that would be useful for all
frameworks.  There are several possible DateTime types possible with
different design trade-offs.  On the other hand, all frameworks likely
already have a function to convert raw bytes to whatever internal
representation used in that particular framework.

Trying to provide more structured types that bytestrings appears to
have two drawbacks:

1. It adds boiler plate type conversion code.  No benefit is gained by
   the extra typing.  Defining more types in the WAI interface adds
   complexity to the interface.

2. It adds an unnecessary performance penalty.

My suggestions is this:

We use a minimal number of types in the interface and leave it up to
higher levels to add these.

Summary
=======

I suggest that the overall design principle should be this:

Give a data type (e.g. Environment) with a minimal amount of structure
corresponding to the one given in the HTTP RFC plus some extra
optional environment provided by the web server and the environment
(e.g. shell) in which it is run.  I suggest we leave the raw
bytestrings in the interface as interpreting them is best done by the
framework.  This also lends itself to an efficient implementation as
the bytestrings in the environment could just be substrings (an O(1)
operation) of the raw input read from the socket.

Let me make a slight reservation here and say that I might want to
split the URL into two parts (e.g. SCRIPT_NAME and PATH_INFO, like in
CGI and WSGI).  The reason for doing this is that it makes it much
easier to nest applications by having each layer consuming one part of
the URL and leave the rest to the nested application.  For example,
consider the task of writing an URL dispatcher that picks different
applications depending on the URL prefix:

storeApp, adminApp, urlMap :: Application

urlMap = mkUrlMap
    [("/admin", adminApp)
    ,("/store", storeApp)
    ]

serve :: Application -> IO ()  -- Provided by the web server.

main = serve urlMap

When a request for /store/items/1 reaches the URL mapper application
it consumes the initial prefix (and puts it in scriptName) and leaves
the remaining URL part in pathInfo.  adminApp or storeApp can then use
what's left in pathInfo to do further dispatching (to a handler
function for example).

Phew, this turned into a longer email than I thought.  If I forgot to
respond to any points raised please don't be afraid to raise them
again.