[Haskell-cafe] standard poll/select interface

[Bulat Ziganshin]

Hello John,

Thursday, February 09, 2006, 3:19:30 AM, you wrote:

>> JM> If we had a good standard poll/select interface in System.IO then we
>> JM> actually could implement a lot of concurrency as a library with no
>> JM> (required) run-time overhead. I'd really like to see such a thing get
>> JM> into the standard. Well, mainly it would just be a really useful thing
>> JM> to have in general. If others think it is a good idea I can try to come
>> JM> up with a suitable API and submit it to the repo.
>> 
>> i have delayed answering to this letter until i announced my Streams
>> library. now i can say that such API already exists - in terms of my
>> library you need just to write an transformer that intercepts
>> vGetBuf/vPutBuf calls and pass them to the select/poll machinery. so
>> you can write such transformer just now and every program that uses
>> Streams will benefit from its usage. Converting programs that use
>> Handles to using Streams should be also an easy task.

JM> I was actually asking for something much more modest, which was the
JM> routine needed to pass them to the select/poll machinery. but yeah, what
JM> you say is one of my expected uses of such a routine. Once a standard IO
JM> library settles down, then I can start working on the exact API such a
JM> routine would have.

but if all will wait while the library settles down, it will never
occur :)  your work can change design of library, like the my library
itself can change the shape of haskell' :)  at this moment, i just
developed the library which satisfy demands in extending current I/O
library by new features, such as Unicode support, high speed,
portability to other compilers, binary i/o, i/o for packed strings,
and asynchronous i/o using methods other than select(). but i don't
implement all these features actually, i just developed
infrastructure, in which all these features can be easily added.
unlike the System.IO library, you don't need to ask someone to
implement new features or make corrections in foreign sources. you
just need to develop module what implements this standard Stream
interface and then it can be used as easy as transformers from the
library itself

as i understand this idea, transformer implementing async i/o should
intercept vGetBuf/vPutBuf calls for the FDs, start the appropriate
async operation, and then switch to another Haskell threads. the I/O
manager thread should run select() in cycle and when the request is
finished, wake up the appropriate thread. what's all. if you will ever
need, this implementation can then be used to extend GHC's System.IO
internals with the support for new async i/o managers (as i
understand, select() is now supported by GHC, but poll(), kqueue() is
not supported?). the only difference that my lib gives an opportunity
to test this implementation without modifying GHC I/O internals, what
is somewhat simpler. so, interface for async vGetBuf/vPutBuf routines
should be the same as for read/write:

type FD = Int
vGetBuf_async :: FD -> Ptr a -> Int -> IO Int
vPutBuf_async :: FD -> Ptr a -> Int -> IO Int

i think that implementations for ghc and jhc should be slightly
different, though, because of different ways to implement
multi-threading. but the I/O manager should be the same - it just
receives info about I/O operations to run and returns information
about completed ones.

... well, this I/O manager should implement just one operation:

performIO :: Request -> IO ()

type Request = (IOType, FD, Ptr a, Int, Notifier)
data IOType = Read | Write | ...
type Notifier = Result -> IO ()
data Result = OK Int | Fail ErrorInfo

"performIO" starts new I/O operation. On the completion of this
operation, Notifier is called with information about results of
execution.

so, for the GHC the following should work:

vGetBuf_async fd ptr size = do
    done <- newMVar
    let notifier = putMVar done ()
    performIO (Read, fd, ptr, size, notifier)
    takeMVar done

for JHC, the body of "vGetBuf_async" may be different

if you will find this interface reasonable, at least for the first
iteration, i will develop appropriate transformer, so for you remains
"only" the implementation of "performIO"

>> of course, Streams library is not some standard just now, and moreover
>> - it is not compatible with JHC. the greatest problem is what i using
>> type classes extensions available in GHC/Hugs what is not in H98
>> standard. so, i'm interested in pushing Haskell' to accept most
>> advanced possible extensions in this area and, of course, in actual
>> implementing these extensions in the Haskell compilers. alternative
>> way to make Streams available to wider range of Haskell compilers is
>> to strip support of streams working in monads other that IO.

JM> Don't take the absence of a feature in jhc to mean I don't like or want
JM> that feature. There are a lot of things I don't have but that I'd
JM> definitly want to see in the language simply because I was only shooting
JM> for H98 to begin with and was more interested in a lot of the back end
JM> stuff. You should figure out the nicest design that uses just the
JM> extensions needed for the design you want. it could help us decide what
JM> goes into haskell-prime to know what is absolutely needed for good
JM> design and what is just nice to have.

this simply means that the Streams library cannot be used with JHC,
what is bad news, because it is even more rich than GHC's System.IO.
jhc had chance to get modern I/O library. but it lost that chance :)

>> if you can make select/poll transformer, at least for testing
>> purposes, that will be really great.

JM> Yeah, I will look into this. the basic select/poll call will have to be
JM> pretty low level, but hopefully it will allow interesting higher level
JM> constructs based on your streams or an evolution of them.

please look. at this moment Sreams library lacks only a few important
features, already implemented in GHC's System.IO: sockets, line
buffering and async i/o. moreover, i don't have an experience in
implementing the async i/o, so foreign help is really necessary

addressing these three issues will allow to propose the Streams
library as possible System.IO replacement. and as you can see,
implementing the "performIO" will allow us to use async i/o for all
possible i/o operations, including "get/put_" or vGetContents, for
example 


-- 
Best regards,
 Bulat                            mailto:bulatz at HotPOP.com