[Haskell-cafe] streaming with logs / metadata

[Olaf Klinke]

> Am 09.04.2019 um 22:55 schrieb Evan Laforge <qdunkan at gmail.com>:
> 
> On Tue, Apr 9, 2019 at 12:53 PM Olaf Klinke <olf at aatal-apotheke.de> wrote:
>> When you say interleaving, do you think of lists like the following?
>> [Left warning1,Right value 1,Left warning 2,Right value2,...]
>> The above is isomorphic to
>> [(warning1,value1),(warning2,value2),...]
>> 
>> I don't think that the writer monad destroys streaming. Below is a somewhat contrived, but ordinary use case.
> 
> Ah, I see what you mean.  I think I'm using a stricter definition of
> "streaming", what I mean is that I can process without a memory leak,
> in more-or-less constant memory (i.e. what packages like streams and
> pipes mean by it).  With the Writer example, once I do:
> 
>> (ys,warnings) = runWriter (mapM half stream)
> 
> If I print warnings, I'm going to force all of 'ys' into memory.
> Taking one thing off of 'warnings' will force an unknown amount of
> 'ys'.
Yes, but each element in the input stream generates at most one warning (in my example), so as long as you e.g. dump the warnings to stderr or a file and don't retain all warnings in memory, the program runs in constant memory. 

> 
> Another way to look at it is that I can always go from [Either a b] to
> ([a], [b]) just by throwing away the interleaving, but then I can't go
> back since I lost that information.
You still have not clarified how [Either a b] is a type of values with annotations. To me this is a stream of computations, some of which have produced a value (Right b) and some have failed (Left a). Does the following match your intentions?

data StreamElement a b = InfoMessage a | Value b

Then you want streaming functions that process the Values in a stream of such StreamElements, passing on the InfoMessages as soon as they arrive? If an InfoMessage is independent of any Value, then this is the right abstraction. If each InfoMessage is really tied to some Value nearby in the stream, then you should go with pairs, i.e. the writer monad. 

> 
> My instinct is that monads are too general for this, because they
> leave the output type polymorphic.  It's more similar to 'streams',
> but focusing on interleaving "transparent" metadata rather than
> interleaving effects.  And since 'streams' reimplements the whole list
> panopoly I guess that bodes ill for my attempt to avoid doing that
> too.  In fact it could be a layer on top of 'streams', and maybe
> should be, to avoid having to do it all over again if I want effects
> some day...
There are other streaming libraries like pipes and conduit, which are very general and powerful. Maybe one of those can do what you want. But I can't help there because I've never had a use case for them and the syntax diverges somewhat from ordinary lambda calculus. 

Olaf
> 
> I still can't think of a good name though.