new i/o library

[Bulat Ziganshin]

Hello Simon,

Wednesday, February 01, 2006, 2:26:45 PM, you wrote:

SM> Bulat Ziganshin wrote:

i can now report about memory-mapped files in Windows. seems that
their support is not intended for implementing sequential file access
but exclusively for database-like files what be hold in memory as much
as possible and requires random access. at least, i used buffers of
64k and unmap them as soon as each buffer is filled. nevertheless,
windows delayed writing to disk already filled and unmapped buffers as
much as possible - even prefer to swap to disk OS and running programs
rather than flushung this data (when memory was filled by 80% with the
data of m/m file)! when i tried to sequentially read this file, the
results was slower than for simple read() calls, and i think that is
because data was not prefetched despite sequential access. i will
implement this stream type ultimately, but it will be more for some
special purposes than fir everyday use or an FD replacement

>> moreover - we can implement locking as special "converter" type, that
>> can be applied to any mutable object - stream, collection, counter.
>> that allows to simplify implementations and add locking only to those
>> Streams where we really need it. like these:
>> 
>> h <- openFD "test"
>>        >>= addUsingOfSelect
>>        >>= addBuffering 65536
>>        >>= addCharEncoding utf8
>>        >>= attachUserData dictionary
>>        >>= addLocking

SM> This is really nice - exactly what I'd like to see in the I/O library. 
SM> The trick is making it perform well, though...  but I'm sure that's your 
SM> main focus too.

basically idea is very simple - every stream implements the Stream
interface, what is clone of Handle interface. Stream transformers is
just a types what has Stream parameters and in turn implement the same
interface. all Stream operations are translated to calls of inner
Stream. typical example:

data WithLocking h = WithLocking h (MVar ())

-- | Add lock to object to ensure its proper use in concurrent threads
withLocking h = do
    mvar <- newMVar ()
    return (WithLocking h mvar)

instance (Stream IO h) => Stream IO (WithLocking h) where
    vMkIOError (WithLocking h _)    = vMkIOError h
    vClose (WithLocking h mvar)     = withMVar mvar (\_ -> vClose h)
    vIsEOF (WithLocking h mvar)     = withMVar mvar (\_ -> vIsEOF h)
    vReady (WithLocking h mvar)     = withMVar mvar (\_ -> vReady h)
    vSeek  (WithLocking h mvar) a b = withMVar mvar (\_ -> vSeek h a b)
    ...............

as a result, these Stream Transformers can be applied recursively,
forming a complex type that is still a proper Stream! for example,
type of 'h' in the

          h <- openRawFD "test" WriteMode
                   >>= bufferBlockStream
                   >>= withEncoding utf8
                   >>= withLocking

will be "WithLocking (WithEncoding (BufferedBlockStream FD))"


moreover, this locking transformer is universal - it can be applied,
for example, to IOArray and resulting value will still implement
MArray interface! it's one of reasons for that i popularize using class
interfaces for all standard libraries and especially data structures

SM> Still, I'm not sure that putting both input and output streams in the 
SM> same type is the right thing, I seem to remember a lot of things being 
SM> simpler with them separate.

i'm interested to hear that things was simpler? in my design it seems
vice versa - i will need to write far more code to separate read,
write and read-write FDs, for example. may be, the difference is
because i have one large Stream class that implements all the
functions while your design had a lot of classes with a few functions
in each


-- 
Best regards,
 Bulat                            mailto:bulatz at HotPOP.com