[Haskell-cafe] binary IO

[Joel Reymont]

Tomasz,

Try http://wagerlabs.com/timeleak.tgz. See the "Killer pickler  
combinators" thread as well.

My desired goal is to have 4k bots (threads?) running at the same  
time. At, say, 1k/s per bot I figure something like 4Mb/s round-trip.  
Each bot cannot spend more than a couple of seconds on pickling/ 
unpickling. I'm not even close to hitting that goal even reading from  
a file.

I'm getting delays of 4s+ with just 100-200 bots reading from a file  
and even less than that in a networked environment. The more bots I  
run the higher the delays, to the point of frequent delays of 10s+.  
The kicker is that some packets come in at 4k compressed with Zlib  
but become something like 50k uncompressed and then expand to a list  
of 500+ records, etc.

Each bot is given 5, 15 or 35 seconds to respond by the poker server  
and this is proving to be too little for my Haskell implementation.

All the data comes in little-endian and there are a lot of zero- 
terminated unicode strings (each short needs to be swapped). My  
customer is running FreeBSD/Intel so swapping should not affect them  
but their performance is that much different than mine. It could well  
be that the scheduler is playing a role in this also.

I spent 3 hard months (barely any weekends, no holidays) polishing my  
Haskell code. I started the Erlang rewrite at about 3pm this saturday  
and about 1.5 workdays later I have 80% of what I need. I expect to  
finish the rest today and will have a good yardstick to measure  
Haskell against.

It took me ~2 hours of adapting the Pickler Combinators paper to  
Erlang. See http://wagerlabs.com/erlang/pickle.erl. Scroll to the  
bottom for examples. I spent the rest of the time converting the 150+  
records to the new format.

Once I deliver my rewritten project I would like to setup a Haskell  
vs. Erlang pickling contest to see if Haskell can match up and figure  
out what is preventing it from doing so if it can't. Then I lend a  
hand to add whatever is missing.

My only requirement is that there be a _single_ spec for pickling and  
unpickling, i.e. no separate methods. The following is not acceptable  
to me ;-).

puTableInfo :: PU TableInfo
puTableInfo = PU fn1 fn2 fn3
     where fn1 ptr ptr ti =
               do ptr <- appP endianW64 ptr $ ti_1 ti
                  ptr <- appP endianW16 ptr $ ti_2 ti
                  ptr <- appP endianW16 ptr $ ti_3 ti
                  ptr <- appP w8 ptr $ ti_4 ti
                  ptr <- appP wstring ptr $ ti_5 ti
                  ptr <- appP endianW32 ptr $ ti_6 ti
                  ptr <- appP (enum endianW16 :: PU GameType) ptr $  
ti_7 ti
                  ptr <- appP endianW16 ptr $ ti_8 ti
                  ptr <- appP bool ptr $ ti_9 ti
                  ptr <- appP endianW64 ptr $ ti_10 ti
                  ptr <- appP endianW64 ptr $ ti_11 ti
                  ptr <- appP endianW64 ptr $ ti_12 ti
                  ptr <- appP endianW64 ptr $ ti_13 ti
                  ptr <- appP endianW16 ptr $ ti_14 ti
                  ptr <- appP (enum w8) ptr $ ti_15 ti
                  ptr <- appP endianW32 ptr $ ti_16 ti
                  ptr <- appP (enum w8) ptr $ ti_17 ti
                  ptr <- appP endianW32 ptr $ ti_18 ti
                  ptr <- appP (list endianW32 w8) ptr $ ti_19 ti
                  ptr <- appP endianW32 ptr $ ti_20 ti
                  return $! ptr
           fn2 ptr ptr =
               do (ti1, ptr) <- appU endianW64 ptr
                  (ti2, ptr) <- appU endianW16 ptr
                  (ti3, ptr) <- appU endianW16 ptr
                  (ti4, ptr) <- appU w8 ptr
                  (ti5, ptr) <- appU wstring ptr
                  (ti6, ptr) <- appU endianW32 ptr
                  (ti7, ptr) <- appU (enum endianW16 :: PU GameType) ptr
                  (ti8, ptr) <- appU endianW16 ptr
                  (ti9, ptr) <- appU bool ptr
                  (ti10, ptr) <- appU endianW64 ptr
                  (ti11, ptr) <- appU endianW64 ptr
                  (ti12, ptr) <- appU endianW64 ptr
                  (ti13, ptr) <- appU endianW64 ptr
                  (ti14, ptr) <- appU endianW16 ptr
                  (ti15, ptr) <- appU (enum w8) ptr
                  (ti16, ptr) <- appU endianW32 ptr
                  (ti17, ptr) <- appU (enum w8) ptr
                  (ti18, ptr) <- appU endianW32 ptr
                  (ti19, ptr) <- appU (list endianW32 w8) ptr
                  (ti20, ptr) <- appU endianW32 ptr
                  return $!
                         (TableInfo ti1 ti2 ti3 ti4 ti5 ti6 ti7 ti8 ti9
                                    ti10 ti11 ti12 ti13 ti14 ti15 ti16
                                    ti17 ti18 ti19 ti20, ptr)

	Joel


On Dec 27, 2005, at 8:44 AM, Tomasz Zielonka wrote:

> What do you mean by too slow? What is the desired input/output
> throughput?
>
> Maybe I'll find some time to improve my BinaryParser/BinaryUnparser
> code.

--
http://wagerlabs.com/