[Haskell-cafe] binary IO

[Tomasz Zielonka]

On Tue, Dec 27, 2005 at 09:18:54AM +0000, Joel Reymont wrote:
> Try http://wagerlabs.com/timeleak.tgz. See the "Killer pickler  
> combinators" thread as well.

Let's see if I understand correctly. There are 17605 messages in
trace.dat. On my hardware the average message unpicking time is 0.0002s
when you only have a single thread. So, it indeed seems that with 1000
threads it should be possible to process every message in under 4
seconds.

Right now I can think of two reasons:
- 1000 treads need much data in the help, which increases the cost
  of GC and with frequent context switches results in poor cache
  performance
- the GHC's process scheduler is not as intelligent as Erlang's

> I'm getting delays of 4s+ with just 100-200 bots reading from a file  
> and even less than that in a networked environment. The more bots I  
> run the higher the delays, to the point of frequent delays of 10s+.  
> The kicker is that some packets come in at 4k compressed with Zlib  
> but become something like 50k uncompressed and then expand to a list  
> of 500+ records, etc.

One possible solution is to reduce the number of simultaneously running
unpicklings/ picklings (I think someone already proposed it, was that
Bulat?). It would reduce the amount of needed memory and should improve
cache usage.

But then, what will be the delay observed by the server?

Anyway, I've made some tests, and I see the problem persists. I'll try
to test with my unpickling code.

> Each bot is given 5, 15 or 35 seconds to respond by the poker server  
> and this is proving to be too little for my Haskell implementation.

This is per message, right?

> My only requirement is that there be a _single_ spec for pickling and  
> unpickling, i.e. no separate methods. The following is not acceptable  
> to me ;-).

What if the spec was the data type itself? When I was dealing with a
proprietary Intel-based binary format, I derived my picklers /
unpicklers with TH from data type definitions. Of course, there were
cases were the derived code would be wrong, and then I had to write the
code myself, but it was for about 2-3 record types out of total 30.

Best regards
Tomasz

-- 
I am searching for a programmer who is good at least in some of
[Haskell, ML, C++, Linux, FreeBSD, math] for work in Warsaw, Poland