[Haskell-cafe] Re: Battling time leaks

[Bulat Ziganshin]

Hello Joel,

Wednesday, December 21, 2005, 9:47:19 PM, you wrote:
>> can you say what it exactly means? we are not mastered in your code.
>> some common explanation like "my program takes 6 seconds to
>> deserialize 50kb of data on Pentium4/3ghz" will be more understabdable

JR> That's why I posted the code at http://wagerlabs.com/timeleak.tgz

JR> The alerts are issued when the time of unstuffing exceeds 3 seconds.

this says nothing to me. you must say

1) your end goal - say, "run 1000 threads each 3 seconds"
2) why you can't share result of one unpickling among all threads?
3) your computer
4) unpickling time for one 50kb record


>> if your current problem is deserialization speed, i can give you my
>> own library. it now runs about 500kb/sec on 1ghz processor

JR> My issue is that I 1) have about 250 records and 2) my wire format is  
JR> different from the Haskell representation. Everything arrives to me  
JR> little-endian, for example.

1) you can use TH (may be, Einar will help you. he already have TH
support in its own SerTH library). even without TH, writing Binary
instances, imho, is an order easier than writing pickler functions -
especially when you need only deserialiation :)
2) Binary library use network format (little-endian) on all platforms

but i'm absolutely not sure that your real problem is raw unpickling speed

JR> It looks like pickling is the bottleneck so I'm converting all the  
JR> structures to Storable :(. In the meantime, I'm looking for suggestions.

JR> Einar offered BinSer which lets me use a single spec for the record  
JR> format, conversions included. See http://cs.helsinki.fi/u/ekarttun/ 
JR> haskell/test.hs. I still can't figure out how I would go from a :+: b  
JR> to Foo a b, though.

interesting thing, i will look. i'm rewrote from scratch my serialization
library 2 times, and still want to fully rewrite it again :)

http://cs.helsinki.fi/u/ekarttun/haskell/test.hs

-- 
Best regards,
 Bulat                            mailto:bulatz at HotPOP.com