[Haskell-cafe] binary IO

[Joel Reymont]

On Dec 27, 2005, at 4:52 PM, Bulat Ziganshin wrote:

> spending several weeks to random
> optimization is like buying a gold computer case trying to speed up
> the game :)

I did not spend several weeks on optimization. I went through about  
25 iterations with the timeleak code and the profiler. I watched the  
contribution of each function to the overall time and tried to  
improve that contribution.

> sorry, but your negative results say more about you than about GHC or
> Haskell. you are working 3 months but still don't know what is a
> bottleneck in your program!

You are right in that I spent the first few weeks learning. By now I  
know that pickling is the bottleneck, though. The timeleak code is  
very simple. It forks X threads where each thread opens a file for  
reading. Each thread reads the file record by record and sounds an  
alarm if the time taken to unpickle a record exceeds, say, 3 seconds.

There's also a clear relationship between the number of threads  
launched and the length of the delay so concurrency has an effect as  
well.

I'm waiting for Simon Marlow to come back to take a look at it with him.

> i say about timer thread. you created complex design using Map to find
> first event to complete, but instead you can use just 3 timer threads,
> each one serving a fixed timing interval. each thread will serve
> requests just in the order they arrive

I use just one timer thread right now, profiling does not tell me  
that timers are a problem. Your solution is also not demonstrably  
superior so I'm not pressed to try it.

> it's strange what your pocker server don't required the same pickler
> library as client. this 50k structure is constructed on the server
> end, i'm right?

I'm working with a different poker server now. Mine actually uses  
lots of small packets that are easy to process. This particular  
server is way different.

> no, type classes allow code economy comparing to combinators you use.
> you need to define only one time how to process each type. just one
> definition for all lists, one for all Word16, one for all GameType and
> so on. all the records defined by the easy code like this one for
> TableInfo

Quite possibly. I'm looking for hard proven results at this point and  
while you are making bold claims about the throughput of your  
pickling library you are not willing to demonstrate this throughput  
in a real-life situation. I cannot wager on your approach as it's not  
proven. I'm going with a proven solution for now (Erlang) but will  
come back to Haskell in a pinch as soon as I know "how to do the  
right thing".

This discussion is a bit of a dead-end unless you are willing to take  
the next step and apply your solution to my timeleak repro case. If  
you or someone else is willing to do that than I can continue my set  
of profiler reports and eventually get some closure. It will happen  
once either 1) the last bit of performance is squeezed out of  
pickling and it's determined that threading or the scheduler is the  
bottleneck or 2) things work out nicely.

I carefully kept track of what changes I made and how they affected  
speed, memory consumption, etc. I maxed out on the optimizations I  
have been able to make without dropping pickler combinators  
altogether. I no longer have the time to continue this but since a  
lot of people are concerned with Haskell speed I'm suggesting that  
someone takes over with their "best of the breed" solution to the  
problem and uses the same process of saving profiling reports and  
tracking changes until they optimize to the max or give up.

Then we can lay out the series of profiling reports in a storyboard  
of sorts, with changes from report to report described. This would  
serve a great "how to write efficient Haskell" manual.

	Joel

--
http://wagerlabs.com/