[Haskell-cafe] Storing big datasets

[Bardur Arantsson]

On 05/06/2016 10:28 PM, Mikhail Volkhov wrote:
> Hi!
> 
> I'm using ACID package as main database -- it's simple and... ACID
> (which is cool of course).
> 

Just for clarification: Do you mean acid-state?

> So now I need to store up to ~30GB of data in a single data structure
> (!) that will be constantly updated (some kind of a huge tree set).
> Here's a question -- how to operate that big structure?
> 1. It doesn't even fit in RAM

Yes it does. My desktop machine has 32GB RAM.

I'm not trying to brag or anything, and *you* may not have that amount
of memory, but it *should* be cheap enough to buy enough RAM if it
avoids radically redesigning an existing system (with all the associated
risk, developer time, etc.).

If this is a one-off or rare occurrence, you could get 160GB RAM on an
Amazon M4 instance (m4.10xlarge) if you wanted to. (Amazon's pricing
structure is incredibly opaque and I didn't do a thorough investigation,
so I apologize if this is wrong.)

> 2. It should be updated atomically and frequently (every 10 seconds up
> to 500 elements out of 10^7).

This is a bit vague. Do you mean that an *average* of 500 (out of 10^7)
will be updated every 10 seconds?

> 3. What structures should I use? I'd like to store up to 10^6~10^7 some
> simple elements there too, that will be gigabytes of data. So it seems
> to me I can't use Data.Set.
> 

Well, if storing things in RAM is out of the question, so is using Data.Set.

Regards,