[Haskell-cafe] Storing big datasets

[Lana Black]

Hi Mikhail,

Have you considered external database engines? I suppose you could benefit from using Redis in your case.

  Original Message  
From: Mikhail Volkhov
Sent: Friday, May 6, 2016 8:28 PM
To: haskell-cafe at haskell.org
Subject: [Haskell-cafe] Storing big datasets

Hi!

I'm using ACID package as main database -- it's simple and... ACID
(which is cool of course).

So now I need to store up to ~30GB of data in a single data structure
(!) that will be constantly updated (some kind of a huge tree set).
Here's a question -- how to operate that big structure?
1. It doesn't even fit in RAM
2. It should be updated atomically and frequently (every 10 seconds up
to 500 elements out of 10^7).
3. What structures should I use? I'd like to store up to 10^6~10^7 some
simple elements there too, that will be gigabytes of data. So it seems
to me I can't use Data.Set.

Thanks for any ideas!
-- 
Volkhov Mikhail
M3#38 ITMO study group 17'
Computer Technologies Department
_______________________________________________
Haskell-Cafe mailing list
Haskell-Cafe at haskell.org
http://mail.haskell.org/cgi-bin/mailman/listinfo/haskell-cafe