[Haskell-cafe] Storing big datasets

[David Turner]

Btrees are good for storing data on disk. And something like postgres is an
extremely efficient implementation of a btree supporting atomic updates and
the like. I'd use that!
On 6 May 2016 21:29, "Mikhail Volkhov" <volhovm.cs at gmail.com> wrote:

> Hi!
>
> I'm using ACID package as main database -- it's simple and... ACID
> (which is cool of course).
>
> So now I need to store up to ~30GB of data in a single data structure
> (!) that will be constantly updated (some kind of a huge tree set).
> Here's a question -- how to operate that big structure?
> 1. It doesn't even fit in RAM
> 2. It should be updated atomically and frequently (every 10 seconds up
> to 500 elements out of 10^7).
> 3. What structures should I use? I'd like to store up to 10^6~10^7 some
> simple elements there too, that will be gigabytes of data. So it seems
> to me I can't use Data.Set.
>
> Thanks for any ideas!
> --
> Volkhov Mikhail
> M3#38 ITMO study group 17'
> Computer Technologies Department
> _______________________________________________
> Haskell-Cafe mailing list
> Haskell-Cafe at haskell.org
> http://mail.haskell.org/cgi-bin/mailman/listinfo/haskell-cafe
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.haskell.org/pipermail/haskell-cafe/attachments/20160507/26980403/attachment.html>