[Haskell-cafe] Storing big datasets
volhovm.cs at gmail.com
Fri May 6 20:55:07 UTC 2016
Thank you for the answer!
1. acid-state, yes.
2. The application is supposed to work on the regular pc, without 32gb
of ram. I agree that it doesn't cost much to buy 32gb of ram, but it's
not the case.
3. Come to think of it, i think it would be more like about 10k elements
per 10-20 second, yes. On average.
On 05/06/2016 11:41 PM, Bardur Arantsson wrote:
> On 05/06/2016 10:28 PM, Mikhail Volkhov wrote:
>> I'm using ACID package as main database -- it's simple and... ACID
>> (which is cool of course).
> Just for clarification: Do you mean acid-state?
>> So now I need to store up to ~30GB of data in a single data structure
>> (!) that will be constantly updated (some kind of a huge tree set).
>> Here's a question -- how to operate that big structure?
>> 1. It doesn't even fit in RAM
> Yes it does. My desktop machine has 32GB RAM.
> I'm not trying to brag or anything, and *you* may not have that amount
> of memory, but it *should* be cheap enough to buy enough RAM if it
> avoids radically redesigning an existing system (with all the associated
> risk, developer time, etc.).
> If this is a one-off or rare occurrence, you could get 160GB RAM on an
> Amazon M4 instance (m4.10xlarge) if you wanted to. (Amazon's pricing
> structure is incredibly opaque and I didn't do a thorough investigation,
> so I apologize if this is wrong.)
>> 2. It should be updated atomically and frequently (every 10 seconds up
>> to 500 elements out of 10^7).
> This is a bit vague. Do you mean that an *average* of 500 (out of 10^7)
> will be updated every 10 seconds?
>> 3. What structures should I use? I'd like to store up to 10^6~10^7 some
>> simple elements there too, that will be gigabytes of data. So it seems
>> to me I can't use Data.Set.
> Well, if storing things in RAM is out of the question, so is using Data.Set.
> Haskell-Cafe mailing list
> Haskell-Cafe at haskell.org
M3#38 ITMO study group 17'
Computer Technologies Department
More information about the Haskell-Cafe