[Haskell-cafe] Storing big datasets

[Mikhail Volkhov]

Thank you for the answer!

1. acid-state, yes.
2. The application is supposed to work on the regular pc, without 32gb
of ram. I agree that it doesn't cost much to buy 32gb of ram, but it's
not the case.
3. Come to think of it, i think it would be more like about 10k elements
per 10-20 second, yes. On average.

On 05/06/2016 11:41 PM, Bardur Arantsson wrote:
> On 05/06/2016 10:28 PM, Mikhail Volkhov wrote:
>> Hi!
>>
>> I'm using ACID package as main database -- it's simple and... ACID
>> (which is cool of course).
>>
> 
> Just for clarification: Do you mean acid-state?
> 
>> So now I need to store up to ~30GB of data in a single data structure
>> (!) that will be constantly updated (some kind of a huge tree set).
>> Here's a question -- how to operate that big structure?
>> 1. It doesn't even fit in RAM
> 
> Yes it does. My desktop machine has 32GB RAM.
> 
> I'm not trying to brag or anything, and *you* may not have that amount
> of memory, but it *should* be cheap enough to buy enough RAM if it
> avoids radically redesigning an existing system (with all the associated
> risk, developer time, etc.).
> 
> If this is a one-off or rare occurrence, you could get 160GB RAM on an
> Amazon M4 instance (m4.10xlarge) if you wanted to. (Amazon's pricing
> structure is incredibly opaque and I didn't do a thorough investigation,
> so I apologize if this is wrong.)
> 
>> 2. It should be updated atomically and frequently (every 10 seconds up
>> to 500 elements out of 10^7).
> 
> This is a bit vague. Do you mean that an *average* of 500 (out of 10^7)
> will be updated every 10 seconds?
> 
>> 3. What structures should I use? I'd like to store up to 10^6~10^7 some
>> simple elements there too, that will be gigabytes of data. So it seems
>> to me I can't use Data.Set.
>>
> 
> Well, if storing things in RAM is out of the question, so is using Data.Set.
> 
> Regards,
> 
> 
> _______________________________________________
> Haskell-Cafe mailing list
> Haskell-Cafe at haskell.org
> http://mail.haskell.org/cgi-bin/mailman/listinfo/haskell-cafe
> 

-- 
Volkhov Mikhail
M3#38 ITMO study group 17'
Computer Technologies Department