[Haskell-cafe] Re: HDBC or HSQL

Thomas Hartman tphyahoo at gmail.com
Sun Oct 19 09:34:18 EDT 2008


>>    I am asking because I am trying to make HAppS a reasonable replacement
>>    for all contexts in which you would otherwise use an external
>> relational
>>    database except those in which an external SQL database is a specific
>>    requirement.

My experience with HAppS so far suggests that if one's data set is in
the millions, an external database -- or some sort of off-ram storage
mechanism is indeed a requirement, because even a 16GB server will
eventually run out of memory.

For example, I have a demo app (happstutorial.com) that stores jobs or
users as records. On average, each record takes about 100B of memory,
when checkpointed on hard disk. Not sure how much this amounts to in
ram, but but on a 256M workstation with 256M of virtual memory, the
checkpointing mechanism had an out of memory error with under 500,000
records.

This doesn't seem totally unreasonable to me: 500,000 * 100 =
50,000,000B, which is 20% of my physical memory and 10% of my total
(including virtual) memory.

Also, as my recent post to haskell cafe says, I can't even fit a 20M
element Data.Map.Map Int Int object into memory on my 256M laptop.

On a 16GB server, you might get 32X the limit I ran into, but you will
still have out of memory errors with > 15 million records.

Perhaps a recordset in the millions isn't a "reasonable" requirement,
but most web 2.0 type projects are of this scope, I think.

Thomas.

2007/8/4 Alex Jacobson <alex at alexjacobson.com>:
> Have you looked at the HAppS.DBMS.IxSet?  It gives you a type safe way to
> query indexed collections.
>
> -Alex-
>
> Isto Aho wrote:
>>
>> Hi,
>>
>> I'd like to store small matrices into a db. Number of rows and columns may
>> vary in a way not
>> known in advance. One might use a relation (matrixId, col, row, value) or
>> something like that
>> but if it is possible to put a matrix in one command into db, some queries
>> will be easier.
>> E.g., one relation can store several matrices and it would be easy to
>> query, how many
>> matrices are stored currently. With that above four tuple you can find out
>> the number of unique
>> matrixId's, too, but it is not as easy as with matrices.
>>
>> Anyhow, now I'm not sure if I should stick with HSQL any more... Earlier
>> comments on this
>> thread made me think that maybe it would be a better idea to try to learn
>> enough HDBC.
>>
>> This would be used in a server application. Is HAppS applicable here?
>>
>> e.g. after some tweaking the following works with HSQL:
>>
>> addRows = do
>>        dbh <- connect server database user_id passwd
>>        intoDB dbh ([555,111, 50, 1000]::[Int])
>> ([21.0,22.0,23.0,24.0]::[Double])
>>        intoDB dbh ([556,111, 50, 1000]::[Int])
>> ([21.0,22.0,23.0,24.0]::[Double])
>>        intoDB dbh ([]::[Int]) ([]::[Double])
>>   where
>>        intoDB dbh i_lst d_lst =
>>                catchSql (do
>>                        let cmd = "INSERT INTO trial (intList, dList)
>> VALUES (" ++
>>                                toSqlValue i_lst ++ "," ++ toSqlValue d_lst
>> ++ ")"
>>                        execute dbh cmd
>>                        )
>>                        (\e -> putStrLn $ "Problem: " ++ show e)
>>
>>
>> Similarly, queries can handle matrices and I like that it is now
>> possible to select those columns or rows from the stored matrix that
>> are needed.  E.g.
>>
>> retrieveRecords2 :: Connection -> IO [[Double]]
>> retrieveRecords2 c = do
>>        -- query c "select dList[1:2] from trial" >>= collectRows getRow
>>        query c "select dList from trial" >>= collectRows getRow
>>        where
>>                getRow :: Statement -> IO [Double]
>>                getRow stmt = do
>>                        lst   <- getFieldValue stmt "dList"
>>                        return lst
>> readTable2 = do
>>        dbh <- connect server database user_id passwd
>>        values <- retrieveRecords2 dbh
>>        putStrLn $ "dLists are : " ++ (show values)
>>
>>
>> br,
>> Isto
>>
>>
>> 2007/8/1, Alex Jacobson <alex at alexjacobson.com
>> <mailto:alex at alexjacobson.com>>:
>>
>>    Out of curiosity, can I ask what you are actually trying to do?
>>
>>    I am asking because I am trying to make HAppS a reasonable replacement
>>    for all contexts in which you would otherwise use an external
>> relational
>>    database except those in which an external SQL database is a specific
>>    requirement.
>>
>>    -Alex-
>>
>
> _______________________________________________
> Haskell-Cafe mailing list
> Haskell-Cafe at haskell.org
> http://www.haskell.org/mailman/listinfo/haskell-cafe
>


More information about the Haskell-Cafe mailing list