[Haskell-cafe] Allocating enormous amounts of memory and wondering why

[Donald Bruce Stewart]

jeff:
> I switched to Data.Binary, which dropped me from 2.6GB to 1.5GB, and
> then I switched this afternoon to unboxed arrays from lists of floats,
> and that dropped me again from 1.5GB to 475MB.  I think, all told, that
> I'm in an acceptable range now, and thank you for pointing out the
> library mistake.  I'm also down from 1.5 minutes load time to under 10
> seconds of load time, which is also very very nice.  Incidentally, the
> code I'm now using is:
> 
> binaryLoadDocumentCoordinates :: 
>   String -> IO (Ptr Float, Array.UArray Int Int)
> binaryLoadDocumentCoordinates path = do
>   putStrLn "File opened"
>   coordinates <- decodeFile (path ++ "/Clusters.bin") :: IO
> (Array.UArray Int Float)
>   print . Array.bounds $ coordinates
>   putStrLn "Got coordinates"
>   galaxies <- decodeFile (path ++ "/Galaxies.bin") :: IO (Array.UArray
> Int Int)
>   putStrLn "Got galaxies"
>   coordinatesArr <- mallocArray . snd . Array.bounds $ coordinates
>   putStrLn "Allocated array"
>   pokeArray coordinatesArr . Array.elems $ coordinates
>   return (coordinatesArr, galaxies)
> 
> binarySaveDocumentCoordinates :: String -> [Point] -> IO ()
> binarySaveDocumentCoordinates path points = do
>   let len = length points
>   encodeFile (path ++ "Clusters.bin") . (Array.listArray (0,len*3) :: 
>     [Float] -> Array.UArray Int Float) . coordinateList . solve $ points
>   encodeFile (path ++ "Galaxies.bin") . (Array.listArray (0,len) :: 
>     [Int] -> Array.UArray Int Int) . galaxyList $ points

I've just pushed a patch to Data.Binary in the darcs version that should
help serialising arrays by avoiding forcing an intermediate list.
You can get that here:

    darcs get http://darcs.haskell.org/binary

I'd still avoid that 'listArray' call though, you may as well just write
the list out, rather than packing it into an array, and then serialising
the array back as a list.

-- Don