[Haskell-cafe] monadic MapReduce

[Anish Muttreja]

On Mon, Mar 02, 2009 at 04:10:41PM +0100, Manlio Perillo wrote:
> Anish Muttreja ha scritto:
>> On Sun, Mar 01, 2009 at 07:25:56PM +0100, Manlio Perillo wrote:
>>> Hi.
>>>
>>> I have a function that do some IO (take a file path, read the file,   
>>> parse, and return some data), and I would like to parallelize it, so  
>>> that multiple files can be parsed in parallel.
>>>
>>> I would like to use the simple mapReduce function,
>>> from Real Word Haskell:
>>>
>>> mapReduce :: Strategy b    -- evaluation strategy for mapping
>>>           -> (a -> b)      -- map function
>>>           -> Strategy c    -- evaluation strategy for reduction
>>>           -> ([b] -> c)    -- reduce function
>>>           -> [a]           -- list to map over
>>>           -> c

>>>
>>> mapReduce mapStrat mapFunc reduceStrat reduceFunc input =
>>>     mapResult `pseq` reduceResult
>>>   where mapResult    = parMap mapStrat mapFunc input
>>>         reduceResult = reduceFunc mapResult `using` reduceStrat
>>>
>>> Is this possible?
>>>
>>>
>>> Thanks  Manlio Perillo
>>
>> Would this work?
>
> I suspect that it will not work..
>
>> Read in each file into a string (or byteString) using a lazy function
>> and then call mapReduce with the strings instead of file paths. 
>>
>> import qualified Data.Bytestring.Lazy.Char8 as L
>> do
>>     let handles =  map (openFile ) files
>>     strings <- mapM L.hGetContents handles
>>     let result = mapReduce ... 
>>
>> The actual work of reading in the file should happen on-demand inside 
>> the parsing function called by mapReduce.
>>
>
> By doing this I will probably lose any control about file resources usage.

OK.

How about this. Is there a reason why I can't 
replace the variables b and c in the type signature of mapReduce with with (IO b') 
and (IO c'). b and c  can be any types. 

mapReduce :: Strategy (IO b')    -- evaluation strategy for mapping
           -> (a -> IO b')      -- map function
           -> Strategy (IO c')    -- evaluation strategy for reduction
           -> ([IO b'] -> (IO c'))    -- reduce function
           -> [a]           -- list to map over
           -> (IO c')

Just remember to wrap all values back in the IO monad.

Anish

>
>
> Thanks  Manlio