[Haskell-cafe] MapReduce reverse engineered

[Daryoush Mehrtash]

>
> since mapReduce assumes that all the code (and the data) is in place in the
> respective nodes.
>


As far as I can tell, the Hadoop, the open source implementation of map
reduce, doesn't require your map reduce code to be in all nodes.  It copies
the jar files of the your application to the nodes that would execute the
code.
http://hadoop.apache.org/core/docs/current/mapred_tutorial.html#Job+Submission+and+Monitoring

Data is also generally available via a network protocol such as http.

I am wondering if  distribution tools (such as Cabal) would be preferable
to  hadoop model.  Instead of having the framework copy the code why not use
the existing tools to install the code prior to running the map-reduce?

daryoush

2009/2/25 Alberto G. Corona <agocorona at gmail.com>

> Galchin,
>
> Maybe you are asking not only about remote execution, but also mobility of
> code. This is a problem that is previous to mapReduce, since mapReduce
> assumes that all the code (and the data) is in place in the respective
> nodes. In fact, the distribution of resources in order to efficiently use
> mapReduce is a design problem that the google people has done by hand.
> But  my intuition says that there are a general algorithm  for distribution
> of  code, data, bandwidth and resources in general that  moves around at
> execution time to achieve better and better performance for a given grid of
> nodes and for any task, for example, a mapReduce task. I would be very
> interesting to read something about this.
>
> I know that some efforts have been carried out the past , for example
> mobile haskell <http://homepages.inf.ed.ac.uk/stg/workshops/TFP/book/DuBois/duboismhaskell/cameraready.pdf%C2%A0>
>
>
> http://homepages.inf.ed.ac.uk/stg/workshops/TFP/book/DuBois/duboismhaskell/cameraready.pdf <http://homepages.inf.ed.ac.uk/stg/workshops/TFP/book/DuBois/duboismhaskell/cameraready.pdf%C2%A0>
>
>
> which is a first step for this goal but I this has been discontinued and
> the source code is not available.
>
> 2009/2/25 Galchin, Vasili <vigalchin at gmail.com>
> >
> > Hello,
> >
> >      Here is an interesting paper of Google's MapReduce reverse
> engineered into Haskell. I apologize if already posted .....
> http://www.cs.vu.nl/~ralf/MapReduce/<http://www.cs.vu.nl/%7Eralf/MapReduce/>
> >
> > Kind regards, Vasili
> >
> > _______________________________________________
> > Haskell-Cafe mailing list
> > Haskell-Cafe at haskell.org
> > http://www.haskell.org/mailman/listinfo/haskell-cafe
> >
>
>
> _______________________________________________
> Haskell-Cafe mailing list
> Haskell-Cafe at haskell.org
> http://www.haskell.org/mailman/listinfo/haskell-cafe
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://www.haskell.org/pipermail/haskell-cafe/attachments/20090225/cd760d86/attachment.htm