<html><head><meta http-equiv="content-type" content="text/html; charset=utf-8"></head><body dir="auto"><div><span></span></div><div><div>Anecdotal support for this idea: This is exactly how we distribute hadron[1]-based Hadoop MapReduce programs to cluster nodes at work. The compiled executable essentially ships itself to the nodes and recognizes the different environment when executed in that context. <br><br>[1] hadron is a haskell hadoop streaming framework that came out of our work. It's on github and close to being released on hackage once the current dev branch is finalized/merged. In case it's helpful: <a href="https://github.com/soostone/hadron">https://github.com/soostone/hadron</a><br><br></div><div>Oz</div><div><br>On Mar 15, 2015, at 8:06 PM, Andrew Cowie <<a href="mailto:andrew@operationaldynamics.com">andrew@operationaldynamics.com</a>> wrote:<br><br></div><blockquote type="cite"><div><div dir="ltr"><div><div><div><div>Bit of a whinger from left-field, but rather than deploying a Main script and then using GHCi, have you considered compiling the program and shipping that?<br><br></div>Before you veto the idea out of hand, statically compiled binaries are good for being almost self-contained, and (depending on what you changed) and they rsync well. And if that doesn't appeal, then consider instead building the Haskell program dynamically; Hello World is only a couple kB; serious program only a hundred or so.<br><br></div>Anyway, I know you're just looking to send a code fragment closure, but if you're dealing with the input and output of the program through a stable interface, then the program is the closure.<br><br></div>Just a thought.<br><br></div>AfC<br></div><br><div class="gmail_quote">On Mon, Mar 16, 2015 at 9:53 AM felipe zapata <<a href="mailto:tifonzafel@gmail.com">tifonzafel@gmail.com</a>> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><div>Hi all,</div><div>I have posted the following question on stackoverflow, but so far I have not received an answer.</div><a href="http://stackoverflow.com/questions/29039815/distributing-haskell-on-a-cluster" target="_blank">http://stackoverflow.com/questions/29039815/distributing-haskell-on-a-cluster</a><br><div><br></div><div><br></div><div><p style="margin:0px 0px 1em;padding:0px;border:0px;font-size:15px;clear:both;font-family:'Helvetica Neue',Helvetica,Arial,sans-serif;line-height:19.5px">I have a piece of code that process files,</p><pre style="margin-top:0px;padding:5px;border:0px;font-size:13px;overflow:auto;width:auto;max-height:600px;font-family:Consolas,Menlo,Monaco,'Lucida Console','Liberation Mono','DejaVu Sans Mono','Bitstream Vera Sans Mono','Courier New',monospace,sans-serif;color:rgb(57,51,24);word-wrap:normal;background-color:rgb(238,238,238)"><code style="margin:0px;padding:0px;border:0px;font-size:13px;font-family:Consolas,Menlo,Monaco,'Lucida Console','Liberation Mono','DejaVu Sans Mono','Bitstream Vera Sans Mono','Courier New',monospace,sans-serif;white-space:inherit"><span style="margin:0px;padding:0px;border:0px;font-size:13px;color:rgb(0,0,0)">processFiles </span><span style="margin:0px;padding:0px;border:0px;font-size:13px;color:rgb(0,0,0)">::</span><span style="margin:0px;padding:0px;border:0px;font-size:13px;color:rgb(0,0,0)"> </span><span style="margin:0px;padding:0px;border:0px;font-size:13px;color:rgb(0,0,0)">[</span><span style="margin:0px;padding:0px;border:0px;font-size:13px;color:rgb(0,0,0)">FilePath</span><span style="margin:0px;padding:0px;border:0px;font-size:13px;color:rgb(0,0,0)">]</span><span style="margin:0px;padding:0px;border:0px;font-size:13px;color:rgb(0,0,0)"> </span><span style="margin:0px;padding:0px;border:0px;font-size:13px;color:rgb(0,0,0)">-></span><span style="margin:0px;padding:0px;border:0px;font-size:13px;color:rgb(0,0,0)"> </span><span style="margin:0px;padding:0px;border:0px;font-size:13px;color:rgb(0,0,0)">(</span><span style="margin:0px;padding:0px;border:0px;font-size:13px;color:rgb(0,0,0)">FilePath </span><span style="margin:0px;padding:0px;border:0px;font-size:13px;color:rgb(0,0,0)">-></span><span style="margin:0px;padding:0px;border:0px;font-size:13px;color:rgb(0,0,0)"> IO </span><span style="margin:0px;padding:0px;border:0px;font-size:13px;color:rgb(0,0,0)">())</span><span style="margin:0px;padding:0px;border:0px;font-size:13px;color:rgb(0,0,0)"> </span><span style="margin:0px;padding:0px;border:0px;font-size:13px;color:rgb(0,0,0)">-></span><span style="margin:0px;padding:0px;border:0px;font-size:13px;color:rgb(0,0,0)"> IO </span><span style="margin:0px;padding:0px;border:0px;font-size:13px;color:rgb(0,0,0)">()</span></code></pre><p style="margin:0px 0px 1em;padding:0px;border:0px;font-size:15px;clear:both;font-family:'Helvetica Neue',Helvetica,Arial,sans-serif;line-height:19.5px">This function spawns an async process that execute an IO action. This IO action must be submitted to a cluster through a job scheduling system (e.g Slurm).</p><p style="margin:0px 0px 1em;padding:0px;border:0px;font-size:15px;clear:both;font-family:'Helvetica Neue',Helvetica,Arial,sans-serif;line-height:19.5px">Because I must use the job scheduling system, it's not possible to use cloudHaskell to distribute the closure. Instead the program writes a new <em style="margin:0px;padding:0px;border:0px;font-size:15px">Main.hs</em> containing the desired computations, that is copy to the cluster node together with all the modules that main depends on and then it is executed remotely with "runhaskell Main.hs [opts]". Then the async process should ask periodically to the job scheduling system (using <em style="margin:0px;padding:0px;border:0px;font-size:15px">threadDelay</em>) if the job is done.</p><p style="margin:0px 0px 1em;padding:0px;border:0px;font-size:15px;clear:both;font-family:'Helvetica Neue',Helvetica,Arial,sans-serif;line-height:19.5px">Is there a way to avoid creating a new Main? Can I serialize the IO action and execute it somehow in the node?</p><p style="margin:0px 0px 1em;padding:0px;border:0px;font-size:15px;clear:both;font-family:'Helvetica Neue',Helvetica,Arial,sans-serif;line-height:19.5px">Best,</p><p style="margin:0px 0px 1em;padding:0px;border:0px;font-size:15px;clear:both;font-family:'Helvetica Neue',Helvetica,Arial,sans-serif;line-height:19.5px">Felipe</p></div></div>
______________________________<u></u>_________________<br>
Haskell-Cafe mailing list<br>
<a href="mailto:Haskell-Cafe@haskell.org" target="_blank">Haskell-Cafe@haskell.org</a><br>
<a href="http://mail.haskell.org/cgi-bin/mailman/listinfo/haskell-cafe" target="_blank">http://mail.haskell.org/cgi-<u></u>bin/mailman/listinfo/haskell-<u></u>cafe</a><br>
</blockquote></div>
</div></blockquote><blockquote type="cite"><div><span>_______________________________________________</span><br><span>Haskell-Cafe mailing list</span><br><span><a href="mailto:Haskell-Cafe@haskell.org">Haskell-Cafe@haskell.org</a></span><br><span><a href="http://mail.haskell.org/cgi-bin/mailman/listinfo/haskell-cafe">http://mail.haskell.org/cgi-bin/mailman/listinfo/haskell-cafe</a></span><br></div></blockquote></div></body></html>