<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
</head>
<body>
<p>I have a feel that given the result data is dense and already in
RAM, your approach should already be the most safe & efficient
one, though benchmarks may favor other slight variants with
different sized arrays, the speed diffs should be neglectable.</p>
<p>But from overall architecture, I suggest it can be even more
proficient to mmap the data file into foreign ptr in order to back
the array to receive computation result, with virtual memory, then
after the computation, do `msync` to guarantee the data is written
to non-volatile storage. This puts no burden at GC in the first
place, and of course demands no further memory pinning etc. at
all, by just leveraging the os' virtual memory system (and modern
file systems that tightly coupled with it) for its designed
purpose. <br>
</p>
I'm myself doing a PoC of an array database thing, I'm currently
using the vector-mmap package's routine to finish the PoC. But the
depended mmap package lacks `msync`, and a test case suggests
resource leakage with GHC 8.6.5, so a stable solution is yet to be
worked out ahead the way.
<p>Btw, when you have more then 10k such array files to mmap, you'll
hit another limit - nofile for Linux, I used to implement a FUSE
filesystem providing virtual large data files viewing many small
files on the remote storage server, but written in Go, and am
porting that to GHC in the near future.<br>
</p>
<p><br>
</p>
<div class="moz-cite-prefix">On 2020/4/3 上午3:25, cyberfined via
Haskell-Cafe wrote:<br>
</div>
<blockquote type="cite"
cite="mid:BU6nBD07EP-WTdwM3YvhBuUkxQ2JKLZ5lsFcHqbWnlqyXtK0AFR8hlJi0xO9f6mXrNEa0lObMHmeVgHi55ucVYgf2s_asu41RR8VMJFZGbU=@protonmail.com">
<meta http-equiv="content-type" content="text/html; charset=UTF-8">
<div>Hello, all. I decide to write parallel ray tracer on haskell
with repa. Now, to save repa array to file I use dirty trick
casting repa array ptr to bytestring with fromForeignPtr and
then writing it to file with hPut. It looks something like that:<br>
</div>
<div><br>
</div>
<div><span style="font-family: menlo, consolas, courier new,
monospace, sans-serif;">import qualified Data.Array.Repa as R</span><span
style="font-family: menlo, consolas, courier new, monospace,
sans-serif;"><br>
</span></div>
<div><span style="font-family: menlo, consolas, courier new,
monospace, sans-serif;">import qualified
Data.Array.Repa.Repr.ForeignPtr as RF</span><br>
</div>
<div><br>
</div>
<div><span style="font-family: menlo, consolas, courier new,
monospace, sans-serif;">import qualified Data.ByteString as B<br>
</span></div>
<div><span style="font-family: menlo, consolas, courier new,
monospace, sans-serif;">import qualified Data.ByteString.Char8
as BC<br>
</span></div>
<div><span style="font-family: menlo, consolas, courier new,
monospace, sans-serif;">import qualified
Data.ByteString.Internal as BI</span><br>
</div>
<div><br>
</div>
<div><span style="font-family: menlo, consolas, courier new,
monospace, sans-serif;">type Image = Array F DIM2 Pixel<br>
</span></div>
<div><span style="font-family: menlo, consolas, courier new,
monospace, sans-serif;"><br>
</span></div>
<div><span style="font-family: menlo, consolas, courier new,
monospace, sans-serif;">writeImage :: FilePath -> Image
-> IO ()<br>
</span></div>
<div><span style="font-family: menlo, consolas, courier new,
monospace, sans-serif;">writeImage path img = bracket
(openFile path WriteMode) (hClose) $ \hdl -> B.hPut hdl
header >> B.hPut hdl body<br>
</span></div>
<div><span style="font-family: menlo, consolas, courier new,
monospace, sans-serif;"> where Z :. h :. w = R.extent img<br>
</span></div>
<div><span style="font-family: menlo, consolas, courier new,
monospace, sans-serif;"> header = BC.pack $ "P6\n" ++
show w ++ ' ':show h ++ "\n255\n"<br>
</span></div>
<div><span style="font-family: menlo, consolas, courier new,
monospace, sans-serif;"> body =
BI.fromForeignPtr(castForeignPtr $ RF.toForeignPtr img) 0
(w*h*3)</span><br>
</div>
<div><br>
</div>
<div>My question is: how to write repa array to file safely and
fast?<br>
</div>
<br>
<fieldset class="mimeAttachmentHeader"></fieldset>
<pre class="moz-quote-pre" wrap="">_______________________________________________
Haskell-Cafe mailing list
To (un)subscribe, modify options or view archives go to:
<a class="moz-txt-link-freetext" href="http://mail.haskell.org/cgi-bin/mailman/listinfo/haskell-cafe">http://mail.haskell.org/cgi-bin/mailman/listinfo/haskell-cafe</a>
Only members subscribed via the mailman list are allowed to post.</pre>
</blockquote>
</body>
</html>