<html>
<head>
<meta http-equiv="Content-Type" content="text/html;
charset=windows-1252">
</head>
<body>
<p>If you want performance, you should use custom solutions...</p>
<p><a class="moz-txt-link-freetext" href="https://benchmarksgame-team.pages.debian.net/benchmarksgame/program/knucleotide-ghc-1.html">https://benchmarksgame-team.pages.debian.net/benchmarksgame/program/knucleotide-ghc-1.html</a></p>
<p>This one was taken because of library hashmap and my custom
solution hashtable was much faster.<br>
</p>
<p>Greets.<br>
</p>
<div class="moz-cite-prefix">On 7/30/20 3:28 PM, YueCompl via
Haskell-Cafe wrote:<br>
</div>
<blockquote type="cite"
cite="mid:8D814D14-3F54-4D2E-A3F1-EC8A2DCB1E31@icloud.com">
<meta http-equiv="Content-Type" content="text/html;
charset=windows-1252">
For the record, overhead of STM over IO (or other means where
manual composition of transactions needed) based concurrency
control, is a price I'm willing to pay in my use case, as it's not
machine-performance critical in distributing input data +
parameters to a cluster of worker nodes, and collecting their
results into permanent storage or a data pipeline. But to keep
professionally crafting well synced, race-free scheduling code is
barely affordable by my org, as shape of datasets, relationship
between them and algorithms processing them are varying at fast
paces, we have difficulty, or lack the willingness, to hire some
workforce specifically to keep each new data pipeline race free,
it has to be, but better at cost of machine-hours instead of human
head counts.
<div class=""><br class="">
</div>
<div class="">While easily compositing stm code, wrapped in
scriptable procedures, will enable our analysts to author the
scheduling scripts without too much concerns. Then our
programmers can focus on performance critical parts of the data
processing code, like optimization of tight-loops.
<div class=""><br class="">
</div>
<div class="">Only if not in the tight loops, I think it's
acceptable by us, that up to 2~3 order of <span
style="caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0);"
class="">magnitude</span> slower for an stm solution
compared to its best rivals, as long as it's scalable. For a
(maybe cheating) example, if fully optimized code can return
result in 10 ms after an analyst clicked a button, we don't
bother if unoptimized stm script needs 10 second, so long as
the result is correct.</div>
<div class=""><br class="">
</div>
<div class="">In a philosophic thinking, I heard that AT&T
had UNIX specifically designed for their Control panel, while
their Data panel runs separate software (and on separate
hardware obviously), while modern systems have powerful CPUs
tempting us to squeeze more performance out of it, and SIMD
instructions make it even more tempting, I think we'd better
resist it when programming something belong to the Control
panel per se, but do it in programming something belong to the
Data panel. And appears Data panel programs are being shifted
to GPUs nowadays, which feels right.</div>
<div class=""><br class="">
</div>
<div class="">Regards,</div>
<div class="">Compl</div>
<div class=""><br class="">
<div><br class="">
<blockquote type="cite" class="">
<div class="">On 2020-07-30, at 20:10, YueCompl via
Haskell-Cafe <<a
href="mailto:haskell-cafe@haskell.org" class=""
moz-do-not-send="true">haskell-cafe@haskell.org</a>>
wrote:</div>
<br class="Apple-interchange-newline">
<div class="">
<meta http-equiv="Content-Type" content="text/html;
charset=windows-1252" class="">
<div style="word-wrap: break-word; -webkit-nbsp-mode:
space; line-break: after-white-space;" class="">Hi
Peter,
<div class=""><br class="">
</div>
<div class="">Great to hear from you! </div>
<div class=""><br class="">
</div>
<div class="">For the record tskiplist (and
stm-containers together) did improve my situation a
great lot with respect to scalability at
concurrency/parallelism!</div>
<div class=""><br class="">
</div>
<div class="">I'm still far from the stage to squeeze
last drops of performance, currently I'm just making
sure performance wise concerns be reasonable during
my PoC in correctness and ergonomics of my HPC
architecture (an in-memory graph + out-of-core
(mmap) array DBMS powered computation cluster, with
shared storage), and after parallelism appears
acceptable, I seemingly suffer from serious GC issue
at up scaling on process working memory size. I'm
suspecting it's because of the added more TVars
and/or aggressive circular structures of them in my
case, and can not find a way to overcome it by far.</div>
<div class=""><br class="">
</div>
<div class="">Thanks for your detailed information!</div>
<div class=""><br class="">
</div>
<div class="">Best regards,</div>
<div class="">Compl</div>
<div class=""><br class="">
<div class=""><br class="">
<blockquote type="cite" class="">
<div class="">On 2020-07-30, at 19:19, Peter
Robinson <<a
href="mailto:pwr@lowerbound.io" class=""
moz-do-not-send="true">pwr@lowerbound.io</a>>
wrote:</div>
<br class="Apple-interchange-newline">
<div class="">
<div dir="ltr" class="">
<div class="gmail_quote">
<div class="">Hi Compl,</div>
<blockquote class="gmail_quote"
style="margin:0px 0px 0px
0.8ex;border-left:1px solid
rgb(204,204,204);padding-left:1ex">
<div dir="ltr" class="">
<div class="gmail_quote">
<blockquote class="gmail_quote"
style="margin:0px 0px 0px
0.8ex;border-left:1px solid
rgb(204,204,204);padding-left:1ex">
<div class="">
<p class="">>+ <span
style="color:rgb(200,195,188);font-family:"PT
Sans",-apple-system,BlinkMacSystemFont,"Segoe
UI",Roboto,Oxygen-Sans,Cantarell,"Helvetica
Neue",sans-serif;font-size:17px;font-style:normal;font-variant-ligatures:normal;font-variant-caps:normal;font-weight:400;letter-spacing:0.024px;text-align:left;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;background-color:rgb(25,27,28);text-decoration-style:initial;text-decoration-color:initial;float:none;display:inline"
class="">This package
provides a proof-of-concept
implementation of a skip
list in STM</span></p>
<p class="">This has to mean
something but I can't figure
out yet.<br class="">
</p>
<p class="">Dear Peter Robinson,
I hope you can see this
message and get in the loop of
discussion. </p>
</div>
</blockquote>
</div>
</div>
</blockquote>
<div class=""><br class="">
</div>
<div class=""> The reason for adding this
sentence was that tskiplist hasn't been
optimized for production use. Later on,
I wrote an implementation of a
concurrent skip list with atomic
operations that performs significantly
better, but it's operations work in the
IO monad. </div>
<br class="">
I'm surprised to hear that you're getting
poor performance even when using the
stm-container package, which I believe was
meant to be used in production. A while
ago, I ran some benchmarks comparing
concurrent dictionary data structures
(such as stm-container) under various
workloads. While STMContainers.Map wasn't
as fast as the concurrent-hashtable
package, the results indicate that the
performance doesn't degrade too much under
larger workloads.<br class="">
<br class="">
You can find these benchmark results here
(10^6 randomly generated
insertion/deletion/lookup requests
distributed among 32 threads): <br
class="">
<a
href="https://lowerbound.io/blog/bench2-32.html"
class="" moz-do-not-send="true">https://lowerbound.io/blog/bench2-32.html</a>
<br class="">
And some explanations about the benchmarks
are here: <br class="">
<a
href="https://lowerbound.io/blog/2019-10-24_concurrent_hash_table_performance.html"
class="" moz-do-not-send="true">https://lowerbound.io/blog/2019-10-24_concurrent_hash_table_performance.html</a><br
class="">
<br class="">
One issue that I came across when
implementing the tskiplist package was
this: If a thread wants to insert some
item into the skip list, it needs to
search for the entry point by performing
readTVar operations starting at the list
head. So, on average, a thread will read
O(log n) TVars (assuming a skip list of n
items) and, if any of these O(log n) TVars
are modified by a simultaneously running
thread, the STM runtime will observe a
(false) conflict and rerun the
transaction. It's not clear to me how to
resolve this issue without access to
something like unreadTVar (see [1]). <br
class="">
<br class="">
Best,<br class="">
Peter<br class="">
<br class="">
[1] UnreadTVar: Extending Haskell Software
Transactional Memory for Performance
(2007) by Nehir Sonmez , Cristian Perfumo
, Srdjan Stipic , Adrian Cristal , Osman
S. Unsal , Mateo Valero.<br class="">
</div>
<div class="gmail_quote"><br class="">
</div>
</div>
_______________________________________________<br class="">
Haskell-Cafe mailing list<br class="">
To (un)subscribe, modify options or view
archives go to:<br class="">
<a
href="http://mail.haskell.org/cgi-bin/mailman/listinfo/haskell-cafe"
class="" moz-do-not-send="true">http://mail.haskell.org/cgi-bin/mailman/listinfo/haskell-cafe</a><br
class="">
Only members subscribed via the mailman list
are allowed to post.</div>
</blockquote>
</div>
<br class="">
</div>
</div>
_______________________________________________<br
class="">
Haskell-Cafe mailing list<br class="">
To (un)subscribe, modify options or view archives go to:<br
class="">
<a
href="http://mail.haskell.org/cgi-bin/mailman/listinfo/haskell-cafe"
class="" moz-do-not-send="true">http://mail.haskell.org/cgi-bin/mailman/listinfo/haskell-cafe</a><br
class="">
Only members subscribed via the mailman list are allowed
to post.</div>
</blockquote>
</div>
<br class="">
</div>
</div>
<br>
<fieldset class="mimeAttachmentHeader"></fieldset>
<pre class="moz-quote-pre" wrap="">_______________________________________________
Haskell-Cafe mailing list
To (un)subscribe, modify options or view archives go to:
<a class="moz-txt-link-freetext" href="http://mail.haskell.org/cgi-bin/mailman/listinfo/haskell-cafe">http://mail.haskell.org/cgi-bin/mailman/listinfo/haskell-cafe</a>
Only members subscribed via the mailman list are allowed to post.</pre>
</blockquote>
</body>
</html>