[Haskell-cafe] Combining ST with STM

Tue Feb 9 08:36:26 UTC 2016

On 02/09/2016 07:20 AM, Thomas Koster wrote:
> On 9 February 2016 at 14:43, Thomas Koster <tkoster at gmail.com> wrote:
>> I have an STM transaction that needs some private, temporary state.
>> The most obvious way is to simply pass pure state as arguments, but
>> for efficiency, I would like this state to be some kind of mutable
>> array, like STArray.

This sounds like optimizing before you know what is really slow,
complicating your code for no good reason...

>>
>> I know, STM has TVars and TArray, but since this state is private to
>> the transaction, I am wondering if using TVars/TArrays for private
>> state might be overkill that will unnecessarily slow down the STM
>> commit process. The private state is, by definition, not shared, so
>> including it in the STM log and commit process is, as far as I can
>> tell, pointless.

The STM log allows you to revert a failed transaction. If you do not
record your writes to an array, you can not revert them and they can
leak outside an aborted transaction.

>>
>> ST and STArray still appear to be the most appropriate tools for the
>> private state, because STRefs and STArrays really, really are private.
>>
>> So this basically means I want to interleave ST and STM in a "safe"
>> way. That is, if the STM transaction retries, I want the ST state to
>> be vaporised as well.

So how should this work? Some log recording what you did would be good,
so the runtime knows which changes you did to the array... If you
however create the array in the same transaction, this would work by
just throwing away the whole array.

>>
>> Ideally, I would love to be able to say something like this:
>>
>> -- | Copy the value from the shared TVar into the private STRef.
>> load :: TVar a -> STRef a -> STSTM s ()
>> load shared private = do
>>   value <- liftSTM (readTVar shared)
>>   liftST (writeSTRef private value)
>>
>> Naturally, that STRef must originate from a call to newSTRef earlier
>> in the same transaction and is private to it, just like the real ST
>> monad. As far as I can tell, I am not trying to weaken either ST or
>> STM in any way here.
>>
>> I found the STMonadTrans package on Hackage [1] that claims to
>> implement ST as a monad transformer, STT, which sounds close to what I
>> want. While its documentation does not mention STM, it does say that
>> some monads are unsafe to use as a base monad for STT.
>>
>> Is STMonadTrans safe to use with STM?

It is not even safe to use with Maybe (for now), as it can share
different STRefs and STArrays. I filed a bug report. After the bug is
fixed, I see no reason, why it should not work with STM, as the complete
ST action should be repeated if the STM transaction aborts.

>>
>> [1] https://hackage.haskell.org/package/STMonadTrans
> 
> On 9 February 2016 at 15:16, Ryan Yates <fryguybob at gmail.com> wrote:
>> I also have found need for what I think you are describing but only in the
>> context of transactional arrays where there are multiple fields to
>> initialize while I know that the array is private to the creating thread.
>> For now I'm adding the primitives I need as I go, but I would like to have
>> better safer story.  You might be interested in how the stm-containers
>> package uses ST to build internal nodes in transactions [1], [2].
>>
>> [1]:
>> https://github.com/nikita-volkov/stm-containers/blob/master/library/STMContainers/HAMT/Nodes.hs#L36
>> [2]:
>> https://github.com/nikita-volkov/stm-containers/blob/master/library/STMContainers/WordArray.hs#L118
> 
> Thank you Ryan.
> 
> Indeed, it is by experimenting with stm-containers that the need for
> mixing ST and STM arose. Where the previous iteration of my program
> used plain ST transactions serialized with an MVar, I am experimenting
> with stm-containers with the hope that I will see improved throughput
> for transactions that do not overlap, which, I believe, could complete
> in parallel, at least some of the time.
> 
> It seems stm-containers itself uses unsafeFreezeArray from the
> "primitive" package. One difference though is that while my private
> array would be thawed, modified and refrozen regularly, the
> stm-containers WordArray stays immutable (not thawed) once frozen, as
> far as I can tell.

So what happens if you thaw an array, write to it and then abort the
transaction? You have to revert the writes because they could be visible
to the transaction you just aborted. When this transaction restarts, the
array will still contain the values written prior to it. Even if nothing
else contains a reference to it, your array is garbage after you aborted
a transaction only once.

> 
> Since I am using only a single array for the entire private state,
> sprinkling some runST calls with unsafeThawArray/unsafeFreezeArray in
> my STM transaction may be enough for my needs, as long as I am
> exceptionally careful not to leak one of these arrays into or out of
> any STM transaction demarcated by the "atomically" block. If anybody
> knows of any reason why I should abort this idea, please speak up.

Keep in mind that ST is only "safe IO" in a sense such that no side
effects are visible to the outside. You lose this if you start to modify
anything which you did not create yourself. I think this is not that
different from using
http://hackage.haskell.org/package/base-4.8.2.0/docs/GHC-Conc-Sync.html#v:unsafeIOToSTM.
To be safe, you should at least copy the arrays instead of unsafely
thawing them... but then it could be faster just to use TArrays from the
start.

> 
> I noticed also that Data.Array.Unsafe in base also has unsafe freezing
> and thawing. Is there a reason to use one over the other?
> 
> --
> Thomas Koster
> _______________________________________________
> Haskell-Cafe mailing list
> Haskell-Cafe at haskell.org
> http://mail.haskell.org/cgi-bin/mailman/listinfo/haskell-cafe
> 

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 473 bytes
Desc: OpenPGP digital signature
URL: <http://mail.haskell.org/pipermail/haskell-cafe/attachments/20160209/052cdf1d/attachment-0001.sig>