From david.feuer at gmail.com Fri Oct 2 20:42:39 2020 From: david.feuer at gmail.com (David Feuer) Date: Fri, 2 Oct 2020 16:42:39 -0400 Subject: Help making exception-related primops open-kinded Message-ID: I'm pretty sure raiseIO#, catch#, maskAsyncExceptions#, and so on can have open-kinded results. For example, we currently have catch# :: (State# RealWorld -> (# State# RealWorld, a #) ) -> (b -> State# RealWorld -> (# State# RealWorld, a #) ) -> State# RealWorld -> (# State# RealWorld, a #) but I think we *should* be able to have catch# :: (State# RealWorld -> (# State# RealWorld, o #) ) -> (b -> State# RealWorld -> (# State# RealWorld, o #) ) -> State# RealWorld -> (# State# RealWorld, o #) This should allow us to avoid boxing things just to be able to get them out of a `catch#` or `maskAsyncExceptions#` block. But when I tried just changing the types in primops.txt.pp, I got a type mismatch error that seems to suggest some sort of GHC build staging issue. How can I work around this? See https://gitlab.haskell.org/ghc/ghc/-/merge_requests/4192/diffs for my attempt. From ryan.gl.scott at gmail.com Mon Oct 12 09:51:19 2020 From: ryan.gl.scott at gmail.com (Ryan Scott) Date: Mon, 12 Oct 2020 05:51:19 -0400 Subject: Seeking performance-minded reviewers on !2965 Message-ID: !2965 [1] seeks to improve the performance of derived Generic(1) instances by heuristically annotating implementations of the from(1) and to(1) class methods with INLINE[1] pragmas. For the most part, this results in performance wins, as the generic intermediate representations of data types can be optimized away in more situations. However, I'm unclear on one aspect of the MR: it actually leads to a compile-time performance increase (specifically, in bytes allocated) in one test case: T12227: Test Metric value New value Change T12227(normal) ghc/alloc 518417756.0 526479616.0 +1.6% BAD The MR author provides an explanation for why this happens here [2]. However, as I am not really a GHC performance guru, it's difficult for me to accurately judge whether this difference is within acceptable limits or not. For those of you who care about compile-time performance, would you be willing to give !2965 a look over and judge whether the difference is acceptable? I think this is the last remaining hurdle to clear before we can land !2965, so your input would be greatly appreciated. Thanks! Ryan S. ----- [1] https://gitlab.haskell.org/ghc/ghc/-/merge_requests/2965 [2] https://gitlab.haskell.org/ghc/ghc/-/merge_requests/2965#note_304501 -------------- next part -------------- An HTML attachment was scrubbed... URL: From david.feuer at gmail.com Tue Oct 13 21:46:49 2020 From: david.feuer at gmail.com (David Feuer) Date: Tue, 13 Oct 2020 17:46:49 -0400 Subject: Restricted sums in BoxedRep Message-ID: Null pointers are widely known to be a lousy language feature in general, but there are certain situations where they're *really* useful for compact representation. For example, we define newtype TMVar a = TMVar (TVar (Maybe a)) We don't, however, actually use the fact that (Maybe a) is lifted. So we could represent this much more efficiently using something like newtype TMVar a = TMVar (TVar a) where Nothing is represented by a distinguished "null" pointer. While it's possible to implement this sort of thing in user code (with lots of fuss and care), it's not very nice at all. What I'd really like to be able to do is represent certain kinds of sums like this natively. Now that we're getting BoxedRep, I think we can probably make it happen. The trick is to add a special Levity constructor representing sums of particular shapes. Specifically, we can represent a type like this if it is a possibly-nested sum which, when flattened into a single sum, consists of some number of nullary tuples and at most one Lifted or Unlifted type. Then we can have (inline) primops to convert between the BoxedRep and the sum-of-sums representations. Anyone have thoughts on details for what the Levity constructor arguments might look like? -------------- next part -------------- An HTML attachment was scrubbed... URL: From arnaud.spiwack at tweag.io Wed Oct 14 11:45:20 2020 From: arnaud.spiwack at tweag.io (Spiwack, Arnaud) Date: Wed, 14 Oct 2020 13:45:20 +0200 Subject: Restricted sums in BoxedRep In-Reply-To: References: Message-ID: I may have misunderstood, but my understanding is the following: - Since a is a boxed type, it can never be the null pointer - So I can use a null pointer unambiguously Let's call this null-pointer expanded type, `Nullable# a`, it is now a different sort than `a`, since it can have the null pointer in it. Is that still what you wish for? The risk being that combining such a `Nullable# a` with another data structure may very well require an additional boxing, which is what, I believe, you were trying to avoid. /Arnaud On Tue, Oct 13, 2020 at 11:47 PM David Feuer wrote: > Null pointers are widely known to be a lousy language feature in general, > but there are certain situations where they're *really* useful for compact > representation. For example, we define > > newtype TMVar a = TMVar (TVar (Maybe a)) > > We don't, however, actually use the fact that (Maybe a) is lifted. So we > could represent this much more efficiently using something like > > newtype TMVar a = TMVar (TVar a) > > where Nothing is represented by a distinguished "null" pointer. > > While it's possible to implement this sort of thing in user code (with > lots of fuss and care), it's not very nice at all. What I'd really like to > be able to do is represent certain kinds of sums like this natively. > > Now that we're getting BoxedRep, I think we can probably make it happen. > The trick is to add a special Levity constructor representing sums of > particular shapes. Specifically, we can represent a type like this if it is > a possibly-nested sum which, when flattened into a single sum, consists of > some number of nullary tuples and at most one Lifted or Unlifted type. > Then we can have (inline) primops to convert between the BoxedRep and the > sum-of-sums representations. > > Anyone have thoughts on details for what the Levity constructor arguments > might look like? > _______________________________________________ > ghc-devs mailing list > ghc-devs at haskell.org > http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs > -------------- next part -------------- An HTML attachment was scrubbed... URL: From sgraf1337 at gmail.com Wed Oct 14 11:54:16 2020 From: sgraf1337 at gmail.com (Sebastian Graf) Date: Wed, 14 Oct 2020 13:54:16 +0200 Subject: Restricted sums in BoxedRep In-Reply-To: References: Message-ID: I believe Simon told me once that NULL pointers in places we assume BoxedRep things are not an option, because the GC assumes it is free to follow that pointer. It won't check if it's NULL or not. That's also the reason why we lower `LitRubbish` (which we use for absent BoxedRep literals) as `()` when going to STG -- We can't use NULL, so supply an arbitrary (untyped!) closure that we know can be followed by the GC. Am Mi., 14. Okt. 2020 um 13:46 Uhr schrieb Spiwack, Arnaud < arnaud.spiwack at tweag.io>: > I may have misunderstood, but my understanding is the following: > > - Since a is a boxed type, it can never be the null pointer > - So I can use a null pointer unambiguously > > Let's call this null-pointer expanded type, `Nullable# a`, it is now a > different sort than `a`, since it can have the null pointer in it. Is that > still what you wish for? The risk being that combining such a `Nullable# a` > with another data structure may very well require an additional boxing, > which is what, I believe, you were trying to avoid. > > /Arnaud > > On Tue, Oct 13, 2020 at 11:47 PM David Feuer > wrote: > >> Null pointers are widely known to be a lousy language feature in general, >> but there are certain situations where they're *really* useful for compact >> representation. For example, we define >> >> newtype TMVar a = TMVar (TVar (Maybe a)) >> >> We don't, however, actually use the fact that (Maybe a) is lifted. So we >> could represent this much more efficiently using something like >> >> newtype TMVar a = TMVar (TVar a) >> >> where Nothing is represented by a distinguished "null" pointer. >> >> While it's possible to implement this sort of thing in user code (with >> lots of fuss and care), it's not very nice at all. What I'd really like to >> be able to do is represent certain kinds of sums like this natively. >> >> Now that we're getting BoxedRep, I think we can probably make it happen. >> The trick is to add a special Levity constructor representing sums of >> particular shapes. Specifically, we can represent a type like this if it is >> a possibly-nested sum which, when flattened into a single sum, consists of >> some number of nullary tuples and at most one Lifted or Unlifted type. >> Then we can have (inline) primops to convert between the BoxedRep and the >> sum-of-sums representations. >> >> Anyone have thoughts on details for what the Levity constructor arguments >> might look like? >> _______________________________________________ >> ghc-devs mailing list >> ghc-devs at haskell.org >> http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs >> > _______________________________________________ > ghc-devs mailing list > ghc-devs at haskell.org > http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs > -------------- next part -------------- An HTML attachment was scrubbed... URL: From arnaud.spiwack at tweag.io Wed Oct 14 11:58:57 2020 From: arnaud.spiwack at tweag.io (Spiwack, Arnaud) Date: Wed, 14 Oct 2020 13:58:57 +0200 Subject: Restricted sums in BoxedRep In-Reply-To: References: Message-ID: I don't imagine it would be hard for the GC to check that a pointer is null before following it. The argument may be, though, that it's too high a price to pay for the occasional use of a null pointer. (I must add that I have no opinion, or idea really, on this) On Wed, Oct 14, 2020 at 1:54 PM Sebastian Graf wrote: > I believe Simon told me once that NULL pointers in places we assume > BoxedRep things are not an option, because the GC assumes it is free to > follow that pointer. It won't check if it's NULL or not. > That's also the reason why we lower `LitRubbish` (which we use for absent > BoxedRep literals) as `()` when going to STG > > -- We can't use NULL, so supply an arbitrary (untyped!) closure that we > know can be followed by the GC. > > Am Mi., 14. Okt. 2020 um 13:46 Uhr schrieb Spiwack, Arnaud < > arnaud.spiwack at tweag.io>: > >> I may have misunderstood, but my understanding is the following: >> >> - Since a is a boxed type, it can never be the null pointer >> - So I can use a null pointer unambiguously >> >> Let's call this null-pointer expanded type, `Nullable# a`, it is now a >> different sort than `a`, since it can have the null pointer in it. Is that >> still what you wish for? The risk being that combining such a `Nullable# a` >> with another data structure may very well require an additional boxing, >> which is what, I believe, you were trying to avoid. >> >> /Arnaud >> >> On Tue, Oct 13, 2020 at 11:47 PM David Feuer >> wrote: >> >>> Null pointers are widely known to be a lousy language feature in >>> general, but there are certain situations where they're *really* useful for >>> compact representation. For example, we define >>> >>> newtype TMVar a = TMVar (TVar (Maybe a)) >>> >>> We don't, however, actually use the fact that (Maybe a) is lifted. So we >>> could represent this much more efficiently using something like >>> >>> newtype TMVar a = TMVar (TVar a) >>> >>> where Nothing is represented by a distinguished "null" pointer. >>> >>> While it's possible to implement this sort of thing in user code (with >>> lots of fuss and care), it's not very nice at all. What I'd really like to >>> be able to do is represent certain kinds of sums like this natively. >>> >>> Now that we're getting BoxedRep, I think we can probably make it happen. >>> The trick is to add a special Levity constructor representing sums of >>> particular shapes. Specifically, we can represent a type like this if it is >>> a possibly-nested sum which, when flattened into a single sum, consists of >>> some number of nullary tuples and at most one Lifted or Unlifted type. >>> Then we can have (inline) primops to convert between the BoxedRep and the >>> sum-of-sums representations. >>> >>> Anyone have thoughts on details for what the Levity constructor >>> arguments might look like? >>> _______________________________________________ >>> ghc-devs mailing list >>> ghc-devs at haskell.org >>> http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs >>> >> _______________________________________________ >> ghc-devs mailing list >> ghc-devs at haskell.org >> http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From david.feuer at gmail.com Wed Oct 14 13:57:02 2020 From: david.feuer at gmail.com (David Feuer) Date: Wed, 14 Oct 2020 09:57:02 -0400 Subject: Restricted sums in BoxedRep In-Reply-To: References: Message-ID: I don't know your terminology, sorry. By "null" I'm referring to something distinguished, of which there can be more than one. These can be pointers to statically allocated objects if necessary, though pointers in an unused address range would probably be nicer. My goal is to be able to shove a compact representation of something like (# (##) | (##) | (##) | a #) into an array/MutVar/etc., rather than needing to box it to do so. On Wed, Oct 14, 2020, 7:45 AM Spiwack, Arnaud wrote: > I may have misunderstood, but my understanding is the following: > > - Since a is a boxed type, it can never be the null pointer > - So I can use a null pointer unambiguously > > Let's call this null-pointer expanded type, `Nullable# a`, it is now a > different sort than `a`, since it can have the null pointer in it. Is that > still what you wish for? The risk being that combining such a `Nullable# a` > with another data structure may very well require an additional boxing, > which is what, I believe, you were trying to avoid. > > /Arnaud > > On Tue, Oct 13, 2020 at 11:47 PM David Feuer > wrote: > >> Null pointers are widely known to be a lousy language feature in general, >> but there are certain situations where they're *really* useful for compact >> representation. For example, we define >> >> newtype TMVar a = TMVar (TVar (Maybe a)) >> >> We don't, however, actually use the fact that (Maybe a) is lifted. So we >> could represent this much more efficiently using something like >> >> newtype TMVar a = TMVar (TVar a) >> >> where Nothing is represented by a distinguished "null" pointer. >> >> While it's possible to implement this sort of thing in user code (with >> lots of fuss and care), it's not very nice at all. What I'd really like to >> be able to do is represent certain kinds of sums like this natively. >> >> Now that we're getting BoxedRep, I think we can probably make it happen. >> The trick is to add a special Levity constructor representing sums of >> particular shapes. Specifically, we can represent a type like this if it is >> a possibly-nested sum which, when flattened into a single sum, consists of >> some number of nullary tuples and at most one Lifted or Unlifted type. >> Then we can have (inline) primops to convert between the BoxedRep and the >> sum-of-sums representations. >> >> Anyone have thoughts on details for what the Levity constructor arguments >> might look like? >> _______________________________________________ >> ghc-devs mailing list >> ghc-devs at haskell.org >> http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From arnaud.spiwack at tweag.io Wed Oct 14 14:29:22 2020 From: arnaud.spiwack at tweag.io (Spiwack, Arnaud) Date: Wed, 14 Oct 2020 16:29:22 +0200 Subject: Restricted sums in BoxedRep In-Reply-To: References: Message-ID: Ok, I believe get it now, Let's imagine (to take only the simplest case) that we have a `Nullable# a` type, such that `Nullable# a = (# (##) | a #)`. What would be the kind of `Nullable#`? I imagine that it would be something like `TYPE (BoxedRep Lifted) -> TYPE (BoxedRep Nullable)`. Then you would want to abstract the type of arrays/tvars/whatnot from `Type -> Type` to `forall r. TYPE (BoxRep r) -> Type`. Is that a correct interpretation of your suggestion? If so, my guess would be that all the above is fine, but I suspect (and I'm quite a bit out of my comfort zone here) that there can be considerable difficulties in implementing pattern-matching for such a type. -------------- next part -------------- An HTML attachment was scrubbed... URL: From david.feuer at gmail.com Wed Oct 14 14:52:24 2020 From: david.feuer at gmail.com (David Feuer) Date: Wed, 14 Oct 2020 10:52:24 -0400 Subject: Restricted sums in BoxedRep In-Reply-To: References: Message-ID: Yes, I think you're getting the gist. Pattern matching with one or two nulls is just an equality test or three. In practice, we'd want a fixed number of evenly spaced nulls, allowing jump table techniques to be used when there are more cases. We'd presumably have a hard limit for the number of nulls a type is allowed to have. On Wed, Oct 14, 2020, 10:29 AM Spiwack, Arnaud wrote: > Ok, I believe get it now, > > Let's imagine (to take only the simplest case) that we have a `Nullable# > a` type, such that `Nullable# a = (# (##) | a #)`. What would be the kind > of `Nullable#`? I imagine that it would be something like `TYPE (BoxedRep > Lifted) -> TYPE (BoxedRep Nullable)`. > > Then you would want to abstract the type of arrays/tvars/whatnot from > `Type -> Type` to `forall r. TYPE (BoxRep r) -> Type`. > > Is that a correct interpretation of your suggestion? > > If so, my guess would be that all the above is fine, but I suspect (and > I'm quite a bit out of my comfort zone here) that there can be considerable > difficulties in implementing pattern-matching for such a type. > -------------- next part -------------- An HTML attachment was scrubbed... URL: From sylvain at haskus.fr Wed Oct 14 18:36:12 2020 From: sylvain at haskus.fr (Sylvain Henry) Date: Wed, 14 Oct 2020 20:36:12 +0200 Subject: Perf test viewer Message-ID: <9dbeb80b-c118-aaa9-53ab-0ef8a0118dc8@haskus.fr> Hello everyone, Since testsuite performance results are stored into Git notes, they are more difficult to visualize. At least with values in .T files we could see the changes over time, but now increase/decrease are only indicated into commit messages (but not necessarily by how much). The only tool we have afaik is the perf_notes.py script [1] but it's not very interactive. So, long story short, I've started another one which is more interactive (in the browser). An instance is running on my server: http://hsyl20.fr:4222 It should be updated every 2 hours with newer metrics/commits from CI. Sources are here: https://github.com/hsyl20/had Any feedback/PR is welcome! Cheers, Sylvain [1] https://gitlab.haskell.org/ghc/ghc/-/wikis/building/running-tests/performance-tests#comparing-commits From david.feuer at gmail.com Wed Oct 14 20:21:21 2020 From: david.feuer at gmail.com (David Feuer) Date: Wed, 14 Oct 2020 16:21:21 -0400 Subject: Fwd: Restricted sums in BoxedRep In-Reply-To: References: Message-ID: Forwarded from Andrew Martin below. I think we want more than just Maybe (more than one null), but the nesting I described is certainly more convenience than necessity. ---------- Forwarded message --------- From: Andrew Martin Date: Wed, Oct 14, 2020, 4:14 PM Subject: Re: Restricted sums in BoxedRep To: David Feuer You'll have to forward this to the ghc-devs list to share it with others since I'm not currently subscribed to it, but I've had this same thought before. It is discussed at https://github.com/andrewthad/impure-containers/issues/12. Here's the relevant excerpt: > Relatedly, I was thinking the other day that after finishing implementing > https://github.com/ghc-proposals/ghc-proposals/blob/master/proposals/0203-pointer-rep.rst, > I should really look at seeing if it's possible to add this > maybe-of-a-lifted value trick straight to GHC. I think that with: > > data RuntimpRep > = BoxedRep Levity > | MaybeBoxedRep Levity > | IntRep > | ... > > data BuiltinMaybe :: forall (v :: Levity). TYPE v -> TYPE ('MaybeBoxedRep v) > > This doesn't have the nesting issues because the kind system prevents > nesting. But anyway, back to the original question. I would recommend not > using Maybe.Unsafe and using unpacked-maybe instead. The latter is > definitely safe, and it only costs an extra machine word of space in each > data constructor it gets used in, and it doesn't introduce more > indirections. > On Tue, Oct 13, 2020 at 5:47 PM David Feuer wrote: > Null pointers are widely known to be a lousy language feature in general, > but there are certain situations where they're *really* useful for compact > representation. For example, we define > > newtype TMVar a = TMVar (TVar (Maybe a)) > > We don't, however, actually use the fact that (Maybe a) is lifted. So we > could represent this much more efficiently using something like > > newtype TMVar a = TMVar (TVar a) > > where Nothing is represented by a distinguished "null" pointer. > > While it's possible to implement this sort of thing in user code (with > lots of fuss and care), it's not very nice at all. What I'd really like to > be able to do is represent certain kinds of sums like this natively. > > Now that we're getting BoxedRep, I think we can probably make it happen. > The trick is to add a special Levity constructor representing sums of > particular shapes. Specifically, we can represent a type like this if it is > a possibly-nested sum which, when flattened into a single sum, consists of > some number of nullary tuples and at most one Lifted or Unlifted type. > Then we can have (inline) primops to convert between the BoxedRep and the > sum-of-sums representations. > > Anyone have thoughts on details for what the Levity constructor arguments > might look like? > -- -Andrew Thaddeus Martin -------------- next part -------------- An HTML attachment was scrubbed... URL: From ben at smart-cactus.org Thu Oct 15 05:04:11 2020 From: ben at smart-cactus.org (Ben Gamari) Date: Thu, 15 Oct 2020 01:04:11 -0400 Subject: Perf test viewer In-Reply-To: <9dbeb80b-c118-aaa9-53ab-0ef8a0118dc8@haskus.fr> References: <9dbeb80b-c118-aaa9-53ab-0ef8a0118dc8@haskus.fr> Message-ID: <87a6wo3ws7.fsf@smart-cactus.org> Sylvain Henry writes: > Hello everyone, > > Since testsuite performance results are stored into Git notes, they are > more difficult to visualize. At least with values in .T files we could > see the changes over time, but now increase/decrease are only indicated > into commit messages (but not necessarily by how much). The only tool we > have afaik is the perf_notes.py script [1] but it's not very interactive. > > So, long story short, I've started another one which is more interactive > (in the browser). An instance is running on my server: http://hsyl20.fr:4222 > Hi Sylvain, This looks great! I would also note that all of the performance metrics produced by the testsuite, nofib, and (soon) head.hackage are preserved in a PostgreSQL database on gitlab.haskell.org which can be accessed by way of Postgrest [1]. The schema can be found here [2]. You will also find the import tools in the same repository (but they are quite rough around the edges; you have been warned). The schema is probably most conveniently used via the `results_view` view. For instance, one can request all metrics for commits in the last month with: $ curl https://perf.ghc.haskell.org/db/results_view?commit_date=gt.2020-09-01 Another useful view is `deltas`, which provides compares metrics for pairs of commits. This I have found quite useful in spotting regressions. Apart from curl, there are a few other options for consuming this information: * I have an Python notebook which has some convenient helpers for quickly plotting things in `ghc-utils` [3]. I find this option to have the greatest power/weight ratio for most use-cases. * There is an incredibly hacky attempt at a web interface here [4] (source here [5]) although it's probably best to bury this sad effort * Sometimes I have needed to fall back to SQL for some queries; if someone would find this useful we can arrange read-only SQL access on an individual basis. Regardless, this project has been gradually evolving as the need arises for the last few years. It's been quite useful but I think it's potential is quite under-realized. Cheers, - Ben [1] https://postgrest.org/ [2] https://github.com/bgamari/ghc-perf-import/blob/master/import/schema.sql [3] https://github.com/bgamari/ghc-perf-import/blob/master/plot.ipynb [4] http://home.smart-cactus.org/ghc-perf/ [5] https://github.com/bgamari/ghc-perf-import/tree/master/web -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 487 bytes Desc: not available URL: From klebinger.andreas at gmx.at Thu Oct 15 09:31:53 2020 From: klebinger.andreas at gmx.at (Andreas Klebinger) Date: Thu, 15 Oct 2020 11:31:53 +0200 Subject: Fwd: Restricted sums in BoxedRep In-Reply-To: References: Message-ID: <03c327e0-edae-c4bb-aaef-66dc5f53cd9d@gmx.at> From a implementors perspective my main questions would be: * How big is the benefit in practice? How many use cases are there? * How bad are the costs? (Runtime overhead, rts complexity, ...) The details of how this would be exposed to a user would are important. But if the costs are too high for the drawbacks then it becomes a moot point. David Feuer schrieb am 14.10.2020 um 22:21: > Forwarded from Andrew Martin below. I think we want more than just > Maybe (more than one null), but the nesting I described is certainly > more convenience than necessity. > > ---------- Forwarded message --------- > From: *Andrew Martin* > > Date: Wed, Oct 14, 2020, 4:14 PM > Subject: Re: Restricted sums in BoxedRep > To: David Feuer > > > > You'll have to forward this to the ghc-devs list to share it with > others since I'm not currently subscribed to it, but I've had this > same thought before. It is discussed at > https://github.com/andrewthad/impure-containers/issues/12. Here's the > relevant excerpt: > > Relatedly, I was thinking the other day that after finishing > implementing > https://github.com/ghc-proposals/ghc-proposals/blob/master/proposals/0203-pointer-rep.rst, > I should really look at seeing if it's possible to add this > maybe-of-a-lifted value trick straight to GHC. I think that with: > > |data RuntimpRep = BoxedRep Levity | MaybeBoxedRep Levity | IntRep > | ... data BuiltinMaybe :: forall (v :: Levity). TYPE v -> TYPE > ('MaybeBoxedRep v) | > > This doesn't have the nesting issues because the kind system > prevents nesting. But anyway, back to the original question. I > would recommend not using |Maybe.Unsafe| and using > |unpacked-maybe| instead. The latter is definitely safe, and it > only costs an extra machine word of space in each data constructor > it gets used in, and it doesn't introduce more indirections. > > > On Tue, Oct 13, 2020 at 5:47 PM David Feuer > wrote: > > Null pointers are widely known to be a lousy language feature in > general, but there are certain situations where they're *really* > useful for compact representation. For example, we define > >     newtype TMVar a = TMVar (TVar (Maybe a)) > > We don't, however, actually use the fact that (Maybe a) is lifted. > So we could represent this much more efficiently using something like > >     newtype TMVar a = TMVar (TVar a) > > where Nothing is represented by a distinguished "null" pointer. > > While it's possible to implement this sort of thing in user code > (with lots of fuss and care), it's not very nice at all. What I'd > really like to be able to do is represent certain kinds of sums > like this natively. > > Now that we're getting BoxedRep, I think we can probably make it > happen. The trick is to add a special Levity constructor > representing sums of particular shapes. Specifically, we can > represent a type like this if it is a possibly-nested sum which, > when flattened into a single sum, consists of some number of > nullary tuples and at most one Lifted or Unlifted type.  Then we > can have (inline) primops to convert between the BoxedRep and the > sum-of-sums representations. > > Anyone have thoughts on details for what the Levity constructor > arguments might look like? > > > > -- > -Andrew Thaddeus Martin > > > _______________________________________________ > ghc-devs mailing list > ghc-devs at haskell.org > http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs -------------- next part -------------- An HTML attachment was scrubbed... URL: From carter.schonwald at gmail.com Thu Oct 15 15:44:06 2020 From: carter.schonwald at gmail.com (Carter Schonwald) Date: Thu, 15 Oct 2020 11:44:06 -0400 Subject: Fwd: Restricted sums in BoxedRep In-Reply-To: <03c327e0-edae-c4bb-aaef-66dc5f53cd9d@gmx.at> References: <03c327e0-edae-c4bb-aaef-66dc5f53cd9d@gmx.at> Message-ID: A related idea that came up recently and is perhaps simpler ties into this via the lens of having unboxed Mvars/tvars (even if it’s restricted to just things we can embed in a word64#) This came up in https://gitlab.haskell.org/ghc/ghc/-/issues/18798#note_307410, where viktor had millions of independent mvars holding what’s essentially a strict unit ()! The motivation in this later scenario is that in high concurrency settings, the less trivial stuff the gc needs to trace under updates, the better ghc scales. This may not be a use case david has in mind, but certainly seems related. Phrased more succinctly: gc perf dominates large heap / many core computation in Haskell via sensitivity to allocation volume / mutation volume (to ensure generational hypothesis stays valid), and providing tools to incrementally reduce the pressure with local changes would be good. So I’d propose / suggest that a baby step towards what david asks would be for us to work out some manner of unboxed tvar/mvar ref machinery that supports unboxed values. On Thu, Oct 15, 2020 at 5:32 AM Andreas Klebinger wrote: > From a implementors perspective my main questions would be: > > * How big is the benefit in practice? How many use cases are there? > * How bad are the costs? (Runtime overhead, rts complexity, ...) > > The details of how this would be exposed to a user would are important. > But if the costs are too high for the drawbacks then it becomes a moot > point. > > > David Feuer schrieb am 14.10.2020 um 22:21: > > Forwarded from Andrew Martin below. I think we want more than just Maybe > (more than one null), but the nesting I described is certainly more > convenience than necessity. > > ---------- Forwarded message --------- > From: Andrew Martin > Date: Wed, Oct 14, 2020, 4:14 PM > Subject: Re: Restricted sums in BoxedRep > To: David Feuer > > > You'll have to forward this to the ghc-devs list to share it with others > since I'm not currently subscribed to it, but I've had this same thought > before. It is discussed at > https://github.com/andrewthad/impure-containers/issues/12. Here's the > relevant excerpt: > >> Relatedly, I was thinking the other day that after finishing implementing >> https://github.com/ghc-proposals/ghc-proposals/blob/master/proposals/0203-pointer-rep.rst, >> I should really look at seeing if it's possible to add this >> maybe-of-a-lifted value trick straight to GHC. I think that with: >> >> data RuntimpRep >> = BoxedRep Levity >> | MaybeBoxedRep Levity >> | IntRep >> | ... >> >> data BuiltinMaybe :: forall (v :: Levity). TYPE v -> TYPE ('MaybeBoxedRep v) >> >> This doesn't have the nesting issues because the kind system prevents >> nesting. But anyway, back to the original question. I would recommend not >> using Maybe.Unsafe and using unpacked-maybe instead. The latter is >> definitely safe, and it only costs an extra machine word of space in each >> data constructor it gets used in, and it doesn't introduce more >> indirections. >> > > On Tue, Oct 13, 2020 at 5:47 PM David Feuer wrote: > >> Null pointers are widely known to be a lousy language feature in general, >> but there are certain situations where they're *really* useful for compact >> representation. For example, we define >> >> newtype TMVar a = TMVar (TVar (Maybe a)) >> >> We don't, however, actually use the fact that (Maybe a) is lifted. So we >> could represent this much more efficiently using something like >> >> newtype TMVar a = TMVar (TVar a) >> >> where Nothing is represented by a distinguished "null" pointer. >> >> While it's possible to implement this sort of thing in user code (with >> lots of fuss and care), it's not very nice at all. What I'd really like to >> be able to do is represent certain kinds of sums like this natively. >> >> Now that we're getting BoxedRep, I think we can probably make it happen. >> The trick is to add a special Levity constructor representing sums of >> particular shapes. Specifically, we can represent a type like this if it is >> a possibly-nested sum which, when flattened into a single sum, consists of >> some number of nullary tuples and at most one Lifted or Unlifted type. >> Then we can have (inline) primops to convert between the BoxedRep and the >> sum-of-sums representations. >> >> Anyone have thoughts on details for what the Levity constructor arguments >> might look like? >> > > > -- > -Andrew Thaddeus Martin > > > _______________________________________________ > ghc-devs mailing listghc-devs at haskell.orghttp://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs > > > _______________________________________________ > ghc-devs mailing list > ghc-devs at haskell.org > http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs > -------------- next part -------------- An HTML attachment was scrubbed... URL: From david.feuer at gmail.com Thu Oct 15 15:57:27 2020 From: david.feuer at gmail.com (David Feuer) Date: Thu, 15 Oct 2020 11:57:27 -0400 Subject: Fwd: Restricted sums in BoxedRep In-Reply-To: References: <03c327e0-edae-c4bb-aaef-66dc5f53cd9d@gmx.at> Message-ID: Putting unboxed things in TVar, MVar, etc., is part of Andrew Martin's accepted BoxedRep proposal. On Thu, Oct 15, 2020, 11:44 AM Carter Schonwald wrote: > A related idea that came up recently and is perhaps simpler ties into this > via the lens of having unboxed Mvars/tvars (even if it’s restricted to just > things we can embed in a word64#) > > This came up in > https://gitlab.haskell.org/ghc/ghc/-/issues/18798#note_307410, where > viktor had millions of independent mvars holding what’s essentially a > strict unit ()! > > The motivation in this later scenario is that in high concurrency > settings, the less trivial stuff the gc needs to trace under updates, the > better ghc scales. > > This may not be a use case david has in mind, but certainly seems related. > > > Phrased more succinctly: gc perf dominates large heap / many core > computation in Haskell via sensitivity to allocation volume / mutation > volume (to ensure generational hypothesis stays valid), and providing tools > to incrementally reduce the pressure with local changes would be good. > > So I’d propose / suggest that a baby step towards what david asks would be > for us to work out some manner of unboxed tvar/mvar ref machinery that > supports unboxed values. > > > > On Thu, Oct 15, 2020 at 5:32 AM Andreas Klebinger < > klebinger.andreas at gmx.at> wrote: > >> From a implementors perspective my main questions would be: >> >> * How big is the benefit in practice? How many use cases are there? >> * How bad are the costs? (Runtime overhead, rts complexity, ...) >> >> The details of how this would be exposed to a user would are important. >> But if the costs are too high for the drawbacks then it becomes a moot >> point. >> >> >> David Feuer schrieb am 14.10.2020 um 22:21: >> >> Forwarded from Andrew Martin below. I think we want more than just Maybe >> (more than one null), but the nesting I described is certainly more >> convenience than necessity. >> >> ---------- Forwarded message --------- >> From: Andrew Martin >> Date: Wed, Oct 14, 2020, 4:14 PM >> Subject: Re: Restricted sums in BoxedRep >> To: David Feuer >> >> >> You'll have to forward this to the ghc-devs list to share it with others >> since I'm not currently subscribed to it, but I've had this same thought >> before. It is discussed at >> https://github.com/andrewthad/impure-containers/issues/12. Here's the >> relevant excerpt: >> >>> Relatedly, I was thinking the other day that after finishing >>> implementing >>> https://github.com/ghc-proposals/ghc-proposals/blob/master/proposals/0203-pointer-rep.rst, >>> I should really look at seeing if it's possible to add this >>> maybe-of-a-lifted value trick straight to GHC. I think that with: >>> >>> data RuntimpRep >>> = BoxedRep Levity >>> | MaybeBoxedRep Levity >>> | IntRep >>> | ... >>> >>> data BuiltinMaybe :: forall (v :: Levity). TYPE v -> TYPE ('MaybeBoxedRep v) >>> >>> This doesn't have the nesting issues because the kind system prevents >>> nesting. But anyway, back to the original question. I would recommend not >>> using Maybe.Unsafe and using unpacked-maybe instead. The latter is >>> definitely safe, and it only costs an extra machine word of space in each >>> data constructor it gets used in, and it doesn't introduce more >>> indirections. >>> >> >> On Tue, Oct 13, 2020 at 5:47 PM David Feuer >> wrote: >> >>> Null pointers are widely known to be a lousy language feature in >>> general, but there are certain situations where they're *really* useful for >>> compact representation. For example, we define >>> >>> newtype TMVar a = TMVar (TVar (Maybe a)) >>> >>> We don't, however, actually use the fact that (Maybe a) is lifted. So we >>> could represent this much more efficiently using something like >>> >>> newtype TMVar a = TMVar (TVar a) >>> >>> where Nothing is represented by a distinguished "null" pointer. >>> >>> While it's possible to implement this sort of thing in user code (with >>> lots of fuss and care), it's not very nice at all. What I'd really like to >>> be able to do is represent certain kinds of sums like this natively. >>> >>> Now that we're getting BoxedRep, I think we can probably make it happen. >>> The trick is to add a special Levity constructor representing sums of >>> particular shapes. Specifically, we can represent a type like this if it is >>> a possibly-nested sum which, when flattened into a single sum, consists of >>> some number of nullary tuples and at most one Lifted or Unlifted type. >>> Then we can have (inline) primops to convert between the BoxedRep and the >>> sum-of-sums representations. >>> >>> Anyone have thoughts on details for what the Levity constructor >>> arguments might look like? >>> >> >> >> -- >> -Andrew Thaddeus Martin >> >> >> _______________________________________________ >> ghc-devs mailing listghc-devs at haskell.orghttp://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs >> >> >> _______________________________________________ >> ghc-devs mailing list >> ghc-devs at haskell.org >> http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From david.feuer at gmail.com Thu Oct 15 15:57:57 2020 From: david.feuer at gmail.com (David Feuer) Date: Thu, 15 Oct 2020 11:57:57 -0400 Subject: Fwd: Restricted sums in BoxedRep In-Reply-To: References: <03c327e0-edae-c4bb-aaef-66dc5f53cd9d@gmx.at> Message-ID: Sorry, unlifted, not unboxed... On Thu, Oct 15, 2020, 11:57 AM David Feuer wrote: > Putting unboxed things in TVar, MVar, etc., is part of Andrew Martin's > accepted BoxedRep proposal. > > On Thu, Oct 15, 2020, 11:44 AM Carter Schonwald < > carter.schonwald at gmail.com> wrote: > >> A related idea that came up recently and is perhaps simpler ties into >> this via the lens of having unboxed Mvars/tvars (even if it’s restricted to >> just things we can embed in a word64#) >> >> This came up in >> https://gitlab.haskell.org/ghc/ghc/-/issues/18798#note_307410, where >> viktor had millions of independent mvars holding what’s essentially a >> strict unit ()! >> >> The motivation in this later scenario is that in high concurrency >> settings, the less trivial stuff the gc needs to trace under updates, the >> better ghc scales. >> >> This may not be a use case david has in mind, but certainly seems >> related. >> >> Phrased more succinctly: gc perf dominates large heap / many core >> computation in Haskell via sensitivity to allocation volume / mutation >> volume (to ensure generational hypothesis stays valid), and providing tools >> to incrementally reduce the pressure with local changes would be good. >> >> So I’d propose / suggest that a baby step towards what david asks would >> be for us to work out some manner of unboxed tvar/mvar ref machinery that >> supports unboxed values. >> >> >> >> On Thu, Oct 15, 2020 at 5:32 AM Andreas Klebinger < >> klebinger.andreas at gmx.at> wrote: >> >>> From a implementors perspective my main questions would be: >>> >>> * How big is the benefit in practice? How many use cases are there? >>> * How bad are the costs? (Runtime overhead, rts complexity, ...) >>> >>> The details of how this would be exposed to a user would are important. >>> But if the costs are too high for the drawbacks then it becomes a moot >>> point. >>> >>> >>> David Feuer schrieb am 14.10.2020 um 22:21: >>> >>> Forwarded from Andrew Martin below. I think we want more than just Maybe >>> (more than one null), but the nesting I described is certainly more >>> convenience than necessity. >>> >>> ---------- Forwarded message --------- >>> From: Andrew Martin >>> Date: Wed, Oct 14, 2020, 4:14 PM >>> Subject: Re: Restricted sums in BoxedRep >>> To: David Feuer >>> >>> >>> You'll have to forward this to the ghc-devs list to share it with others >>> since I'm not currently subscribed to it, but I've had this same thought >>> before. It is discussed at >>> https://github.com/andrewthad/impure-containers/issues/12. Here's the >>> relevant excerpt: >>> >>>> Relatedly, I was thinking the other day that after finishing >>>> implementing >>>> https://github.com/ghc-proposals/ghc-proposals/blob/master/proposals/0203-pointer-rep.rst, >>>> I should really look at seeing if it's possible to add this >>>> maybe-of-a-lifted value trick straight to GHC. I think that with: >>>> >>>> data RuntimpRep >>>> = BoxedRep Levity >>>> | MaybeBoxedRep Levity >>>> | IntRep >>>> | ... >>>> >>>> data BuiltinMaybe :: forall (v :: Levity). TYPE v -> TYPE ('MaybeBoxedRep v) >>>> >>>> This doesn't have the nesting issues because the kind system prevents >>>> nesting. But anyway, back to the original question. I would recommend not >>>> using Maybe.Unsafe and using unpacked-maybe instead. The latter is >>>> definitely safe, and it only costs an extra machine word of space in each >>>> data constructor it gets used in, and it doesn't introduce more >>>> indirections. >>>> >>> >>> On Tue, Oct 13, 2020 at 5:47 PM David Feuer >>> wrote: >>> >>>> Null pointers are widely known to be a lousy language feature in >>>> general, but there are certain situations where they're *really* useful for >>>> compact representation. For example, we define >>>> >>>> newtype TMVar a = TMVar (TVar (Maybe a)) >>>> >>>> We don't, however, actually use the fact that (Maybe a) is lifted. So >>>> we could represent this much more efficiently using something like >>>> >>>> newtype TMVar a = TMVar (TVar a) >>>> >>>> where Nothing is represented by a distinguished "null" pointer. >>>> >>>> While it's possible to implement this sort of thing in user code (with >>>> lots of fuss and care), it's not very nice at all. What I'd really like to >>>> be able to do is represent certain kinds of sums like this natively. >>>> >>>> Now that we're getting BoxedRep, I think we can probably make it >>>> happen. The trick is to add a special Levity constructor representing sums >>>> of particular shapes. Specifically, we can represent a type like this if it >>>> is a possibly-nested sum which, when flattened into a single sum, consists >>>> of some number of nullary tuples and at most one Lifted or Unlifted type. >>>> Then we can have (inline) primops to convert between the BoxedRep and the >>>> sum-of-sums representations. >>>> >>>> Anyone have thoughts on details for what the Levity constructor >>>> arguments might look like? >>>> >>> >>> >>> -- >>> -Andrew Thaddeus Martin >>> >>> >>> _______________________________________________ >>> ghc-devs mailing listghc-devs at haskell.orghttp://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs >>> >>> >>> _______________________________________________ >>> ghc-devs mailing list >>> ghc-devs at haskell.org >>> http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs >>> >> -------------- next part -------------- An HTML attachment was scrubbed... URL: From carter.schonwald at gmail.com Thu Oct 15 16:50:15 2020 From: carter.schonwald at gmail.com (Carter Schonwald) Date: Thu, 15 Oct 2020 12:50:15 -0400 Subject: Fwd: Restricted sums in BoxedRep In-Reply-To: References: <03c327e0-edae-c4bb-aaef-66dc5f53cd9d@gmx.at> Message-ID: Indeed, I mean things that aren’t pointery, and could be represented by a tvar paired with a mutable byte array or mvar with mutable byte array, but which we’d want considered as a single heap object from the rts/gc perspective. On Thu, Oct 15, 2020 at 11:58 AM David Feuer wrote: > Sorry, unlifted, not unboxed... > > On Thu, Oct 15, 2020, 11:57 AM David Feuer wrote: > >> Putting unboxed things in TVar, MVar, etc., is part of Andrew Martin's >> accepted BoxedRep proposal. >> >> On Thu, Oct 15, 2020, 11:44 AM Carter Schonwald < >> carter.schonwald at gmail.com> wrote: >> >>> A related idea that came up recently and is perhaps simpler ties into >>> this via the lens of having unboxed Mvars/tvars (even if it’s restricted to >>> just things we can embed in a word64#) >>> >>> This came up in >>> https://gitlab.haskell.org/ghc/ghc/-/issues/18798#note_307410, where >>> viktor had millions of independent mvars holding what’s essentially a >>> strict unit ()! >>> >>> The motivation in this later scenario is that in high concurrency >>> settings, the less trivial stuff the gc needs to trace under updates, the >>> better ghc scales. >>> >>> This may not be a use case david has in mind, but certainly seems >>> related. >>> >>> Phrased more succinctly: gc perf dominates large heap / many core >>> computation in Haskell via sensitivity to allocation volume / mutation >>> volume (to ensure generational hypothesis stays valid), and providing tools >>> to incrementally reduce the pressure with local changes would be good. >>> >>> So I’d propose / suggest that a baby step towards what david asks would >>> be for us to work out some manner of unboxed tvar/mvar ref machinery that >>> supports unboxed values. >>> >>> >>> >>> On Thu, Oct 15, 2020 at 5:32 AM Andreas Klebinger < >>> klebinger.andreas at gmx.at> wrote: >>> >>>> From a implementors perspective my main questions would be: >>>> >>>> * How big is the benefit in practice? How many use cases are there? >>>> * How bad are the costs? (Runtime overhead, rts complexity, ...) >>>> >>>> The details of how this would be exposed to a user would are important. >>>> But if the costs are too high for the drawbacks then it becomes a moot >>>> point. >>>> >>>> >>>> David Feuer schrieb am 14.10.2020 um 22:21: >>>> >>>> Forwarded from Andrew Martin below. I think we want more than just >>>> Maybe (more than one null), but the nesting I described is certainly more >>>> convenience than necessity. >>>> >>>> ---------- Forwarded message --------- >>>> From: Andrew Martin >>>> Date: Wed, Oct 14, 2020, 4:14 PM >>>> Subject: Re: Restricted sums in BoxedRep >>>> To: David Feuer >>>> >>>> >>>> You'll have to forward this to the ghc-devs list to share it with >>>> others since I'm not currently subscribed to it, but I've had this same >>>> thought before. It is discussed at >>>> https://github.com/andrewthad/impure-containers/issues/12. Here's the >>>> relevant excerpt: >>>> >>>>> Relatedly, I was thinking the other day that after finishing >>>>> implementing >>>>> https://github.com/ghc-proposals/ghc-proposals/blob/master/proposals/0203-pointer-rep.rst, >>>>> I should really look at seeing if it's possible to add this >>>>> maybe-of-a-lifted value trick straight to GHC. I think that with: >>>>> >>>>> data RuntimpRep >>>>> = BoxedRep Levity >>>>> | MaybeBoxedRep Levity >>>>> | IntRep >>>>> | ... >>>>> >>>>> data BuiltinMaybe :: forall (v :: Levity). TYPE v -> TYPE ('MaybeBoxedRep v) >>>>> >>>>> This doesn't have the nesting issues because the kind system prevents >>>>> nesting. But anyway, back to the original question. I would recommend not >>>>> using Maybe.Unsafe and using unpacked-maybe instead. The latter is >>>>> definitely safe, and it only costs an extra machine word of space in each >>>>> data constructor it gets used in, and it doesn't introduce more >>>>> indirections. >>>>> >>>> >>>> On Tue, Oct 13, 2020 at 5:47 PM David Feuer >>>> wrote: >>>> >>>>> Null pointers are widely known to be a lousy language feature in >>>>> general, but there are certain situations where they're *really* useful for >>>>> compact representation. For example, we define >>>>> >>>>> newtype TMVar a = TMVar (TVar (Maybe a)) >>>>> >>>>> We don't, however, actually use the fact that (Maybe a) is lifted. So >>>>> we could represent this much more efficiently using something like >>>>> >>>>> newtype TMVar a = TMVar (TVar a) >>>>> >>>>> where Nothing is represented by a distinguished "null" pointer. >>>>> >>>>> While it's possible to implement this sort of thing in user code (with >>>>> lots of fuss and care), it's not very nice at all. What I'd really like to >>>>> be able to do is represent certain kinds of sums like this natively. >>>>> >>>>> Now that we're getting BoxedRep, I think we can probably make it >>>>> happen. The trick is to add a special Levity constructor representing sums >>>>> of particular shapes. Specifically, we can represent a type like this if it >>>>> is a possibly-nested sum which, when flattened into a single sum, consists >>>>> of some number of nullary tuples and at most one Lifted or Unlifted type. >>>>> Then we can have (inline) primops to convert between the BoxedRep and the >>>>> sum-of-sums representations. >>>>> >>>>> Anyone have thoughts on details for what the Levity constructor >>>>> arguments might look like? >>>>> >>>> >>>> >>>> -- >>>> -Andrew Thaddeus Martin >>>> >>>> >>>> _______________________________________________ >>>> ghc-devs mailing listghc-devs at haskell.orghttp://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs >>>> >>>> >>>> _______________________________________________ >>>> ghc-devs mailing list >>>> ghc-devs at haskell.org >>>> http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs >>>> >>> -------------- next part -------------- An HTML attachment was scrubbed... URL: From david.feuer at gmail.com Thu Oct 15 16:58:05 2020 From: david.feuer at gmail.com (David Feuer) Date: Thu, 15 Oct 2020 12:58:05 -0400 Subject: Fwd: Restricted sums in BoxedRep In-Reply-To: References: <03c327e0-edae-c4bb-aaef-66dc5f53cd9d@gmx.at> Message-ID: Yes, that's something quite different. We'd need a whole different heap object type for such MVars and TVars. My approach covers the case where the unboxed thing can only take on a few values, for some value of "a few" which, depending on implementation, may or may not be very small. If the nulls point to actual heap objects in pre-allocated pinned memory (say), then up to 64 or so might be reasonable. If they point to "invalid" address space, then the numbers could go up a good bit. On Thu, Oct 15, 2020, 12:50 PM Carter Schonwald wrote: > Indeed, I mean things that aren’t pointery, and could be represented by a > tvar paired with a mutable byte array or mvar with mutable byte array, but > which we’d want considered as a single heap object from the rts/gc > perspective. > > On Thu, Oct 15, 2020 at 11:58 AM David Feuer > wrote: > >> Sorry, unlifted, not unboxed... >> >> On Thu, Oct 15, 2020, 11:57 AM David Feuer wrote: >> >>> Putting unboxed things in TVar, MVar, etc., is part of Andrew Martin's >>> accepted BoxedRep proposal. >>> >>> On Thu, Oct 15, 2020, 11:44 AM Carter Schonwald < >>> carter.schonwald at gmail.com> wrote: >>> >>>> A related idea that came up recently and is perhaps simpler ties into >>>> this via the lens of having unboxed Mvars/tvars (even if it’s restricted to >>>> just things we can embed in a word64#) >>>> >>>> This came up in >>>> https://gitlab.haskell.org/ghc/ghc/-/issues/18798#note_307410, where >>>> viktor had millions of independent mvars holding what’s essentially a >>>> strict unit ()! >>>> >>>> The motivation in this later scenario is that in high concurrency >>>> settings, the less trivial stuff the gc needs to trace under updates, the >>>> better ghc scales. >>>> >>>> This may not be a use case david has in mind, but certainly seems >>>> related. >>>> >>>> Phrased more succinctly: gc perf dominates large heap / many core >>>> computation in Haskell via sensitivity to allocation volume / mutation >>>> volume (to ensure generational hypothesis stays valid), and providing tools >>>> to incrementally reduce the pressure with local changes would be good. >>>> >>>> So I’d propose / suggest that a baby step towards what david asks would >>>> be for us to work out some manner of unboxed tvar/mvar ref machinery that >>>> supports unboxed values. >>>> >>>> >>>> >>>> On Thu, Oct 15, 2020 at 5:32 AM Andreas Klebinger < >>>> klebinger.andreas at gmx.at> wrote: >>>> >>>>> From a implementors perspective my main questions would be: >>>>> >>>>> * How big is the benefit in practice? How many use cases are there? >>>>> * How bad are the costs? (Runtime overhead, rts complexity, ...) >>>>> >>>>> The details of how this would be exposed to a user would are important. >>>>> But if the costs are too high for the drawbacks then it becomes a moot >>>>> point. >>>>> >>>>> >>>>> David Feuer schrieb am 14.10.2020 um 22:21: >>>>> >>>>> Forwarded from Andrew Martin below. I think we want more than just >>>>> Maybe (more than one null), but the nesting I described is certainly more >>>>> convenience than necessity. >>>>> >>>>> ---------- Forwarded message --------- >>>>> From: Andrew Martin >>>>> Date: Wed, Oct 14, 2020, 4:14 PM >>>>> Subject: Re: Restricted sums in BoxedRep >>>>> To: David Feuer >>>>> >>>>> >>>>> You'll have to forward this to the ghc-devs list to share it with >>>>> others since I'm not currently subscribed to it, but I've had this same >>>>> thought before. It is discussed at >>>>> https://github.com/andrewthad/impure-containers/issues/12. Here's the >>>>> relevant excerpt: >>>>> >>>>>> Relatedly, I was thinking the other day that after finishing >>>>>> implementing >>>>>> https://github.com/ghc-proposals/ghc-proposals/blob/master/proposals/0203-pointer-rep.rst, >>>>>> I should really look at seeing if it's possible to add this >>>>>> maybe-of-a-lifted value trick straight to GHC. I think that with: >>>>>> >>>>>> data RuntimpRep >>>>>> = BoxedRep Levity >>>>>> | MaybeBoxedRep Levity >>>>>> | IntRep >>>>>> | ... >>>>>> >>>>>> data BuiltinMaybe :: forall (v :: Levity). TYPE v -> TYPE ('MaybeBoxedRep v) >>>>>> >>>>>> This doesn't have the nesting issues because the kind system prevents >>>>>> nesting. But anyway, back to the original question. I would recommend not >>>>>> using Maybe.Unsafe and using unpacked-maybe instead. The latter is >>>>>> definitely safe, and it only costs an extra machine word of space in each >>>>>> data constructor it gets used in, and it doesn't introduce more >>>>>> indirections. >>>>>> >>>>> >>>>> On Tue, Oct 13, 2020 at 5:47 PM David Feuer >>>>> wrote: >>>>> >>>>>> Null pointers are widely known to be a lousy language feature in >>>>>> general, but there are certain situations where they're *really* useful for >>>>>> compact representation. For example, we define >>>>>> >>>>>> newtype TMVar a = TMVar (TVar (Maybe a)) >>>>>> >>>>>> We don't, however, actually use the fact that (Maybe a) is lifted. So >>>>>> we could represent this much more efficiently using something like >>>>>> >>>>>> newtype TMVar a = TMVar (TVar a) >>>>>> >>>>>> where Nothing is represented by a distinguished "null" pointer. >>>>>> >>>>>> While it's possible to implement this sort of thing in user code >>>>>> (with lots of fuss and care), it's not very nice at all. What I'd really >>>>>> like to be able to do is represent certain kinds of sums like this natively. >>>>>> >>>>>> Now that we're getting BoxedRep, I think we can probably make it >>>>>> happen. The trick is to add a special Levity constructor representing sums >>>>>> of particular shapes. Specifically, we can represent a type like this if it >>>>>> is a possibly-nested sum which, when flattened into a single sum, consists >>>>>> of some number of nullary tuples and at most one Lifted or Unlifted type. >>>>>> Then we can have (inline) primops to convert between the BoxedRep and the >>>>>> sum-of-sums representations. >>>>>> >>>>>> Anyone have thoughts on details for what the Levity constructor >>>>>> arguments might look like? >>>>>> >>>>> >>>>> >>>>> -- >>>>> -Andrew Thaddeus Martin >>>>> >>>>> >>>>> _______________________________________________ >>>>> ghc-devs mailing listghc-devs at haskell.orghttp://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs >>>>> >>>>> >>>>> _______________________________________________ >>>>> ghc-devs mailing list >>>>> ghc-devs at haskell.org >>>>> http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs >>>>> >>>> -------------- next part -------------- An HTML attachment was scrubbed... URL: From fryguybob at gmail.com Thu Oct 15 17:10:16 2020 From: fryguybob at gmail.com (Ryan Yates) Date: Thu, 15 Oct 2020 13:10:16 -0400 Subject: Fwd: Restricted sums in BoxedRep In-Reply-To: References: <03c327e0-edae-c4bb-aaef-66dc5f53cd9d@gmx.at> Message-ID: I haven't been following this discussion too closely, but in my research work I found that some of the benefits that I wanted in this direction were already there with pointer tagging. On Thu, Oct 15, 2020 at 12:59 PM David Feuer wrote: > Yes, that's something quite different. We'd need a whole different heap > object type for such MVars and TVars. My approach covers the case where the > unboxed thing can only take on a few values, for some value of "a few" > which, depending on implementation, may or may not be very small. If the > nulls point to actual heap objects in pre-allocated pinned memory (say), > then up to 64 or so might be reasonable. If they point to "invalid" address > space, then the numbers could go up a good bit. > > On Thu, Oct 15, 2020, 12:50 PM Carter Schonwald < > carter.schonwald at gmail.com> wrote: > >> Indeed, I mean things that aren’t pointery, and could be represented by a >> tvar paired with a mutable byte array or mvar with mutable byte array, but >> which we’d want considered as a single heap object from the rts/gc >> perspective. >> >> On Thu, Oct 15, 2020 at 11:58 AM David Feuer >> wrote: >> >>> Sorry, unlifted, not unboxed... >>> >>> On Thu, Oct 15, 2020, 11:57 AM David Feuer >>> wrote: >>> >>>> Putting unboxed things in TVar, MVar, etc., is part of Andrew Martin's >>>> accepted BoxedRep proposal. >>>> >>>> On Thu, Oct 15, 2020, 11:44 AM Carter Schonwald < >>>> carter.schonwald at gmail.com> wrote: >>>> >>>>> A related idea that came up recently and is perhaps simpler ties into >>>>> this via the lens of having unboxed Mvars/tvars (even if it’s restricted to >>>>> just things we can embed in a word64#) >>>>> >>>>> This came up in >>>>> https://gitlab.haskell.org/ghc/ghc/-/issues/18798#note_307410, where >>>>> viktor had millions of independent mvars holding what’s essentially a >>>>> strict unit ()! >>>>> >>>>> The motivation in this later scenario is that in high concurrency >>>>> settings, the less trivial stuff the gc needs to trace under updates, the >>>>> better ghc scales. >>>>> >>>>> This may not be a use case david has in mind, but certainly seems >>>>> related. >>>>> >>>>> Phrased more succinctly: gc perf dominates large heap / many core >>>>> computation in Haskell via sensitivity to allocation volume / mutation >>>>> volume (to ensure generational hypothesis stays valid), and providing tools >>>>> to incrementally reduce the pressure with local changes would be good. >>>>> >>>>> So I’d propose / suggest that a baby step towards what david asks >>>>> would be for us to work out some manner of unboxed tvar/mvar ref machinery >>>>> that supports unboxed values. >>>>> >>>>> >>>>> >>>>> On Thu, Oct 15, 2020 at 5:32 AM Andreas Klebinger < >>>>> klebinger.andreas at gmx.at> wrote: >>>>> >>>>>> From a implementors perspective my main questions would be: >>>>>> >>>>>> * How big is the benefit in practice? How many use cases are there? >>>>>> * How bad are the costs? (Runtime overhead, rts complexity, ...) >>>>>> >>>>>> The details of how this would be exposed to a user would are >>>>>> important. >>>>>> But if the costs are too high for the drawbacks then it becomes a >>>>>> moot point. >>>>>> >>>>>> >>>>>> David Feuer schrieb am 14.10.2020 um 22:21: >>>>>> >>>>>> Forwarded from Andrew Martin below. I think we want more than just >>>>>> Maybe (more than one null), but the nesting I described is certainly more >>>>>> convenience than necessity. >>>>>> >>>>>> ---------- Forwarded message --------- >>>>>> From: Andrew Martin >>>>>> Date: Wed, Oct 14, 2020, 4:14 PM >>>>>> Subject: Re: Restricted sums in BoxedRep >>>>>> To: David Feuer >>>>>> >>>>>> >>>>>> You'll have to forward this to the ghc-devs list to share it with >>>>>> others since I'm not currently subscribed to it, but I've had this same >>>>>> thought before. It is discussed at >>>>>> https://github.com/andrewthad/impure-containers/issues/12. Here's >>>>>> the relevant excerpt: >>>>>> >>>>>>> Relatedly, I was thinking the other day that after finishing >>>>>>> implementing >>>>>>> https://github.com/ghc-proposals/ghc-proposals/blob/master/proposals/0203-pointer-rep.rst, >>>>>>> I should really look at seeing if it's possible to add this >>>>>>> maybe-of-a-lifted value trick straight to GHC. I think that with: >>>>>>> >>>>>>> data RuntimpRep >>>>>>> = BoxedRep Levity >>>>>>> | MaybeBoxedRep Levity >>>>>>> | IntRep >>>>>>> | ... >>>>>>> >>>>>>> data BuiltinMaybe :: forall (v :: Levity). TYPE v -> TYPE ('MaybeBoxedRep v) >>>>>>> >>>>>>> This doesn't have the nesting issues because the kind system >>>>>>> prevents nesting. But anyway, back to the original question. I would >>>>>>> recommend not using Maybe.Unsafe and using unpacked-maybe instead. >>>>>>> The latter is definitely safe, and it only costs an extra machine word of >>>>>>> space in each data constructor it gets used in, and it doesn't introduce >>>>>>> more indirections. >>>>>>> >>>>>> >>>>>> On Tue, Oct 13, 2020 at 5:47 PM David Feuer >>>>>> wrote: >>>>>> >>>>>>> Null pointers are widely known to be a lousy language feature in >>>>>>> general, but there are certain situations where they're *really* useful for >>>>>>> compact representation. For example, we define >>>>>>> >>>>>>> newtype TMVar a = TMVar (TVar (Maybe a)) >>>>>>> >>>>>>> We don't, however, actually use the fact that (Maybe a) is lifted. >>>>>>> So we could represent this much more efficiently using something like >>>>>>> >>>>>>> newtype TMVar a = TMVar (TVar a) >>>>>>> >>>>>>> where Nothing is represented by a distinguished "null" pointer. >>>>>>> >>>>>>> While it's possible to implement this sort of thing in user code >>>>>>> (with lots of fuss and care), it's not very nice at all. What I'd really >>>>>>> like to be able to do is represent certain kinds of sums like this natively. >>>>>>> >>>>>>> Now that we're getting BoxedRep, I think we can probably make it >>>>>>> happen. The trick is to add a special Levity constructor representing sums >>>>>>> of particular shapes. Specifically, we can represent a type like this if it >>>>>>> is a possibly-nested sum which, when flattened into a single sum, consists >>>>>>> of some number of nullary tuples and at most one Lifted or Unlifted type. >>>>>>> Then we can have (inline) primops to convert between the BoxedRep and the >>>>>>> sum-of-sums representations. >>>>>>> >>>>>>> Anyone have thoughts on details for what the Levity constructor >>>>>>> arguments might look like? >>>>>>> >>>>>> >>>>>> >>>>>> -- >>>>>> -Andrew Thaddeus Martin >>>>>> >>>>>> >>>>>> _______________________________________________ >>>>>> ghc-devs mailing listghc-devs at haskell.orghttp://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs >>>>>> >>>>>> >>>>>> _______________________________________________ >>>>>> ghc-devs mailing list >>>>>> ghc-devs at haskell.org >>>>>> http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs >>>>>> >>>>> _______________________________________________ > ghc-devs mailing list > ghc-devs at haskell.org > http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs > -------------- next part -------------- An HTML attachment was scrubbed... URL: From david.feuer at gmail.com Thu Oct 15 20:08:16 2020 From: david.feuer at gmail.com (David Feuer) Date: Thu, 15 Oct 2020 16:08:16 -0400 Subject: Fwd: Restricted sums in BoxedRep In-Reply-To: References: <03c327e0-edae-c4bb-aaef-66dc5f53cd9d@gmx.at> Message-ID: This might be lost in the noise for MVar and TVar, but for arrays, there are tremendous advantages to cutting out the extra heap objects. On Thu, Oct 15, 2020, 1:10 PM Ryan Yates wrote: > I haven't been following this discussion too closely, but in my research > work I found that some of the benefits that I wanted in this direction were > already there with pointer tagging. > > On Thu, Oct 15, 2020 at 12:59 PM David Feuer > wrote: > >> Yes, that's something quite different. We'd need a whole different heap >> object type for such MVars and TVars. My approach covers the case where the >> unboxed thing can only take on a few values, for some value of "a few" >> which, depending on implementation, may or may not be very small. If the >> nulls point to actual heap objects in pre-allocated pinned memory (say), >> then up to 64 or so might be reasonable. If they point to "invalid" address >> space, then the numbers could go up a good bit. >> >> On Thu, Oct 15, 2020, 12:50 PM Carter Schonwald < >> carter.schonwald at gmail.com> wrote: >> >>> Indeed, I mean things that aren’t pointery, and could be represented by >>> a tvar paired with a mutable byte array or mvar with mutable byte array, >>> but which we’d want considered as a single heap object from the rts/gc >>> perspective. >>> >>> On Thu, Oct 15, 2020 at 11:58 AM David Feuer >>> wrote: >>> >>>> Sorry, unlifted, not unboxed... >>>> >>>> On Thu, Oct 15, 2020, 11:57 AM David Feuer >>>> wrote: >>>> >>>>> Putting unboxed things in TVar, MVar, etc., is part of Andrew Martin's >>>>> accepted BoxedRep proposal. >>>>> >>>>> On Thu, Oct 15, 2020, 11:44 AM Carter Schonwald < >>>>> carter.schonwald at gmail.com> wrote: >>>>> >>>>>> A related idea that came up recently and is perhaps simpler ties into >>>>>> this via the lens of having unboxed Mvars/tvars (even if it’s restricted to >>>>>> just things we can embed in a word64#) >>>>>> >>>>>> This came up in >>>>>> https://gitlab.haskell.org/ghc/ghc/-/issues/18798#note_307410, where >>>>>> viktor had millions of independent mvars holding what’s essentially a >>>>>> strict unit ()! >>>>>> >>>>>> The motivation in this later scenario is that in high concurrency >>>>>> settings, the less trivial stuff the gc needs to trace under updates, the >>>>>> better ghc scales. >>>>>> >>>>>> This may not be a use case david has in mind, but certainly seems >>>>>> related. >>>>>> >>>>>> Phrased more succinctly: gc perf dominates large heap / many core >>>>>> computation in Haskell via sensitivity to allocation volume / mutation >>>>>> volume (to ensure generational hypothesis stays valid), and providing tools >>>>>> to incrementally reduce the pressure with local changes would be good. >>>>>> >>>>>> So I’d propose / suggest that a baby step towards what david asks >>>>>> would be for us to work out some manner of unboxed tvar/mvar ref machinery >>>>>> that supports unboxed values. >>>>>> >>>>>> >>>>>> >>>>>> On Thu, Oct 15, 2020 at 5:32 AM Andreas Klebinger < >>>>>> klebinger.andreas at gmx.at> wrote: >>>>>> >>>>>>> From a implementors perspective my main questions would be: >>>>>>> >>>>>>> * How big is the benefit in practice? How many use cases are there? >>>>>>> * How bad are the costs? (Runtime overhead, rts complexity, ...) >>>>>>> >>>>>>> The details of how this would be exposed to a user would are >>>>>>> important. >>>>>>> But if the costs are too high for the drawbacks then it becomes a >>>>>>> moot point. >>>>>>> >>>>>>> >>>>>>> David Feuer schrieb am 14.10.2020 um 22:21: >>>>>>> >>>>>>> Forwarded from Andrew Martin below. I think we want more than just >>>>>>> Maybe (more than one null), but the nesting I described is certainly more >>>>>>> convenience than necessity. >>>>>>> >>>>>>> ---------- Forwarded message --------- >>>>>>> From: Andrew Martin >>>>>>> Date: Wed, Oct 14, 2020, 4:14 PM >>>>>>> Subject: Re: Restricted sums in BoxedRep >>>>>>> To: David Feuer >>>>>>> >>>>>>> >>>>>>> You'll have to forward this to the ghc-devs list to share it with >>>>>>> others since I'm not currently subscribed to it, but I've had this same >>>>>>> thought before. It is discussed at >>>>>>> https://github.com/andrewthad/impure-containers/issues/12. Here's >>>>>>> the relevant excerpt: >>>>>>> >>>>>>>> Relatedly, I was thinking the other day that after finishing >>>>>>>> implementing >>>>>>>> https://github.com/ghc-proposals/ghc-proposals/blob/master/proposals/0203-pointer-rep.rst, >>>>>>>> I should really look at seeing if it's possible to add this >>>>>>>> maybe-of-a-lifted value trick straight to GHC. I think that with: >>>>>>>> >>>>>>>> data RuntimpRep >>>>>>>> = BoxedRep Levity >>>>>>>> | MaybeBoxedRep Levity >>>>>>>> | IntRep >>>>>>>> | ... >>>>>>>> >>>>>>>> data BuiltinMaybe :: forall (v :: Levity). TYPE v -> TYPE ('MaybeBoxedRep v) >>>>>>>> >>>>>>>> This doesn't have the nesting issues because the kind system >>>>>>>> prevents nesting. But anyway, back to the original question. I would >>>>>>>> recommend not using Maybe.Unsafe and using unpacked-maybe instead. >>>>>>>> The latter is definitely safe, and it only costs an extra machine word of >>>>>>>> space in each data constructor it gets used in, and it doesn't introduce >>>>>>>> more indirections. >>>>>>>> >>>>>>> >>>>>>> On Tue, Oct 13, 2020 at 5:47 PM David Feuer >>>>>>> wrote: >>>>>>> >>>>>>>> Null pointers are widely known to be a lousy language feature in >>>>>>>> general, but there are certain situations where they're *really* useful for >>>>>>>> compact representation. For example, we define >>>>>>>> >>>>>>>> newtype TMVar a = TMVar (TVar (Maybe a)) >>>>>>>> >>>>>>>> We don't, however, actually use the fact that (Maybe a) is lifted. >>>>>>>> So we could represent this much more efficiently using something like >>>>>>>> >>>>>>>> newtype TMVar a = TMVar (TVar a) >>>>>>>> >>>>>>>> where Nothing is represented by a distinguished "null" pointer. >>>>>>>> >>>>>>>> While it's possible to implement this sort of thing in user code >>>>>>>> (with lots of fuss and care), it's not very nice at all. What I'd really >>>>>>>> like to be able to do is represent certain kinds of sums like this natively. >>>>>>>> >>>>>>>> Now that we're getting BoxedRep, I think we can probably make it >>>>>>>> happen. The trick is to add a special Levity constructor representing sums >>>>>>>> of particular shapes. Specifically, we can represent a type like this if it >>>>>>>> is a possibly-nested sum which, when flattened into a single sum, consists >>>>>>>> of some number of nullary tuples and at most one Lifted or Unlifted type. >>>>>>>> Then we can have (inline) primops to convert between the BoxedRep and the >>>>>>>> sum-of-sums representations. >>>>>>>> >>>>>>>> Anyone have thoughts on details for what the Levity constructor >>>>>>>> arguments might look like? >>>>>>>> >>>>>>> >>>>>>> >>>>>>> -- >>>>>>> -Andrew Thaddeus Martin >>>>>>> >>>>>>> >>>>>>> _______________________________________________ >>>>>>> ghc-devs mailing listghc-devs at haskell.orghttp://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs >>>>>>> >>>>>>> >>>>>>> _______________________________________________ >>>>>>> ghc-devs mailing list >>>>>>> ghc-devs at haskell.org >>>>>>> http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs >>>>>>> >>>>>> _______________________________________________ >> ghc-devs mailing list >> ghc-devs at haskell.org >> http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From fryguybob at gmail.com Thu Oct 15 20:23:44 2020 From: fryguybob at gmail.com (Ryan Yates) Date: Thu, 15 Oct 2020 16:23:44 -0400 Subject: Fwd: Restricted sums in BoxedRep In-Reply-To: References: <03c327e0-edae-c4bb-aaef-66dc5f53cd9d@gmx.at> Message-ID: Ah yes. In my research work I extended Mutable Constructor fields to allow a mutable array field to help with that problem. This allowed me to have a Nil constructor in a sum type and a Node constructor with normal fields as well as an array of mutable fields. Pointer tagging worked as expected so a case match on a dereference of a field would not need to follow the pointer and no intermediate objects were between Node objects. On Thu, Oct 15, 2020 at 4:08 PM David Feuer wrote: > This might be lost in the noise for MVar and TVar, but for arrays, there > are tremendous advantages to cutting out the extra heap objects. > > On Thu, Oct 15, 2020, 1:10 PM Ryan Yates wrote: > >> I haven't been following this discussion too closely, but in my research >> work I found that some of the benefits that I wanted in this direction were >> already there with pointer tagging. >> >> On Thu, Oct 15, 2020 at 12:59 PM David Feuer >> wrote: >> >>> Yes, that's something quite different. We'd need a whole different heap >>> object type for such MVars and TVars. My approach covers the case where the >>> unboxed thing can only take on a few values, for some value of "a few" >>> which, depending on implementation, may or may not be very small. If the >>> nulls point to actual heap objects in pre-allocated pinned memory (say), >>> then up to 64 or so might be reasonable. If they point to "invalid" address >>> space, then the numbers could go up a good bit. >>> >>> On Thu, Oct 15, 2020, 12:50 PM Carter Schonwald < >>> carter.schonwald at gmail.com> wrote: >>> >>>> Indeed, I mean things that aren’t pointery, and could be represented by >>>> a tvar paired with a mutable byte array or mvar with mutable byte array, >>>> but which we’d want considered as a single heap object from the rts/gc >>>> perspective. >>>> >>>> On Thu, Oct 15, 2020 at 11:58 AM David Feuer >>>> wrote: >>>> >>>>> Sorry, unlifted, not unboxed... >>>>> >>>>> On Thu, Oct 15, 2020, 11:57 AM David Feuer >>>>> wrote: >>>>> >>>>>> Putting unboxed things in TVar, MVar, etc., is part of Andrew >>>>>> Martin's accepted BoxedRep proposal. >>>>>> >>>>>> On Thu, Oct 15, 2020, 11:44 AM Carter Schonwald < >>>>>> carter.schonwald at gmail.com> wrote: >>>>>> >>>>>>> A related idea that came up recently and is perhaps simpler ties >>>>>>> into this via the lens of having unboxed Mvars/tvars (even if it’s >>>>>>> restricted to just things we can embed in a word64#) >>>>>>> >>>>>>> This came up in >>>>>>> https://gitlab.haskell.org/ghc/ghc/-/issues/18798#note_307410, >>>>>>> where viktor had millions of independent mvars holding what’s essentially a >>>>>>> strict unit ()! >>>>>>> >>>>>>> The motivation in this later scenario is that in high concurrency >>>>>>> settings, the less trivial stuff the gc needs to trace under updates, the >>>>>>> better ghc scales. >>>>>>> >>>>>>> This may not be a use case david has in mind, but certainly seems >>>>>>> related. >>>>>>> >>>>>>> Phrased more succinctly: gc perf dominates large heap / many core >>>>>>> computation in Haskell via sensitivity to allocation volume / mutation >>>>>>> volume (to ensure generational hypothesis stays valid), and providing tools >>>>>>> to incrementally reduce the pressure with local changes would be good. >>>>>>> >>>>>>> So I’d propose / suggest that a baby step towards what david asks >>>>>>> would be for us to work out some manner of unboxed tvar/mvar ref machinery >>>>>>> that supports unboxed values. >>>>>>> >>>>>>> >>>>>>> >>>>>>> On Thu, Oct 15, 2020 at 5:32 AM Andreas Klebinger < >>>>>>> klebinger.andreas at gmx.at> wrote: >>>>>>> >>>>>>>> From a implementors perspective my main questions would be: >>>>>>>> >>>>>>>> * How big is the benefit in practice? How many use cases are there? >>>>>>>> * How bad are the costs? (Runtime overhead, rts complexity, ...) >>>>>>>> >>>>>>>> The details of how this would be exposed to a user would are >>>>>>>> important. >>>>>>>> But if the costs are too high for the drawbacks then it becomes a >>>>>>>> moot point. >>>>>>>> >>>>>>>> >>>>>>>> David Feuer schrieb am 14.10.2020 um 22:21: >>>>>>>> >>>>>>>> Forwarded from Andrew Martin below. I think we want more than just >>>>>>>> Maybe (more than one null), but the nesting I described is certainly more >>>>>>>> convenience than necessity. >>>>>>>> >>>>>>>> ---------- Forwarded message --------- >>>>>>>> From: Andrew Martin >>>>>>>> Date: Wed, Oct 14, 2020, 4:14 PM >>>>>>>> Subject: Re: Restricted sums in BoxedRep >>>>>>>> To: David Feuer >>>>>>>> >>>>>>>> >>>>>>>> You'll have to forward this to the ghc-devs list to share it with >>>>>>>> others since I'm not currently subscribed to it, but I've had this same >>>>>>>> thought before. It is discussed at >>>>>>>> https://github.com/andrewthad/impure-containers/issues/12. Here's >>>>>>>> the relevant excerpt: >>>>>>>> >>>>>>>>> Relatedly, I was thinking the other day that after finishing >>>>>>>>> implementing >>>>>>>>> https://github.com/ghc-proposals/ghc-proposals/blob/master/proposals/0203-pointer-rep.rst, >>>>>>>>> I should really look at seeing if it's possible to add this >>>>>>>>> maybe-of-a-lifted value trick straight to GHC. I think that with: >>>>>>>>> >>>>>>>>> data RuntimpRep >>>>>>>>> = BoxedRep Levity >>>>>>>>> | MaybeBoxedRep Levity >>>>>>>>> | IntRep >>>>>>>>> | ... >>>>>>>>> >>>>>>>>> data BuiltinMaybe :: forall (v :: Levity). TYPE v -> TYPE ('MaybeBoxedRep v) >>>>>>>>> >>>>>>>>> This doesn't have the nesting issues because the kind system >>>>>>>>> prevents nesting. But anyway, back to the original question. I would >>>>>>>>> recommend not using Maybe.Unsafe and using unpacked-maybe >>>>>>>>> instead. The latter is definitely safe, and it only costs an extra machine >>>>>>>>> word of space in each data constructor it gets used in, and it doesn't >>>>>>>>> introduce more indirections. >>>>>>>>> >>>>>>>> >>>>>>>> On Tue, Oct 13, 2020 at 5:47 PM David Feuer >>>>>>>> wrote: >>>>>>>> >>>>>>>>> Null pointers are widely known to be a lousy language feature in >>>>>>>>> general, but there are certain situations where they're *really* useful for >>>>>>>>> compact representation. For example, we define >>>>>>>>> >>>>>>>>> newtype TMVar a = TMVar (TVar (Maybe a)) >>>>>>>>> >>>>>>>>> We don't, however, actually use the fact that (Maybe a) is lifted. >>>>>>>>> So we could represent this much more efficiently using something like >>>>>>>>> >>>>>>>>> newtype TMVar a = TMVar (TVar a) >>>>>>>>> >>>>>>>>> where Nothing is represented by a distinguished "null" pointer. >>>>>>>>> >>>>>>>>> While it's possible to implement this sort of thing in user code >>>>>>>>> (with lots of fuss and care), it's not very nice at all. What I'd really >>>>>>>>> like to be able to do is represent certain kinds of sums like this natively. >>>>>>>>> >>>>>>>>> Now that we're getting BoxedRep, I think we can probably make it >>>>>>>>> happen. The trick is to add a special Levity constructor representing sums >>>>>>>>> of particular shapes. Specifically, we can represent a type like this if it >>>>>>>>> is a possibly-nested sum which, when flattened into a single sum, consists >>>>>>>>> of some number of nullary tuples and at most one Lifted or Unlifted type. >>>>>>>>> Then we can have (inline) primops to convert between the BoxedRep and the >>>>>>>>> sum-of-sums representations. >>>>>>>>> >>>>>>>>> Anyone have thoughts on details for what the Levity constructor >>>>>>>>> arguments might look like? >>>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> -- >>>>>>>> -Andrew Thaddeus Martin >>>>>>>> >>>>>>>> >>>>>>>> _______________________________________________ >>>>>>>> ghc-devs mailing listghc-devs at haskell.orghttp://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs >>>>>>>> >>>>>>>> >>>>>>>> _______________________________________________ >>>>>>>> ghc-devs mailing list >>>>>>>> ghc-devs at haskell.org >>>>>>>> http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs >>>>>>>> >>>>>>> _______________________________________________ >>> ghc-devs mailing list >>> ghc-devs at haskell.org >>> http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs >>> >> -------------- next part -------------- An HTML attachment was scrubbed... URL: From carter.schonwald at gmail.com Thu Oct 15 20:27:47 2020 From: carter.schonwald at gmail.com (Carter Schonwald) Date: Thu, 15 Oct 2020 16:27:47 -0400 Subject: Fwd: Restricted sums in BoxedRep In-Reply-To: References: <03c327e0-edae-c4bb-aaef-66dc5f53cd9d@gmx.at> Message-ID: Ryan: would an unboxed value Mvar, Eg to write StrictMvar (), avoid creating extra gc pressure / traversal a? On Thu, Oct 15, 2020 at 4:23 PM Ryan Yates wrote: > Ah yes. In my research work I extended Mutable Constructor fields to > allow a mutable array field to help with that problem. This allowed me to > have a Nil constructor in a sum type and a Node constructor with normal > fields as well as an array of mutable fields. Pointer tagging worked as > expected so a case match on a dereference of a field would not need to > follow the pointer and no intermediate objects were between Node objects. > > > > On Thu, Oct 15, 2020 at 4:08 PM David Feuer wrote: > >> This might be lost in the noise for MVar and TVar, but for arrays, there >> are tremendous advantages to cutting out the extra heap objects. >> >> On Thu, Oct 15, 2020, 1:10 PM Ryan Yates wrote: >> >>> I haven't been following this discussion too closely, but in my research >>> work I found that some of the benefits that I wanted in this direction were >>> already there with pointer tagging. >>> >>> On Thu, Oct 15, 2020 at 12:59 PM David Feuer >>> wrote: >>> >>>> Yes, that's something quite different. We'd need a whole different heap >>>> object type for such MVars and TVars. My approach covers the case where the >>>> unboxed thing can only take on a few values, for some value of "a few" >>>> which, depending on implementation, may or may not be very small. If the >>>> nulls point to actual heap objects in pre-allocated pinned memory (say), >>>> then up to 64 or so might be reasonable. If they point to "invalid" address >>>> space, then the numbers could go up a good bit. >>>> >>>> On Thu, Oct 15, 2020, 12:50 PM Carter Schonwald < >>>> carter.schonwald at gmail.com> wrote: >>>> >>>>> Indeed, I mean things that aren’t pointery, and could be represented >>>>> by a tvar paired with a mutable byte array or mvar with mutable byte array, >>>>> but which we’d want considered as a single heap object from the rts/gc >>>>> perspective. >>>>> >>>>> On Thu, Oct 15, 2020 at 11:58 AM David Feuer >>>>> wrote: >>>>> >>>>>> Sorry, unlifted, not unboxed... >>>>>> >>>>>> On Thu, Oct 15, 2020, 11:57 AM David Feuer >>>>>> wrote: >>>>>> >>>>>>> Putting unboxed things in TVar, MVar, etc., is part of Andrew >>>>>>> Martin's accepted BoxedRep proposal. >>>>>>> >>>>>>> On Thu, Oct 15, 2020, 11:44 AM Carter Schonwald < >>>>>>> carter.schonwald at gmail.com> wrote: >>>>>>> >>>>>>>> A related idea that came up recently and is perhaps simpler ties >>>>>>>> into this via the lens of having unboxed Mvars/tvars (even if it’s >>>>>>>> restricted to just things we can embed in a word64#) >>>>>>>> >>>>>>>> This came up in >>>>>>>> https://gitlab.haskell.org/ghc/ghc/-/issues/18798#note_307410, >>>>>>>> where viktor had millions of independent mvars holding what’s essentially a >>>>>>>> strict unit ()! >>>>>>>> >>>>>>>> The motivation in this later scenario is that in high concurrency >>>>>>>> settings, the less trivial stuff the gc needs to trace under updates, the >>>>>>>> better ghc scales. >>>>>>>> >>>>>>>> This may not be a use case david has in mind, but certainly seems >>>>>>>> related. >>>>>>>> >>>>>>>> Phrased more succinctly: gc perf dominates large heap / many core >>>>>>>> computation in Haskell via sensitivity to allocation volume / mutation >>>>>>>> volume (to ensure generational hypothesis stays valid), and providing tools >>>>>>>> to incrementally reduce the pressure with local changes would be good. >>>>>>>> >>>>>>>> So I’d propose / suggest that a baby step towards what david asks >>>>>>>> would be for us to work out some manner of unboxed tvar/mvar ref machinery >>>>>>>> that supports unboxed values. >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> On Thu, Oct 15, 2020 at 5:32 AM Andreas Klebinger < >>>>>>>> klebinger.andreas at gmx.at> wrote: >>>>>>>> >>>>>>>>> From a implementors perspective my main questions would be: >>>>>>>>> >>>>>>>>> * How big is the benefit in practice? How many use cases are there? >>>>>>>>> * How bad are the costs? (Runtime overhead, rts complexity, ...) >>>>>>>>> >>>>>>>>> The details of how this would be exposed to a user would are >>>>>>>>> important. >>>>>>>>> But if the costs are too high for the drawbacks then it becomes a >>>>>>>>> moot point. >>>>>>>>> >>>>>>>>> >>>>>>>>> David Feuer schrieb am 14.10.2020 um 22:21: >>>>>>>>> >>>>>>>>> Forwarded from Andrew Martin below. I think we want more than just >>>>>>>>> Maybe (more than one null), but the nesting I described is certainly more >>>>>>>>> convenience than necessity. >>>>>>>>> >>>>>>>>> ---------- Forwarded message --------- >>>>>>>>> From: Andrew Martin >>>>>>>>> Date: Wed, Oct 14, 2020, 4:14 PM >>>>>>>>> Subject: Re: Restricted sums in BoxedRep >>>>>>>>> To: David Feuer >>>>>>>>> >>>>>>>>> >>>>>>>>> You'll have to forward this to the ghc-devs list to share it with >>>>>>>>> others since I'm not currently subscribed to it, but I've had this same >>>>>>>>> thought before. It is discussed at >>>>>>>>> https://github.com/andrewthad/impure-containers/issues/12. Here's >>>>>>>>> the relevant excerpt: >>>>>>>>> >>>>>>>>>> Relatedly, I was thinking the other day that after finishing >>>>>>>>>> implementing >>>>>>>>>> https://github.com/ghc-proposals/ghc-proposals/blob/master/proposals/0203-pointer-rep.rst, >>>>>>>>>> I should really look at seeing if it's possible to add this >>>>>>>>>> maybe-of-a-lifted value trick straight to GHC. I think that with: >>>>>>>>>> >>>>>>>>>> data RuntimpRep >>>>>>>>>> = BoxedRep Levity >>>>>>>>>> | MaybeBoxedRep Levity >>>>>>>>>> | IntRep >>>>>>>>>> | ... >>>>>>>>>> >>>>>>>>>> data BuiltinMaybe :: forall (v :: Levity). TYPE v -> TYPE ('MaybeBoxedRep v) >>>>>>>>>> >>>>>>>>>> This doesn't have the nesting issues because the kind system >>>>>>>>>> prevents nesting. But anyway, back to the original question. I would >>>>>>>>>> recommend not using Maybe.Unsafe and using unpacked-maybe >>>>>>>>>> instead. The latter is definitely safe, and it only costs an extra machine >>>>>>>>>> word of space in each data constructor it gets used in, and it doesn't >>>>>>>>>> introduce more indirections. >>>>>>>>>> >>>>>>>>> >>>>>>>>> On Tue, Oct 13, 2020 at 5:47 PM David Feuer >>>>>>>>> wrote: >>>>>>>>> >>>>>>>>>> Null pointers are widely known to be a lousy language feature in >>>>>>>>>> general, but there are certain situations where they're *really* useful for >>>>>>>>>> compact representation. For example, we define >>>>>>>>>> >>>>>>>>>> newtype TMVar a = TMVar (TVar (Maybe a)) >>>>>>>>>> >>>>>>>>>> We don't, however, actually use the fact that (Maybe a) is >>>>>>>>>> lifted. So we could represent this much more efficiently using something >>>>>>>>>> like >>>>>>>>>> >>>>>>>>>> newtype TMVar a = TMVar (TVar a) >>>>>>>>>> >>>>>>>>>> where Nothing is represented by a distinguished "null" pointer. >>>>>>>>>> >>>>>>>>>> While it's possible to implement this sort of thing in user code >>>>>>>>>> (with lots of fuss and care), it's not very nice at all. What I'd really >>>>>>>>>> like to be able to do is represent certain kinds of sums like this natively. >>>>>>>>>> >>>>>>>>>> Now that we're getting BoxedRep, I think we can probably make it >>>>>>>>>> happen. The trick is to add a special Levity constructor representing sums >>>>>>>>>> of particular shapes. Specifically, we can represent a type like this if it >>>>>>>>>> is a possibly-nested sum which, when flattened into a single sum, consists >>>>>>>>>> of some number of nullary tuples and at most one Lifted or Unlifted type. >>>>>>>>>> Then we can have (inline) primops to convert between the BoxedRep and the >>>>>>>>>> sum-of-sums representations. >>>>>>>>>> >>>>>>>>>> Anyone have thoughts on details for what the Levity constructor >>>>>>>>>> arguments might look like? >>>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> -- >>>>>>>>> -Andrew Thaddeus Martin >>>>>>>>> >>>>>>>>> >>>>>>>>> _______________________________________________ >>>>>>>>> ghc-devs mailing listghc-devs at haskell.orghttp://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs >>>>>>>>> >>>>>>>>> >>>>>>>>> _______________________________________________ >>>>>>>>> ghc-devs mailing list >>>>>>>>> ghc-devs at haskell.org >>>>>>>>> http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs >>>>>>>>> >>>>>>>> _______________________________________________ >>>> ghc-devs mailing list >>>> ghc-devs at haskell.org >>>> http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs >>>> >>> -------------- next part -------------- An HTML attachment was scrubbed... URL: From fryguybob at gmail.com Thu Oct 15 23:54:16 2020 From: fryguybob at gmail.com (Ryan Yates) Date: Thu, 15 Oct 2020 19:54:16 -0400 Subject: Fwd: Restricted sums in BoxedRep In-Reply-To: References: <03c327e0-edae-c4bb-aaef-66dc5f53cd9d@gmx.at> Message-ID: I'm not sure what would lead you to have abundant enough unboxed MVar's to make a difference with GC. I'm happy to be wrong, but I think most situations where you want an MVar you want the indirection. If you don't want the indirection you probably don't want some feature of MVar and should use something else. On Thu, Oct 15, 2020 at 4:27 PM Carter Schonwald wrote: > Ryan: would an unboxed value Mvar, Eg to write StrictMvar (), avoid > creating extra gc pressure / traversal a? > > On Thu, Oct 15, 2020 at 4:23 PM Ryan Yates wrote: > >> Ah yes. In my research work I extended Mutable Constructor fields to >> allow a mutable array field to help with that problem. This allowed me to >> have a Nil constructor in a sum type and a Node constructor with normal >> fields as well as an array of mutable fields. Pointer tagging worked as >> expected so a case match on a dereference of a field would not need to >> follow the pointer and no intermediate objects were between Node objects. >> >> >> >> On Thu, Oct 15, 2020 at 4:08 PM David Feuer >> wrote: >> >>> This might be lost in the noise for MVar and TVar, but for arrays, there >>> are tremendous advantages to cutting out the extra heap objects. >>> >>> On Thu, Oct 15, 2020, 1:10 PM Ryan Yates wrote: >>> >>>> I haven't been following this discussion too closely, but in my >>>> research work I found that some of the benefits that I wanted in this >>>> direction were already there with pointer tagging. >>>> >>>> On Thu, Oct 15, 2020 at 12:59 PM David Feuer >>>> wrote: >>>> >>>>> Yes, that's something quite different. We'd need a whole different >>>>> heap object type for such MVars and TVars. My approach covers the case >>>>> where the unboxed thing can only take on a few values, for some value of "a >>>>> few" which, depending on implementation, may or may not be very small. If >>>>> the nulls point to actual heap objects in pre-allocated pinned memory >>>>> (say), then up to 64 or so might be reasonable. If they point to "invalid" >>>>> address space, then the numbers could go up a good bit. >>>>> >>>>> On Thu, Oct 15, 2020, 12:50 PM Carter Schonwald < >>>>> carter.schonwald at gmail.com> wrote: >>>>> >>>>>> Indeed, I mean things that aren’t pointery, and could be represented >>>>>> by a tvar paired with a mutable byte array or mvar with mutable byte array, >>>>>> but which we’d want considered as a single heap object from the rts/gc >>>>>> perspective. >>>>>> >>>>>> On Thu, Oct 15, 2020 at 11:58 AM David Feuer >>>>>> wrote: >>>>>> >>>>>>> Sorry, unlifted, not unboxed... >>>>>>> >>>>>>> On Thu, Oct 15, 2020, 11:57 AM David Feuer >>>>>>> wrote: >>>>>>> >>>>>>>> Putting unboxed things in TVar, MVar, etc., is part of Andrew >>>>>>>> Martin's accepted BoxedRep proposal. >>>>>>>> >>>>>>>> On Thu, Oct 15, 2020, 11:44 AM Carter Schonwald < >>>>>>>> carter.schonwald at gmail.com> wrote: >>>>>>>> >>>>>>>>> A related idea that came up recently and is perhaps simpler ties >>>>>>>>> into this via the lens of having unboxed Mvars/tvars (even if it’s >>>>>>>>> restricted to just things we can embed in a word64#) >>>>>>>>> >>>>>>>>> This came up in >>>>>>>>> https://gitlab.haskell.org/ghc/ghc/-/issues/18798#note_307410, >>>>>>>>> where viktor had millions of independent mvars holding what’s essentially a >>>>>>>>> strict unit ()! >>>>>>>>> >>>>>>>>> The motivation in this later scenario is that in high concurrency >>>>>>>>> settings, the less trivial stuff the gc needs to trace under updates, the >>>>>>>>> better ghc scales. >>>>>>>>> >>>>>>>>> This may not be a use case david has in mind, but certainly seems >>>>>>>>> related. >>>>>>>>> >>>>>>>>> Phrased more succinctly: gc perf dominates large heap / many core >>>>>>>>> computation in Haskell via sensitivity to allocation volume / mutation >>>>>>>>> volume (to ensure generational hypothesis stays valid), and providing tools >>>>>>>>> to incrementally reduce the pressure with local changes would be good. >>>>>>>>> >>>>>>>>> So I’d propose / suggest that a baby step towards what david asks >>>>>>>>> would be for us to work out some manner of unboxed tvar/mvar ref machinery >>>>>>>>> that supports unboxed values. >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> On Thu, Oct 15, 2020 at 5:32 AM Andreas Klebinger < >>>>>>>>> klebinger.andreas at gmx.at> wrote: >>>>>>>>> >>>>>>>>>> From a implementors perspective my main questions would be: >>>>>>>>>> >>>>>>>>>> * How big is the benefit in practice? How many use cases are >>>>>>>>>> there? >>>>>>>>>> * How bad are the costs? (Runtime overhead, rts complexity, ...) >>>>>>>>>> >>>>>>>>>> The details of how this would be exposed to a user would are >>>>>>>>>> important. >>>>>>>>>> But if the costs are too high for the drawbacks then it becomes a >>>>>>>>>> moot point. >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> David Feuer schrieb am 14.10.2020 um 22:21: >>>>>>>>>> >>>>>>>>>> Forwarded from Andrew Martin below. I think we want more than >>>>>>>>>> just Maybe (more than one null), but the nesting I described is certainly >>>>>>>>>> more convenience than necessity. >>>>>>>>>> >>>>>>>>>> ---------- Forwarded message --------- >>>>>>>>>> From: Andrew Martin >>>>>>>>>> Date: Wed, Oct 14, 2020, 4:14 PM >>>>>>>>>> Subject: Re: Restricted sums in BoxedRep >>>>>>>>>> To: David Feuer >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> You'll have to forward this to the ghc-devs list to share it with >>>>>>>>>> others since I'm not currently subscribed to it, but I've had this same >>>>>>>>>> thought before. It is discussed at >>>>>>>>>> https://github.com/andrewthad/impure-containers/issues/12. >>>>>>>>>> Here's the relevant excerpt: >>>>>>>>>> >>>>>>>>>>> Relatedly, I was thinking the other day that after finishing >>>>>>>>>>> implementing >>>>>>>>>>> https://github.com/ghc-proposals/ghc-proposals/blob/master/proposals/0203-pointer-rep.rst, >>>>>>>>>>> I should really look at seeing if it's possible to add this >>>>>>>>>>> maybe-of-a-lifted value trick straight to GHC. I think that with: >>>>>>>>>>> >>>>>>>>>>> data RuntimpRep >>>>>>>>>>> = BoxedRep Levity >>>>>>>>>>> | MaybeBoxedRep Levity >>>>>>>>>>> | IntRep >>>>>>>>>>> | ... >>>>>>>>>>> >>>>>>>>>>> data BuiltinMaybe :: forall (v :: Levity). TYPE v -> TYPE ('MaybeBoxedRep v) >>>>>>>>>>> >>>>>>>>>>> This doesn't have the nesting issues because the kind system >>>>>>>>>>> prevents nesting. But anyway, back to the original question. I would >>>>>>>>>>> recommend not using Maybe.Unsafe and using unpacked-maybe >>>>>>>>>>> instead. The latter is definitely safe, and it only costs an extra machine >>>>>>>>>>> word of space in each data constructor it gets used in, and it doesn't >>>>>>>>>>> introduce more indirections. >>>>>>>>>>> >>>>>>>>>> >>>>>>>>>> On Tue, Oct 13, 2020 at 5:47 PM David Feuer < >>>>>>>>>> david.feuer at gmail.com> wrote: >>>>>>>>>> >>>>>>>>>>> Null pointers are widely known to be a lousy language feature in >>>>>>>>>>> general, but there are certain situations where they're *really* useful for >>>>>>>>>>> compact representation. For example, we define >>>>>>>>>>> >>>>>>>>>>> newtype TMVar a = TMVar (TVar (Maybe a)) >>>>>>>>>>> >>>>>>>>>>> We don't, however, actually use the fact that (Maybe a) is >>>>>>>>>>> lifted. So we could represent this much more efficiently using something >>>>>>>>>>> like >>>>>>>>>>> >>>>>>>>>>> newtype TMVar a = TMVar (TVar a) >>>>>>>>>>> >>>>>>>>>>> where Nothing is represented by a distinguished "null" pointer. >>>>>>>>>>> >>>>>>>>>>> While it's possible to implement this sort of thing in user code >>>>>>>>>>> (with lots of fuss and care), it's not very nice at all. What I'd really >>>>>>>>>>> like to be able to do is represent certain kinds of sums like this natively. >>>>>>>>>>> >>>>>>>>>>> Now that we're getting BoxedRep, I think we can probably make it >>>>>>>>>>> happen. The trick is to add a special Levity constructor representing sums >>>>>>>>>>> of particular shapes. Specifically, we can represent a type like this if it >>>>>>>>>>> is a possibly-nested sum which, when flattened into a single sum, consists >>>>>>>>>>> of some number of nullary tuples and at most one Lifted or Unlifted type. >>>>>>>>>>> Then we can have (inline) primops to convert between the BoxedRep and the >>>>>>>>>>> sum-of-sums representations. >>>>>>>>>>> >>>>>>>>>>> Anyone have thoughts on details for what the Levity constructor >>>>>>>>>>> arguments might look like? >>>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> -- >>>>>>>>>> -Andrew Thaddeus Martin >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> _______________________________________________ >>>>>>>>>> ghc-devs mailing listghc-devs at haskell.orghttp://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> _______________________________________________ >>>>>>>>>> ghc-devs mailing list >>>>>>>>>> ghc-devs at haskell.org >>>>>>>>>> http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs >>>>>>>>>> >>>>>>>>> _______________________________________________ >>>>> ghc-devs mailing list >>>>> ghc-devs at haskell.org >>>>> http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs >>>>> >>>> -------------- next part -------------- An HTML attachment was scrubbed... URL: From moritz.angermann at gmail.com Tue Oct 20 01:50:41 2020 From: moritz.angermann at gmail.com (Moritz Angermann) Date: Tue, 20 Oct 2020 09:50:41 +0800 Subject: GHC's internal confusion about Ints and Words Message-ID: Hi there! So there is a procedure calling convention that for reasons I did not fully understand, but seem to be historically grown, uses packed arguments for those that are spilled onto the stack. On top of that, CInt is 32bit, Word is 64bits. This provides the following spectacle: While we know in STG that the CInt is 32bits wide, when lowered into Cmm, it's represented as I64 in the arguments to the C function. Thus packing based on the format of the Cmm type would yield 8 bytes. And now, all further packed arguments have the wrong offset (by four). Specifically in GHC.Cmm.Utils we find: primRepCmmType :: Platform -> PrimRep -> CmmType primRepCmmType platform IntRep = bWord platform mkIntCLit :: Platform -> Int -> CmmLit mkIntCLit platform i = CmmInt (toInteger i) (wordWidth platform) The naive idea to just fix this and make them return cIntWidth instead, seemingly produces the correct Cmm expressions at a local level, but produces a broken compiler. A second approach could be to extend the Hints into providing sizes, and using those during the foreign call generation to pack spilled arguments. This however appears to be more of a patching up of some fundamental underlying issue, instead of rectifying it properly. Maybe I'll have to go down the Hint path, it does however break current Eq assumptions, as they are sized now, and what was equal before, is only equal now if they represent the same size. >From a cursory glance at the issues with naively fixing the width for Int, it seems that GHC internally assumes sizeof(Int) = sizeof(Word). Maybe there is a whole level of HsInt vs CInt discrimination missing? Cheers, Moritz -------------- next part -------------- An HTML attachment was scrubbed... URL: From simonpj at microsoft.com Tue Oct 20 07:33:23 2020 From: simonpj at microsoft.com (Simon Peyton Jones) Date: Tue, 20 Oct 2020 07:33:23 +0000 Subject: GHC's internal confusion about Ints and Words In-Reply-To: References: Message-ID: Moritz I’m afraid I don’t understand any of this. Not your fault, but I just don’t have enough context to know what you mean. Is there a current bug? If so, can you demonstrate it? If not, what is the problem you want to solve? Examples are always helpful. Maybe it’s worth opening a ticket too? Thanks! Simon From: ghc-devs On Behalf Of Moritz Angermann Sent: 20 October 2020 02:51 To: ghc-devs Subject: GHC's internal confusion about Ints and Words Hi there! So there is a procedure calling convention that for reasons I did not fully understand, but seem to be historically grown, uses packed arguments for those that are spilled onto the stack. On top of that, CInt is 32bit, Word is 64bits. This provides the following spectacle: While we know in STG that the CInt is 32bits wide, when lowered into Cmm, it's represented as I64 in the arguments to the C function. Thus packing based on the format of the Cmm type would yield 8 bytes. And now, all further packed arguments have the wrong offset (by four). Specifically in GHC.Cmm.Utils we find: primRepCmmType :: Platform -> PrimRep -> CmmType primRepCmmType platform IntRep = bWord platform mkIntCLit :: Platform -> Int -> CmmLit mkIntCLit platform i = CmmInt (toInteger i) (wordWidth platform) The naive idea to just fix this and make them return cIntWidth instead, seemingly produces the correct Cmm expressions at a local level, but produces a broken compiler. A second approach could be to extend the Hints into providing sizes, and using those during the foreign call generation to pack spilled arguments. This however appears to be more of a patching up of some fundamental underlying issue, instead of rectifying it properly. Maybe I'll have to go down the Hint path, it does however break current Eq assumptions, as they are sized now, and what was equal before, is only equal now if they represent the same size. From a cursory glance at the issues with naively fixing the width for Int, it seems that GHC internally assumes sizeof(Int) = sizeof(Word). Maybe there is a whole level of HsInt vs CInt discrimination missing? Cheers, Moritz -------------- next part -------------- An HTML attachment was scrubbed... URL: From moritz.angermann at gmail.com Tue Oct 20 09:48:22 2020 From: moritz.angermann at gmail.com (Moritz Angermann) Date: Tue, 20 Oct 2020 17:48:22 +0800 Subject: GHC's internal confusion about Ints and Words In-Reply-To: References: Message-ID: Alright, let me expand a bit. I've been looking at aarch64 NCG for ghc. The Linux side of things is looking really good, so I've moved onto the macOS side (I'm afraid I don't have any Windows aarch64 hardware, nor much windows knowledge to even attempt a Windows version yet). When calling C functions, the usual approach is to pass the first few arguments in registers, and then arguments that exceed the argument passing slots on the stack. The Arm AArch64 Procedure Call Standard (aapcs) for C does this by assigning 8byte slots to each overflow argument on the stack. A company I won't name, has decided to implement a slightly different variation of the Procedure Call Standard, often referred to as darwinpcs. This deviates from the aapcs for vargs, as well as for handling of spilled arguments on the stack. The aapcs allows us to generate calls to C functions without knowing the actual prototype of the function, as all arguments are simply spilled into 8byte slots on the stack. The darwinpcs however requires us to know the size of the arguments, so we can properly pack them onto the stack. Ints have 4 bytes, so we need to pack them into 4byte slots. In the process library we have this rather fun foreign import: foreign import ccall unsafe "runInteractiveProcess" c_runInteractiveProcess :: Ptr CString -> CString -> Ptr CString -> FD -> FD -> FD -> Ptr FD -> Ptr FD -> Ptr FD -> Ptr CGid -> Ptr CUid -> CInt -- reset child's SIGINT & SIGQUIT handlers -> CInt -- flags -> Ptr CString -> IO PHANDLE with the corresponding C declaration: extern ProcHandle runInteractiveProcess( char *const args[], char *workingDirectory, char **environment, int fdStdIn, int fdStdOut, int fdStdErr, int *pfdStdInput, int *pfdStdOutput, int *pfdStdError, gid_t *childGroup, uid_t *childUser, int reset_int_quit_handlers, int flags, char **failed_doing); This function thus takes 14 arguments. We pass only the first 8 arguments in registers, and the others on the stack. Argument 12 and 13 are of type int. On linux using the aapcs, we can pass those in 8byte slots on the stack. That is both of them are effectively 64bits wide when passed. However for darwinpcs, it is expected that these adhere to their size and are packed as such. Therefore Argument 12 and 13 need to be passed as 4byte slots each on the stack. This yields a moderate 8byte saving on the stack for the same function call on darwinpcs compared to aapcs. Now onto GHC. When we generate function calls for foreign C functions, we deal with something like: genCCall :: ForeignTarget -- function to call -> [CmmFormal] -- where to put the result -> [CmmActual] -- arguments (of mixed type) -> BlockId -- The block we are in -> NatM (InstrBlock, Maybe BlockId) based on Cmm Nodes of the form CmmUnsafeForeignCall target result_regs args The CmmActual in the care of runInteractiveProcess hold the arguments for the function, however contrary to the function declaration, it contains I64 slots for Argument 12 and 13. Thus computing the space needed for them based on their Cmm Representations yields 8bytes, when they should really be 32bit and consume only 4 byte. To illustrate this a bit better: here is what we see in the pretty printed cmm: (_s6w3::I64) = call "ccall" arg hints: [PtrHint, PtrHint, PtrHint, signed, signed, signed, PtrHint, PtrHint, PtrHint, PtrHint, PtrHint, signed, signed, PtrHint] result hints: [signed] _runInteractiveProcess(I64[Sp + 96], I64[Sp + 88], I64[Sp + 104], I64[Sp + 112], I64[Sp + 120], I64[Sp + 56], I64[Sp + 64], I64[Sp + 72], I64[Sp + 24], 0, 0, I64[Sp + 8], I64[Sp + 40] | I64[Sp + 48] | I64[Sp + 80] | 3, I64[R1 + 7]); I've added size information to the ForeignHints (NoHint, AddrHint, SignedHint) we have, and computed both, which yields: [(CmmReg (CmmLocal (LocalReg s6Gi (CmmType BitsCat W64))),AddrHint) ,(CmmReg (CmmLocal (LocalReg s6Gk (CmmType BitsCat W64))),AddrHint) ,(CmmReg (CmmLocal (LocalReg s6Gm (CmmType BitsCat W64))),AddrHint) ,(CmmReg (CmmLocal (LocalReg s6Go (CmmType BitsCat W64))),SignedHint W32) ,(CmmReg (CmmLocal (LocalReg s6Gq (CmmType BitsCat W64))),SignedHint W32) ,(CmmReg (CmmLocal (LocalReg s6Gs (CmmType BitsCat W64))),SignedHint W32) ,(CmmReg (CmmLocal (LocalReg s6Gu (CmmType BitsCat W64))),AddrHint) ,(CmmReg (CmmLocal (LocalReg s6Gw (CmmType BitsCat W64))),AddrHint) ,(CmmReg (CmmLocal (LocalReg s6Gy (CmmType BitsCat W64))),AddrHint) ,(CmmReg (CmmLocal (LocalReg s6Cp (CmmType BitsCat W64))),AddrHint) ,(CmmReg (CmmLocal (LocalReg s6FU (CmmType BitsCat W64))),AddrHint) ,(CmmReg (CmmLocal (LocalReg s6GA (CmmType BitsCat W64))),SignedHint W32) ,(CmmReg (CmmLocal (LocalReg s6GR (CmmType BitsCat W64))),SignedHint W32) ,(CmmReg (CmmLocal (LocalReg s6GM (CmmType BitsCat W64))),AddrHint)] Thus, while we *do* know the right size from STG (which is what the Hints are computed from), we loose this information when lowering into Cmm, where we represent them with W64. This is what I was alluding to in the previous email. In primRepCmmType, and mkIntCLit, we set their type to 64bit for Ints; which on this platform does not hold. Now I've gone ahead and effectively assume Cmm is lying to me when generating Foreign Function Calls, and rely on the (new) sized hints to produce the appropriate argument packing on the stack. However I believe the correct way would be for GHC not to conflate Ints and Words in Cmm; implicitly assuming they are the same width. Sadly it's not as simple as having primRepCmmType and mkIntCLit produce 32bit types. I fear GHC internally assumes "Int" means 64bit Integer, and then just happens to make the Int ~ CInt assumption. Cheers, Moritz On Tue, Oct 20, 2020 at 3:33 PM Simon Peyton Jones wrote: > Moritz > > > > I’m afraid I don’t understand any of this. Not your fault, but I just > don’t have enough context to know what you mean. > > > > Is there a current bug? If so, can you demonstrate it? If not, what is > the problem you want to solve? Examples are always helpful. > > > > Maybe it’s worth opening a ticket too? > > > > Thanks! > > > > Simon > > > > *From:* ghc-devs *On Behalf Of *Moritz > Angermann > *Sent:* 20 October 2020 02:51 > *To:* ghc-devs > *Subject:* GHC's internal confusion about Ints and Words > > > > Hi there! > > > > So there is a procedure calling convention that for reasons I did not > fully understand, but seem to be historically grown, uses packed arguments > for those that are spilled onto the stack. On top of that, CInt is 32bit, > Word is 64bits. This provides the following spectacle: > > > > While we know in STG that the CInt is 32bits wide, when lowered into Cmm, > it's represented as I64 in the arguments to the C function. Thus packing > based on the format of the Cmm type would yield 8 bytes. And now, all > further packed arguments have the wrong offset (by four). > > > > Specifically in GHC.Cmm.Utils we find: > > primRepCmmType :: Platform -> PrimRep -> CmmType > > primRepCmmType platform IntRep = bWord platform > > > > mkIntCLit :: Platform -> Int -> CmmLit > mkIntCLit platform i = CmmInt (toInteger i) (wordWidth platform) > > > > The naive idea to just fix this and make them return cIntWidth instead, > seemingly produces the correct Cmm expressions at a local level, but > produces a broken compiler. > > > > A second approach could be to extend the Hints into providing sizes, and > using those during the foreign call generation to pack spilled arguments. > This however appears to be more of a patching up of some fundamental > underlying issue, instead of rectifying it properly. > > > > Maybe I'll have to go down the Hint path, it does however break current Eq > assumptions, as they are sized now, and what was equal before, is only > equal now if they represent the same size. > > > > From a cursory glance at the issues with naively fixing the width for Int, > it seems that GHC internally assumes sizeof(Int) = sizeof(Word). Maybe > there is a whole level of HsInt vs CInt discrimination missing? > > > > Cheers, > > Moritz > -------------- next part -------------- An HTML attachment was scrubbed... URL: From carter.schonwald at gmail.com Tue Oct 20 13:57:26 2020 From: carter.schonwald at gmail.com (Carter Schonwald) Date: Tue, 20 Oct 2020 09:57:26 -0400 Subject: GHC's internal confusion about Ints and Words In-Reply-To: References: Message-ID: ... are you talking about Haskell Int and word? Those are always the same size in bits and should match native point size. That is definitely an assumption of ghc It sounds like some information that is dropped after core is needed to correctly do something in stg/cmm in the context of the ARM64 ncg that was recently added to handle cint being 32bit in this context ? On Tue, Oct 20, 2020 at 5:49 AM Moritz Angermann wrote: > Alright, let me expand a bit. I've been looking at aarch64 NCG for ghc. > The Linux side of things is looking really good, > so I've moved onto the macOS side (I'm afraid I don't have any Windows > aarch64 hardware, nor much windows knowledge > to even attempt a Windows version yet). > > When calling C functions, the usual approach is to pass the first few > arguments in registers, and then arguments that exceed > the argument passing slots on the stack. The Arm AArch64 Procedure Call > Standard (aapcs) for C does this by assigning 8byte > slots to each overflow argument on the stack. A company I won't name, has > decided to implement a slightly different variation of > the Procedure Call Standard, often referred to as darwinpcs. This > deviates from the aapcs for vargs, as well as for handling of > spilled arguments on the stack. > > The aapcs allows us to generate calls to C functions without knowing the > actual prototype of the function, as all arguments are > simply spilled into 8byte slots on the stack. The darwinpcs however > requires us to know the size of the arguments, so we can > properly pack them onto the stack. Ints have 4 bytes, so we need to pack > them into 4byte slots. > > In the process library we have this rather fun foreign import: > foreign import ccall unsafe "runInteractiveProcess" > c_runInteractiveProcess > :: Ptr CString > -> CString > -> Ptr CString > -> FD > -> FD > -> FD > -> Ptr FD > -> Ptr FD > -> Ptr FD > -> Ptr CGid > -> Ptr CUid > -> CInt -- reset child's SIGINT & SIGQUIT > handlers > -> CInt -- flags > -> Ptr CString > -> IO PHANDLE > > with the corresponding C declaration: > > extern ProcHandle runInteractiveProcess( char *const args[], > char *workingDirectory, > char **environment, > int fdStdIn, > int fdStdOut, > int fdStdErr, > int *pfdStdInput, > int *pfdStdOutput, > int *pfdStdError, > gid_t *childGroup, > uid_t *childUser, > int reset_int_quit_handlers, > int flags, > char **failed_doing); > This function thus takes 14 arguments. We pass only the first 8 arguments > in registers, and the others on the stack. > Argument 12 and 13 are of type int. On linux using the aapcs, we can pass > those in 8byte slots on the stack. That is > both of them are effectively 64bits wide when passed. However for > darwinpcs, it is expected that these adhere to their > size and are packed as such. Therefore Argument 12 and 13 need to be > passed as 4byte slots each on the stack. > > This yields a moderate 8byte saving on the stack for the same function > call on darwinpcs compared to aapcs. > > Now onto GHC. When we generate function calls for foreign C functions, we > deal with something like: > > genCCall > :: ForeignTarget -- function to call > -> [CmmFormal] -- where to put the result > -> [CmmActual] -- arguments (of mixed type) > -> BlockId -- The block we are in > -> NatM (InstrBlock, Maybe BlockId) > > based on Cmm Nodes of the form CmmUnsafeForeignCall target result_regs > args > > The CmmActual in the care of runInteractiveProcess hold the arguments for > the function, however contrary to the function > declaration, it contains I64 slots for Argument 12 and 13. Thus computing > the space needed for them based on their Cmm > Representations yields 8bytes, when they should really be 32bit and > consume only 4 byte. > > To illustrate this a bit better: here is what we see in the pretty printed > cmm: > > (_s6w3::I64) = call "ccall" arg hints: [PtrHint, PtrHint, PtrHint, > signed, signed, signed, PtrHint, PtrHint, PtrHint, PtrHint, PtrHint, > signed, signed, PtrHint] result hints: [signed] > _runInteractiveProcess(I64[Sp + 96], I64[Sp + 88], I64[Sp + 104], I64[Sp + > 112], I64[Sp + 120], I64[Sp + 56], I64[Sp + 64], I64[Sp + 72], I64[Sp + > 24], 0, 0, I64[Sp + 8], I64[Sp + 40] | I64[Sp + 48] | I64[Sp + 80] | 3, > I64[R1 + 7]); > > I've added size information to the ForeignHints (NoHint, AddrHint, > SignedHint) we have, and computed both, which yields: > > [(CmmReg (CmmLocal (LocalReg s6Gi (CmmType BitsCat W64))),AddrHint) > ,(CmmReg (CmmLocal (LocalReg s6Gk (CmmType BitsCat W64))),AddrHint) > ,(CmmReg (CmmLocal (LocalReg s6Gm (CmmType BitsCat W64))),AddrHint) > ,(CmmReg (CmmLocal (LocalReg s6Go (CmmType BitsCat W64))),SignedHint W32) > ,(CmmReg (CmmLocal (LocalReg s6Gq (CmmType BitsCat W64))),SignedHint W32) > ,(CmmReg (CmmLocal (LocalReg s6Gs (CmmType BitsCat W64))),SignedHint W32) > ,(CmmReg (CmmLocal (LocalReg s6Gu (CmmType BitsCat W64))),AddrHint) > ,(CmmReg (CmmLocal (LocalReg s6Gw (CmmType BitsCat W64))),AddrHint) > ,(CmmReg (CmmLocal (LocalReg s6Gy (CmmType BitsCat W64))),AddrHint) > ,(CmmReg (CmmLocal (LocalReg s6Cp (CmmType BitsCat W64))),AddrHint) > ,(CmmReg (CmmLocal (LocalReg s6FU (CmmType BitsCat W64))),AddrHint) > ,(CmmReg (CmmLocal (LocalReg s6GA (CmmType BitsCat W64))),SignedHint W32) > ,(CmmReg (CmmLocal (LocalReg s6GR (CmmType BitsCat W64))),SignedHint W32) > ,(CmmReg (CmmLocal (LocalReg s6GM (CmmType BitsCat W64))),AddrHint)] > > Thus, while we *do* know the right size from STG (which is what the Hints > are computed from), we loose this information when lowering > into Cmm, where we represent them with W64. This is what I was alluding to > in the previous email. In primRepCmmType, and mkIntCLit, we set their type > to 64bit for Ints; which on this platform does not hold. > > Now I've gone ahead and effectively assume Cmm is lying to me when > generating Foreign Function Calls, and rely on the (new) sized > hints to produce the appropriate argument packing on the stack. However I > believe the correct way would be for GHC not to conflate Ints > and Words in Cmm; implicitly assuming they are the same width. Sadly it's > not as simple as having primRepCmmType and mkIntCLit produce 32bit types. I > fear GHC internally assumes "Int" means 64bit Integer, and then just > happens to make the Int ~ CInt assumption. > > Cheers, > Moritz > > On Tue, Oct 20, 2020 at 3:33 PM Simon Peyton Jones > wrote: > >> Moritz >> >> >> >> I’m afraid I don’t understand any of this. Not your fault, but I just >> don’t have enough context to know what you mean. >> >> >> >> Is there a current bug? If so, can you demonstrate it? If not, what is >> the problem you want to solve? Examples are always helpful. >> >> >> >> Maybe it’s worth opening a ticket too? >> >> >> >> Thanks! >> >> >> >> Simon >> >> >> >> *From:* ghc-devs *On Behalf Of *Moritz >> Angermann >> *Sent:* 20 October 2020 02:51 >> *To:* ghc-devs >> *Subject:* GHC's internal confusion about Ints and Words >> >> >> >> Hi there! >> >> >> >> So there is a procedure calling convention that for reasons I did not >> fully understand, but seem to be historically grown, uses packed arguments >> for those that are spilled onto the stack. On top of that, CInt is 32bit, >> Word is 64bits. This provides the following spectacle: >> >> >> >> While we know in STG that the CInt is 32bits wide, when lowered into Cmm, >> it's represented as I64 in the arguments to the C function. Thus packing >> based on the format of the Cmm type would yield 8 bytes. And now, all >> further packed arguments have the wrong offset (by four). >> >> >> >> Specifically in GHC.Cmm.Utils we find: >> >> primRepCmmType :: Platform -> PrimRep -> CmmType >> >> primRepCmmType platform IntRep = bWord platform >> >> >> >> mkIntCLit :: Platform -> Int -> CmmLit >> mkIntCLit platform i = CmmInt (toInteger i) (wordWidth platform) >> >> >> >> The naive idea to just fix this and make them return cIntWidth instead, >> seemingly produces the correct Cmm expressions at a local level, but >> produces a broken compiler. >> >> >> >> A second approach could be to extend the Hints into providing sizes, and >> using those during the foreign call generation to pack spilled arguments. >> This however appears to be more of a patching up of some fundamental >> underlying issue, instead of rectifying it properly. >> >> >> >> Maybe I'll have to go down the Hint path, it does however break current >> Eq assumptions, as they are sized now, and what was equal before, is only >> equal now if they represent the same size. >> >> >> >> From a cursory glance at the issues with naively fixing the width for >> Int, it seems that GHC internally assumes sizeof(Int) = sizeof(Word). >> Maybe there is a whole level of HsInt vs CInt discrimination missing? >> >> >> >> Cheers, >> >> Moritz >> > _______________________________________________ > ghc-devs mailing list > ghc-devs at haskell.org > http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs > -------------- next part -------------- An HTML attachment was scrubbed... URL: From moritz.angermann at gmail.com Tue Oct 20 14:02:55 2020 From: moritz.angermann at gmail.com (Moritz Angermann) Date: Tue, 20 Oct 2020 22:02:55 +0800 Subject: GHC's internal confusion about Ints and Words In-Reply-To: References: Message-ID: Yes, that's right. I'm not sure it's in core though, as the width information still seems to be available in Stg. However the lowering from stg into cmm widens it. On Tue, Oct 20, 2020 at 9:57 PM Carter Schonwald wrote: > ... are you talking about Haskell Int and word? Those are always the same > size in bits and should match native point size. That is definitely an > assumption of ghc > > It sounds like some information that is dropped after core is needed to > correctly do something in stg/cmm in the context of the ARM64 ncg that was > recently added to handle cint being 32bit in this context ? > > > On Tue, Oct 20, 2020 at 5:49 AM Moritz Angermann < > moritz.angermann at gmail.com> wrote: > >> Alright, let me expand a bit. I've been looking at aarch64 NCG for ghc. >> The Linux side of things is looking really good, >> so I've moved onto the macOS side (I'm afraid I don't have any Windows >> aarch64 hardware, nor much windows knowledge >> to even attempt a Windows version yet). >> >> When calling C functions, the usual approach is to pass the first few >> arguments in registers, and then arguments that exceed >> the argument passing slots on the stack. The Arm AArch64 Procedure Call >> Standard (aapcs) for C does this by assigning 8byte >> slots to each overflow argument on the stack. A company I won't name, >> has decided to implement a slightly different variation of >> the Procedure Call Standard, often referred to as darwinpcs. This >> deviates from the aapcs for vargs, as well as for handling of >> spilled arguments on the stack. >> >> The aapcs allows us to generate calls to C functions without knowing the >> actual prototype of the function, as all arguments are >> simply spilled into 8byte slots on the stack. The darwinpcs however >> requires us to know the size of the arguments, so we can >> properly pack them onto the stack. Ints have 4 bytes, so we need to pack >> them into 4byte slots. >> >> In the process library we have this rather fun foreign import: >> foreign import ccall unsafe "runInteractiveProcess" >> c_runInteractiveProcess >> :: Ptr CString >> -> CString >> -> Ptr CString >> -> FD >> -> FD >> -> FD >> -> Ptr FD >> -> Ptr FD >> -> Ptr FD >> -> Ptr CGid >> -> Ptr CUid >> -> CInt -- reset child's SIGINT & SIGQUIT >> handlers >> -> CInt -- flags >> -> Ptr CString >> -> IO PHANDLE >> >> with the corresponding C declaration: >> >> extern ProcHandle runInteractiveProcess( char *const args[], >> char *workingDirectory, >> char **environment, >> int fdStdIn, >> int fdStdOut, >> int fdStdErr, >> int *pfdStdInput, >> int *pfdStdOutput, >> int *pfdStdError, >> gid_t *childGroup, >> uid_t *childUser, >> int reset_int_quit_handlers, >> int flags, >> char **failed_doing); >> This function thus takes 14 arguments. We pass only the first 8 arguments >> in registers, and the others on the stack. >> Argument 12 and 13 are of type int. On linux using the aapcs, we can >> pass those in 8byte slots on the stack. That is >> both of them are effectively 64bits wide when passed. However for >> darwinpcs, it is expected that these adhere to their >> size and are packed as such. Therefore Argument 12 and 13 need to be >> passed as 4byte slots each on the stack. >> >> This yields a moderate 8byte saving on the stack for the same function >> call on darwinpcs compared to aapcs. >> >> Now onto GHC. When we generate function calls for foreign C functions, >> we deal with something like: >> >> genCCall >> :: ForeignTarget -- function to call >> -> [CmmFormal] -- where to put the result >> -> [CmmActual] -- arguments (of mixed type) >> -> BlockId -- The block we are in >> -> NatM (InstrBlock, Maybe BlockId) >> >> based on Cmm Nodes of the form CmmUnsafeForeignCall target result_regs >> args >> >> The CmmActual in the care of runInteractiveProcess hold the arguments >> for the function, however contrary to the function >> declaration, it contains I64 slots for Argument 12 and 13. Thus computing >> the space needed for them based on their Cmm >> Representations yields 8bytes, when they should really be 32bit and >> consume only 4 byte. >> >> To illustrate this a bit better: here is what we see in the pretty >> printed cmm: >> >> (_s6w3::I64) = call "ccall" arg hints: [PtrHint, PtrHint, PtrHint, >> signed, signed, signed, PtrHint, PtrHint, PtrHint, PtrHint, PtrHint, >> signed, signed, PtrHint] result hints: [signed] >> _runInteractiveProcess(I64[Sp + 96], I64[Sp + 88], I64[Sp + 104], I64[Sp + >> 112], I64[Sp + 120], I64[Sp + 56], I64[Sp + 64], I64[Sp + 72], I64[Sp + >> 24], 0, 0, I64[Sp + 8], I64[Sp + 40] | I64[Sp + 48] | I64[Sp + 80] | 3, >> I64[R1 + 7]); >> >> I've added size information to the ForeignHints (NoHint, AddrHint, >> SignedHint) we have, and computed both, which yields: >> >> [(CmmReg (CmmLocal (LocalReg s6Gi (CmmType BitsCat W64))),AddrHint) >> ,(CmmReg (CmmLocal (LocalReg s6Gk (CmmType BitsCat W64))),AddrHint) >> ,(CmmReg (CmmLocal (LocalReg s6Gm (CmmType BitsCat W64))),AddrHint) >> ,(CmmReg (CmmLocal (LocalReg s6Go (CmmType BitsCat W64))),SignedHint W32) >> ,(CmmReg (CmmLocal (LocalReg s6Gq (CmmType BitsCat W64))),SignedHint W32) >> ,(CmmReg (CmmLocal (LocalReg s6Gs (CmmType BitsCat W64))),SignedHint W32) >> ,(CmmReg (CmmLocal (LocalReg s6Gu (CmmType BitsCat W64))),AddrHint) >> ,(CmmReg (CmmLocal (LocalReg s6Gw (CmmType BitsCat W64))),AddrHint) >> ,(CmmReg (CmmLocal (LocalReg s6Gy (CmmType BitsCat W64))),AddrHint) >> ,(CmmReg (CmmLocal (LocalReg s6Cp (CmmType BitsCat W64))),AddrHint) >> ,(CmmReg (CmmLocal (LocalReg s6FU (CmmType BitsCat W64))),AddrHint) >> ,(CmmReg (CmmLocal (LocalReg s6GA (CmmType BitsCat W64))),SignedHint W32) >> ,(CmmReg (CmmLocal (LocalReg s6GR (CmmType BitsCat W64))),SignedHint W32) >> ,(CmmReg (CmmLocal (LocalReg s6GM (CmmType BitsCat W64))),AddrHint)] >> >> Thus, while we *do* know the right size from STG (which is what the Hints >> are computed from), we loose this information when lowering >> into Cmm, where we represent them with W64. This is what I was alluding >> to in the previous email. In primRepCmmType, and mkIntCLit, we set their >> type to 64bit for Ints; which on this platform does not hold. >> >> Now I've gone ahead and effectively assume Cmm is lying to me when >> generating Foreign Function Calls, and rely on the (new) sized >> hints to produce the appropriate argument packing on the stack. However >> I believe the correct way would be for GHC not to conflate Ints >> and Words in Cmm; implicitly assuming they are the same width. Sadly >> it's not as simple as having primRepCmmType and mkIntCLit produce 32bit >> types. I fear GHC internally assumes "Int" means 64bit Integer, and then >> just happens to make the Int ~ CInt assumption. >> >> Cheers, >> Moritz >> >> On Tue, Oct 20, 2020 at 3:33 PM Simon Peyton Jones >> wrote: >> >>> Moritz >>> >>> >>> >>> I’m afraid I don’t understand any of this. Not your fault, but I just >>> don’t have enough context to know what you mean. >>> >>> >>> >>> Is there a current bug? If so, can you demonstrate it? If not, what >>> is the problem you want to solve? Examples are always helpful. >>> >>> >>> >>> Maybe it’s worth opening a ticket too? >>> >>> >>> >>> Thanks! >>> >>> >>> >>> Simon >>> >>> >>> >>> *From:* ghc-devs *On Behalf Of *Moritz >>> Angermann >>> *Sent:* 20 October 2020 02:51 >>> *To:* ghc-devs >>> *Subject:* GHC's internal confusion about Ints and Words >>> >>> >>> >>> Hi there! >>> >>> >>> >>> So there is a procedure calling convention that for reasons I did not >>> fully understand, but seem to be historically grown, uses packed arguments >>> for those that are spilled onto the stack. On top of that, CInt is 32bit, >>> Word is 64bits. This provides the following spectacle: >>> >>> >>> >>> While we know in STG that the CInt is 32bits wide, when lowered into >>> Cmm, it's represented as I64 in the arguments to the C function. Thus >>> packing based on the format of the Cmm type would yield 8 bytes. And now, >>> all further packed arguments have the wrong offset (by four). >>> >>> >>> >>> Specifically in GHC.Cmm.Utils we find: >>> >>> primRepCmmType :: Platform -> PrimRep -> CmmType >>> >>> primRepCmmType platform IntRep = bWord platform >>> >>> >>> >>> mkIntCLit :: Platform -> Int -> CmmLit >>> mkIntCLit platform i = CmmInt (toInteger i) (wordWidth platform) >>> >>> >>> >>> The naive idea to just fix this and make them return cIntWidth instead, >>> seemingly produces the correct Cmm expressions at a local level, but >>> produces a broken compiler. >>> >>> >>> >>> A second approach could be to extend the Hints into providing sizes, and >>> using those during the foreign call generation to pack spilled arguments. >>> This however appears to be more of a patching up of some fundamental >>> underlying issue, instead of rectifying it properly. >>> >>> >>> >>> Maybe I'll have to go down the Hint path, it does however break current >>> Eq assumptions, as they are sized now, and what was equal before, is only >>> equal now if they represent the same size. >>> >>> >>> >>> From a cursory glance at the issues with naively fixing the width for >>> Int, it seems that GHC internally assumes sizeof(Int) = sizeof(Word). >>> Maybe there is a whole level of HsInt vs CInt discrimination missing? >>> >>> >>> >>> Cheers, >>> >>> Moritz >>> >> _______________________________________________ >> ghc-devs mailing list >> ghc-devs at haskell.org >> http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From simonpj at microsoft.com Tue Oct 20 14:29:20 2020 From: simonpj at microsoft.com (Simon Peyton Jones) Date: Tue, 20 Oct 2020 14:29:20 +0000 Subject: GHC's internal confusion about Ints and Words In-Reply-To: References: Message-ID: Now I've gone ahead and effectively assume Cmm is lying to me when generating Foreign Function Calls, and rely on the (new) sized hints to produce the appropriate argument packing on the stack Why not instead just stop Cmm lying? S From: Moritz Angermann Sent: 20 October 2020 15:03 To: Carter Schonwald Cc: Simon Peyton Jones ; ghc-devs Subject: Re: GHC's internal confusion about Ints and Words Yes, that's right. I'm not sure it's in core though, as the width information still seems to be available in Stg. However the lowering from stg into cmm widens it. On Tue, Oct 20, 2020 at 9:57 PM Carter Schonwald > wrote: ... are you talking about Haskell Int and word? Those are always the same size in bits and should match native point size. That is definitely an assumption of ghc It sounds like some information that is dropped after core is needed to correctly do something in stg/cmm in the context of the ARM64 ncg that was recently added to handle cint being 32bit in this context ? On Tue, Oct 20, 2020 at 5:49 AM Moritz Angermann > wrote: Alright, let me expand a bit. I've been looking at aarch64 NCG for ghc. The Linux side of things is looking really good, so I've moved onto the macOS side (I'm afraid I don't have any Windows aarch64 hardware, nor much windows knowledge to even attempt a Windows version yet). When calling C functions, the usual approach is to pass the first few arguments in registers, and then arguments that exceed the argument passing slots on the stack. The Arm AArch64 Procedure Call Standard (aapcs) for C does this by assigning 8byte slots to each overflow argument on the stack. A company I won't name, has decided to implement a slightly different variation of the Procedure Call Standard, often referred to as darwinpcs. This deviates from the aapcs for vargs, as well as for handling of spilled arguments on the stack. The aapcs allows us to generate calls to C functions without knowing the actual prototype of the function, as all arguments are simply spilled into 8byte slots on the stack. The darwinpcs however requires us to know the size of the arguments, so we can properly pack them onto the stack. Ints have 4 bytes, so we need to pack them into 4byte slots. In the process library we have this rather fun foreign import: foreign import ccall unsafe "runInteractiveProcess" c_runInteractiveProcess :: Ptr CString -> CString -> Ptr CString -> FD -> FD -> FD -> Ptr FD -> Ptr FD -> Ptr FD -> Ptr CGid -> Ptr CUid -> CInt -- reset child's SIGINT & SIGQUIT handlers -> CInt -- flags -> Ptr CString -> IO PHANDLE with the corresponding C declaration: extern ProcHandle runInteractiveProcess( char *const args[], char *workingDirectory, char **environment, int fdStdIn, int fdStdOut, int fdStdErr, int *pfdStdInput, int *pfdStdOutput, int *pfdStdError, gid_t *childGroup, uid_t *childUser, int reset_int_quit_handlers, int flags, char **failed_doing); This function thus takes 14 arguments. We pass only the first 8 arguments in registers, and the others on the stack. Argument 12 and 13 are of type int. On linux using the aapcs, we can pass those in 8byte slots on the stack. That is both of them are effectively 64bits wide when passed. However for darwinpcs, it is expected that these adhere to their size and are packed as such. Therefore Argument 12 and 13 need to be passed as 4byte slots each on the stack. This yields a moderate 8byte saving on the stack for the same function call on darwinpcs compared to aapcs. Now onto GHC. When we generate function calls for foreign C functions, we deal with something like: genCCall :: ForeignTarget -- function to call -> [CmmFormal] -- where to put the result -> [CmmActual] -- arguments (of mixed type) -> BlockId -- The block we are in -> NatM (InstrBlock, Maybe BlockId) based on Cmm Nodes of the form CmmUnsafeForeignCall target result_regs args The CmmActual in the care of runInteractiveProcess hold the arguments for the function, however contrary to the function declaration, it contains I64 slots for Argument 12 and 13. Thus computing the space needed for them based on their Cmm Representations yields 8bytes, when they should really be 32bit and consume only 4 byte. To illustrate this a bit better: here is what we see in the pretty printed cmm: (_s6w3::I64) = call "ccall" arg hints: [PtrHint, PtrHint, PtrHint, signed, signed, signed, PtrHint, PtrHint, PtrHint, PtrHint, PtrHint, signed, signed, PtrHint] result hints: [signed] _runInteractiveProcess(I64[Sp + 96], I64[Sp + 88], I64[Sp + 104], I64[Sp + 112], I64[Sp + 120], I64[Sp + 56], I64[Sp + 64], I64[Sp + 72], I64[Sp + 24], 0, 0, I64[Sp + 8], I64[Sp + 40] | I64[Sp + 48] | I64[Sp + 80] | 3, I64[R1 + 7]); I've added size information to the ForeignHints (NoHint, AddrHint, SignedHint) we have, and computed both, which yields: [(CmmReg (CmmLocal (LocalReg s6Gi (CmmType BitsCat W64))),AddrHint) ,(CmmReg (CmmLocal (LocalReg s6Gk (CmmType BitsCat W64))),AddrHint) ,(CmmReg (CmmLocal (LocalReg s6Gm (CmmType BitsCat W64))),AddrHint) ,(CmmReg (CmmLocal (LocalReg s6Go (CmmType BitsCat W64))),SignedHint W32) ,(CmmReg (CmmLocal (LocalReg s6Gq (CmmType BitsCat W64))),SignedHint W32) ,(CmmReg (CmmLocal (LocalReg s6Gs (CmmType BitsCat W64))),SignedHint W32) ,(CmmReg (CmmLocal (LocalReg s6Gu (CmmType BitsCat W64))),AddrHint) ,(CmmReg (CmmLocal (LocalReg s6Gw (CmmType BitsCat W64))),AddrHint) ,(CmmReg (CmmLocal (LocalReg s6Gy (CmmType BitsCat W64))),AddrHint) ,(CmmReg (CmmLocal (LocalReg s6Cp (CmmType BitsCat W64))),AddrHint) ,(CmmReg (CmmLocal (LocalReg s6FU (CmmType BitsCat W64))),AddrHint) ,(CmmReg (CmmLocal (LocalReg s6GA (CmmType BitsCat W64))),SignedHint W32) ,(CmmReg (CmmLocal (LocalReg s6GR (CmmType BitsCat W64))),SignedHint W32) ,(CmmReg (CmmLocal (LocalReg s6GM (CmmType BitsCat W64))),AddrHint)] Thus, while we *do* know the right size from STG (which is what the Hints are computed from), we loose this information when lowering into Cmm, where we represent them with W64. This is what I was alluding to in the previous email. In primRepCmmType, and mkIntCLit, we set their type to 64bit for Ints; which on this platform does not hold. Now I've gone ahead and effectively assume Cmm is lying to me when generating Foreign Function Calls, and rely on the (new) sized hints to produce the appropriate argument packing on the stack. However I believe the correct way would be for GHC not to conflate Ints and Words in Cmm; implicitly assuming they are the same width. Sadly it's not as simple as having primRepCmmType and mkIntCLit produce 32bit types. I fear GHC internally assumes "Int" means 64bit Integer, and then just happens to make the Int ~ CInt assumption. Cheers, Moritz On Tue, Oct 20, 2020 at 3:33 PM Simon Peyton Jones > wrote: Moritz I’m afraid I don’t understand any of this. Not your fault, but I just don’t have enough context to know what you mean. Is there a current bug? If so, can you demonstrate it? If not, what is the problem you want to solve? Examples are always helpful. Maybe it’s worth opening a ticket too? Thanks! Simon From: ghc-devs > On Behalf Of Moritz Angermann Sent: 20 October 2020 02:51 To: ghc-devs > Subject: GHC's internal confusion about Ints and Words Hi there! So there is a procedure calling convention that for reasons I did not fully understand, but seem to be historically grown, uses packed arguments for those that are spilled onto the stack. On top of that, CInt is 32bit, Word is 64bits. This provides the following spectacle: While we know in STG that the CInt is 32bits wide, when lowered into Cmm, it's represented as I64 in the arguments to the C function. Thus packing based on the format of the Cmm type would yield 8 bytes. And now, all further packed arguments have the wrong offset (by four). Specifically in GHC.Cmm.Utils we find: primRepCmmType :: Platform -> PrimRep -> CmmType primRepCmmType platform IntRep = bWord platform mkIntCLit :: Platform -> Int -> CmmLit mkIntCLit platform i = CmmInt (toInteger i) (wordWidth platform) The naive idea to just fix this and make them return cIntWidth instead, seemingly produces the correct Cmm expressions at a local level, but produces a broken compiler. A second approach could be to extend the Hints into providing sizes, and using those during the foreign call generation to pack spilled arguments. This however appears to be more of a patching up of some fundamental underlying issue, instead of rectifying it properly. Maybe I'll have to go down the Hint path, it does however break current Eq assumptions, as they are sized now, and what was equal before, is only equal now if they represent the same size. From a cursory glance at the issues with naively fixing the width for Int, it seems that GHC internally assumes sizeof(Int) = sizeof(Word). Maybe there is a whole level of HsInt vs CInt discrimination missing? Cheers, Moritz _______________________________________________ ghc-devs mailing list ghc-devs at haskell.org http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs -------------- next part -------------- An HTML attachment was scrubbed... URL: From cheng.shao at tweag.io Tue Oct 20 14:34:21 2020 From: cheng.shao at tweag.io (Cheng Shao) Date: Tue, 20 Oct 2020 16:34:21 +0200 Subject: GHC's internal confusion about Ints and Words In-Reply-To: References: Message-ID: Indeed STG to Cmm lowering drops the correct size information for ccall arguments, there's even a TODO comment that has been around for quite a few years: https://gitlab.haskell.org/ghc/ghc/-/blob/master/compiler/GHC/StgToCmm/Foreign.hs#L83 This has been an annoyance for Asterius as well. When we try to translate a CmmUnsafeForeignCall node to a wasm function call, a CInt argument (which should be i32 in wasm) can be mistyped as i64 which causes a validation error. We have to insert wrap/extend opcodes based on the callee function signature, but if we preserve correct argument size in Cmm (or at least enrich the hints to include it), we won't need such a hack. On Tue, Oct 20, 2020 at 4:05 PM Moritz Angermann wrote: > > Yes, that's right. I'm not sure it's in core though, as the width information still seems to be available in Stg. However the lowering from > stg into cmm widens it. > > On Tue, Oct 20, 2020 at 9:57 PM Carter Schonwald wrote: >> >> ... are you talking about Haskell Int and word? Those are always the same size in bits and should match native point size. That is definitely an assumption of ghc >> >> It sounds like some information that is dropped after core is needed to correctly do something in stg/cmm in the context of the ARM64 ncg that was recently added to handle cint being 32bit in this context ? >> >> >> On Tue, Oct 20, 2020 at 5:49 AM Moritz Angermann wrote: >>> >>> Alright, let me expand a bit. I've been looking at aarch64 NCG for ghc. The Linux side of things is looking really good, >>> so I've moved onto the macOS side (I'm afraid I don't have any Windows aarch64 hardware, nor much windows knowledge >>> to even attempt a Windows version yet). >>> >>> When calling C functions, the usual approach is to pass the first few arguments in registers, and then arguments that exceed >>> the argument passing slots on the stack. The Arm AArch64 Procedure Call Standard (aapcs) for C does this by assigning 8byte >>> slots to each overflow argument on the stack. A company I won't name, has decided to implement a slightly different variation of >>> the Procedure Call Standard, often referred to as darwinpcs. This deviates from the aapcs for vargs, as well as for handling of >>> spilled arguments on the stack. >>> >>> The aapcs allows us to generate calls to C functions without knowing the actual prototype of the function, as all arguments are >>> simply spilled into 8byte slots on the stack. The darwinpcs however requires us to know the size of the arguments, so we can >>> properly pack them onto the stack. Ints have 4 bytes, so we need to pack them into 4byte slots. >>> >>> In the process library we have this rather fun foreign import: >>> foreign import ccall unsafe "runInteractiveProcess" >>> c_runInteractiveProcess >>> :: Ptr CString >>> -> CString >>> -> Ptr CString >>> -> FD >>> -> FD >>> -> FD >>> -> Ptr FD >>> -> Ptr FD >>> -> Ptr FD >>> -> Ptr CGid >>> -> Ptr CUid >>> -> CInt -- reset child's SIGINT & SIGQUIT handlers >>> -> CInt -- flags >>> -> Ptr CString >>> -> IO PHANDLE >>> >>> with the corresponding C declaration: >>> >>> extern ProcHandle runInteractiveProcess( char *const args[], >>> char *workingDirectory, >>> char **environment, >>> int fdStdIn, >>> int fdStdOut, >>> int fdStdErr, >>> int *pfdStdInput, >>> int *pfdStdOutput, >>> int *pfdStdError, >>> gid_t *childGroup, >>> uid_t *childUser, >>> int reset_int_quit_handlers, >>> int flags, >>> char **failed_doing); >>> This function thus takes 14 arguments. We pass only the first 8 arguments in registers, and the others on the stack. >>> Argument 12 and 13 are of type int. On linux using the aapcs, we can pass those in 8byte slots on the stack. That is >>> both of them are effectively 64bits wide when passed. However for darwinpcs, it is expected that these adhere to their >>> size and are packed as such. Therefore Argument 12 and 13 need to be passed as 4byte slots each on the stack. >>> >>> This yields a moderate 8byte saving on the stack for the same function call on darwinpcs compared to aapcs. >>> >>> Now onto GHC. When we generate function calls for foreign C functions, we deal with something like: >>> >>> genCCall >>> :: ForeignTarget -- function to call >>> -> [CmmFormal] -- where to put the result >>> -> [CmmActual] -- arguments (of mixed type) >>> -> BlockId -- The block we are in >>> -> NatM (InstrBlock, Maybe BlockId) >>> >>> based on Cmm Nodes of the form CmmUnsafeForeignCall target result_regs args >>> >>> The CmmActual in the care of runInteractiveProcess hold the arguments for the function, however contrary to the function >>> declaration, it contains I64 slots for Argument 12 and 13. Thus computing the space needed for them based on their Cmm >>> Representations yields 8bytes, when they should really be 32bit and consume only 4 byte. >>> >>> To illustrate this a bit better: here is what we see in the pretty printed cmm: >>> >>> (_s6w3::I64) = call "ccall" arg hints: [PtrHint, PtrHint, PtrHint, signed, signed, signed, PtrHint, PtrHint, PtrHint, PtrHint, PtrHint, signed, signed, PtrHint] result hints: [signed] _runInteractiveProcess(I64[Sp + 96], I64[Sp + 88], I64[Sp + 104], I64[Sp + 112], I64[Sp + 120], I64[Sp + 56], I64[Sp + 64], I64[Sp + 72], I64[Sp + 24], 0, 0, I64[Sp + 8], I64[Sp + 40] | I64[Sp + 48] | I64[Sp + 80] | 3, I64[R1 + 7]); >>> >>> I've added size information to the ForeignHints (NoHint, AddrHint, SignedHint) we have, and computed both, which yields: >>> >>> [(CmmReg (CmmLocal (LocalReg s6Gi (CmmType BitsCat W64))),AddrHint) >>> ,(CmmReg (CmmLocal (LocalReg s6Gk (CmmType BitsCat W64))),AddrHint) >>> ,(CmmReg (CmmLocal (LocalReg s6Gm (CmmType BitsCat W64))),AddrHint) >>> ,(CmmReg (CmmLocal (LocalReg s6Go (CmmType BitsCat W64))),SignedHint W32) >>> ,(CmmReg (CmmLocal (LocalReg s6Gq (CmmType BitsCat W64))),SignedHint W32) >>> ,(CmmReg (CmmLocal (LocalReg s6Gs (CmmType BitsCat W64))),SignedHint W32) >>> ,(CmmReg (CmmLocal (LocalReg s6Gu (CmmType BitsCat W64))),AddrHint) >>> ,(CmmReg (CmmLocal (LocalReg s6Gw (CmmType BitsCat W64))),AddrHint) >>> ,(CmmReg (CmmLocal (LocalReg s6Gy (CmmType BitsCat W64))),AddrHint) >>> ,(CmmReg (CmmLocal (LocalReg s6Cp (CmmType BitsCat W64))),AddrHint) >>> ,(CmmReg (CmmLocal (LocalReg s6FU (CmmType BitsCat W64))),AddrHint) >>> ,(CmmReg (CmmLocal (LocalReg s6GA (CmmType BitsCat W64))),SignedHint W32) >>> ,(CmmReg (CmmLocal (LocalReg s6GR (CmmType BitsCat W64))),SignedHint W32) >>> ,(CmmReg (CmmLocal (LocalReg s6GM (CmmType BitsCat W64))),AddrHint)] >>> >>> Thus, while we *do* know the right size from STG (which is what the Hints are computed from), we loose this information when lowering >>> into Cmm, where we represent them with W64. This is what I was alluding to in the previous email. In primRepCmmType, and mkIntCLit, we set their type to 64bit for Ints; which on this platform does not hold. >>> >>> Now I've gone ahead and effectively assume Cmm is lying to me when generating Foreign Function Calls, and rely on the (new) sized >>> hints to produce the appropriate argument packing on the stack. However I believe the correct way would be for GHC not to conflate Ints >>> and Words in Cmm; implicitly assuming they are the same width. Sadly it's not as simple as having primRepCmmType and mkIntCLit produce 32bit types. I fear GHC internally assumes "Int" means 64bit Integer, and then just happens to make the Int ~ CInt assumption. >>> >>> Cheers, >>> Moritz >>> >>> On Tue, Oct 20, 2020 at 3:33 PM Simon Peyton Jones wrote: >>>> >>>> Moritz >>>> >>>> >>>> >>>> I’m afraid I don’t understand any of this. Not your fault, but I just don’t have enough context to know what you mean. >>>> >>>> >>>> >>>> Is there a current bug? If so, can you demonstrate it? If not, what is the problem you want to solve? Examples are always helpful. >>>> >>>> >>>> >>>> Maybe it’s worth opening a ticket too? >>>> >>>> >>>> >>>> Thanks! >>>> >>>> >>>> >>>> Simon >>>> >>>> >>>> >>>> From: ghc-devs On Behalf Of Moritz Angermann >>>> Sent: 20 October 2020 02:51 >>>> To: ghc-devs >>>> Subject: GHC's internal confusion about Ints and Words >>>> >>>> >>>> >>>> Hi there! >>>> >>>> >>>> >>>> So there is a procedure calling convention that for reasons I did not fully understand, but seem to be historically grown, uses packed arguments for those that are spilled onto the stack. On top of that, CInt is 32bit, Word is 64bits. This provides the following spectacle: >>>> >>>> >>>> >>>> While we know in STG that the CInt is 32bits wide, when lowered into Cmm, it's represented as I64 in the arguments to the C function. Thus packing based on the format of the Cmm type would yield 8 bytes. And now, all further packed arguments have the wrong offset (by four). >>>> >>>> >>>> >>>> Specifically in GHC.Cmm.Utils we find: >>>> >>>> primRepCmmType :: Platform -> PrimRep -> CmmType >>>> >>>> primRepCmmType platform IntRep = bWord platform >>>> >>>> >>>> >>>> mkIntCLit :: Platform -> Int -> CmmLit >>>> mkIntCLit platform i = CmmInt (toInteger i) (wordWidth platform) >>>> >>>> >>>> >>>> The naive idea to just fix this and make them return cIntWidth instead, seemingly produces the correct Cmm expressions at a local level, but produces a broken compiler. >>>> >>>> >>>> >>>> A second approach could be to extend the Hints into providing sizes, and using those during the foreign call generation to pack spilled arguments. This however appears to be more of a patching up of some fundamental underlying issue, instead of rectifying it properly. >>>> >>>> >>>> >>>> Maybe I'll have to go down the Hint path, it does however break current Eq assumptions, as they are sized now, and what was equal before, is only equal now if they represent the same size. >>>> >>>> >>>> >>>> From a cursory glance at the issues with naively fixing the width for Int, it seems that GHC internally assumes sizeof(Int) = sizeof(Word). Maybe there is a whole level of HsInt vs CInt discrimination missing? >>>> >>>> >>>> >>>> Cheers, >>>> >>>> Moritz >>> >>> _______________________________________________ >>> ghc-devs mailing list >>> ghc-devs at haskell.org >>> http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs > > _______________________________________________ > ghc-devs mailing list > ghc-devs at haskell.org > http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs From ben at well-typed.com Tue Oct 20 16:06:03 2020 From: ben at well-typed.com (Ben Gamari) Date: Tue, 20 Oct 2020 12:06:03 -0400 Subject: Fat interface files? Message-ID: <87eelsq3vb.fsf@smart-cactus.org> Hi Edward, While chatting with the ghc-ide folks recently I realized that it would be useful to be able to preserve Core such that compilation can be restarted (e.g. to be pushed down the bytecode pipeline to evaluate TH splices). As I recall this is precisely what you implemented in your "fat interface file" work. Do you recall what the state of this work was? Do you have a branch with last-known-good work? Do you recall any tricky questions that remained outstanding? Cheers, - Ben -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 487 bytes Desc: not available URL: From moritz.angermann at gmail.com Wed Oct 21 09:35:58 2020 From: moritz.angermann at gmail.com (Moritz Angermann) Date: Wed, 21 Oct 2020 17:35:58 +0800 Subject: Fat interface files? In-Reply-To: <87eelsq3vb.fsf@smart-cactus.org> References: <87eelsq3vb.fsf@smart-cactus.org> Message-ID: Just to make sure we are aware of all the ongoing efforts. We've been working on ebedding Core into interface files as well. Josh has updated the Wiki page here https://gitlab.haskell.org/ghc/ghc/-/wikis/Extensible-Interface-Files. Cheers, Moritz On Wed, Oct 21, 2020 at 12:06 AM Ben Gamari wrote: > Hi Edward, > > While chatting with the ghc-ide folks recently I realized that it would > be useful to be able to preserve Core such that compilation can be > restarted (e.g. to be pushed down the bytecode pipeline to evaluate TH > splices). > > As I recall this is precisely what you implemented in your "fat > interface file" work. Do you recall what the state of this work was? Do > you have a branch with last-known-good work? Do you recall any tricky > questions that remained outstanding? > > Cheers, > > - Ben > _______________________________________________ > ghc-devs mailing list > ghc-devs at haskell.org > http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs > -------------- next part -------------- An HTML attachment was scrubbed... URL: From simonpj at microsoft.com Wed Oct 21 09:56:15 2020 From: simonpj at microsoft.com (Simon Peyton Jones) Date: Wed, 21 Oct 2020 09:56:15 +0000 Subject: Fat interface files? In-Reply-To: References: <87eelsq3vb.fsf@smart-cactus.org> Message-ID: Thanks Mortiz That wiki page is about extensible interface files in general. It says nothing about specifically putting Core terms into interface files. For the Core part, since GHC already puts Core into unfoldings, the simple thing is just to expose all unfoldings, no? Simon From: ghc-devs On Behalf Of Moritz Angermann Sent: 21 October 2020 10:36 To: Ben Gamari Cc: Edward Yang (ezyang at cs.stanford.edu) ; ghc-devs at haskell.org Subject: Re: Fat interface files? Just to make sure we are aware of all the ongoing efforts. We've been working on ebedding Core into interface files as well. Josh has updated the Wiki page here https://gitlab.haskell.org/ghc/ghc/-/wikis/Extensible-Interface-Files. Cheers, Moritz On Wed, Oct 21, 2020 at 12:06 AM Ben Gamari > wrote: Hi Edward, While chatting with the ghc-ide folks recently I realized that it would be useful to be able to preserve Core such that compilation can be restarted (e.g. to be pushed down the bytecode pipeline to evaluate TH splices). As I recall this is precisely what you implemented in your "fat interface file" work. Do you recall what the state of this work was? Do you have a branch with last-known-good work? Do you recall any tricky questions that remained outstanding? Cheers, - Ben _______________________________________________ ghc-devs mailing list ghc-devs at haskell.org http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs -------------- next part -------------- An HTML attachment was scrubbed... URL: From moritz.angermann at gmail.com Wed Oct 21 10:03:50 2020 From: moritz.angermann at gmail.com (Moritz Angermann) Date: Wed, 21 Oct 2020 18:03:50 +0800 Subject: Fat interface files? In-Reply-To: References: <87eelsq3vb.fsf@smart-cactus.org> Message-ID: Right, my understanding is that they are not sufficient however, as Michael layed out here https://gitlab.haskell.org/ghc/ghc/-/wikis/Core-interface-section#unfoldings This should be linked together better. We'll improve this. On Wed, Oct 21, 2020 at 5:56 PM Simon Peyton Jones wrote: > Thanks Mortiz > > > > That wiki page is about extensible interface files in general. It says > nothing about specifically putting Core terms into interface files. > > > > For the Core part, since GHC already puts Core into unfoldings, the simple > thing is just to expose all unfoldings, no? > > > > Simon > > > > *From:* ghc-devs *On Behalf Of *Moritz > Angermann > *Sent:* 21 October 2020 10:36 > *To:* Ben Gamari > *Cc:* Edward Yang (ezyang at cs.stanford.edu) ; > ghc-devs at haskell.org > *Subject:* Re: Fat interface files? > > > > Just to make sure we are aware of all the ongoing efforts. We've been > working on ebedding Core into interface files as well. > > Josh has updated the Wiki page here > https://gitlab.haskell.org/ghc/ghc/-/wikis/Extensible-Interface-Files > > . > > > > Cheers, > > Moritz > > > > On Wed, Oct 21, 2020 at 12:06 AM Ben Gamari wrote: > > Hi Edward, > > While chatting with the ghc-ide folks recently I realized that it would > be useful to be able to preserve Core such that compilation can be > restarted (e.g. to be pushed down the bytecode pipeline to evaluate TH > splices). > > As I recall this is precisely what you implemented in your "fat > interface file" work. Do you recall what the state of this work was? Do > you have a branch with last-known-good work? Do you recall any tricky > questions that remained outstanding? > > Cheers, > > - Ben > _______________________________________________ > ghc-devs mailing list > ghc-devs at haskell.org > http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From zubin.duggal at gmail.com Wed Oct 21 12:11:08 2020 From: zubin.duggal at gmail.com (Zubin Duggal) Date: Wed, 21 Oct 2020 17:41:08 +0530 Subject: Fat interface files? In-Reply-To: <87eelsq3vb.fsf@smart-cactus.org> References: <87eelsq3vb.fsf@smart-cactus.org> Message-ID: <20201021121108.3qqimf6o2sqixu3h@zubin-msi> I found the branch here: https://github.com/ezyang/ghc/commit/13615ca4e4bf759f323de22a3d182b06c4050f38 On 20/10/20 12:06, Ben Gamari wrote: >Hi Edward, > >While chatting with the ghc-ide folks recently I realized that it would >be useful to be able to preserve Core such that compilation can be >restarted (e.g. to be pushed down the bytecode pipeline to evaluate TH >splices). > >As I recall this is precisely what you implemented in your "fat >interface file" work. Do you recall what the state of this work was? Do >you have a branch with last-known-good work? Do you recall any tricky >questions that remained outstanding? > >Cheers, > >- Ben -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 488 bytes Desc: not available URL: From zubin.duggal at gmail.com Wed Oct 21 12:25:10 2020 From: zubin.duggal at gmail.com (Zubin Duggal) Date: Wed, 21 Oct 2020 17:55:10 +0530 Subject: Fat interface files? In-Reply-To: References: <87eelsq3vb.fsf@smart-cactus.org> Message-ID: <20201021122510.bplhdmn6dasajft4@zubin-msi> I've been considering sneaking in the Core into the iface as `mi_rules` with specially chosen names and activation `NeverActive`. The benefit of this is that we can hijack the compilation pipeline, but still remain compatible with versions of GHC going quite a while back. On 20/10/21 18:03, Moritz Angermann wrote: >Right, my understanding is that they are not sufficient however, as Michael >layed out here >https://gitlab.haskell.org/ghc/ghc/-/wikis/Core-interface-section#unfoldings > >This should be linked together better. We'll improve this. > >On Wed, Oct 21, 2020 at 5:56 PM Simon Peyton Jones >wrote: > >> Thanks Mortiz >> >> >> >> That wiki page is about extensible interface files in general. It says >> nothing about specifically putting Core terms into interface files. >> >> >> >> For the Core part, since GHC already puts Core into unfoldings, the simple >> thing is just to expose all unfoldings, no? >> >> >> >> Simon >> >> >> >> *From:* ghc-devs *On Behalf Of *Moritz >> Angermann >> *Sent:* 21 October 2020 10:36 >> *To:* Ben Gamari >> *Cc:* Edward Yang (ezyang at cs.stanford.edu) ; >> ghc-devs at haskell.org >> *Subject:* Re: Fat interface files? >> >> >> >> Just to make sure we are aware of all the ongoing efforts. We've been >> working on ebedding Core into interface files as well. >> >> Josh has updated the Wiki page here >> https://gitlab.haskell.org/ghc/ghc/-/wikis/Extensible-Interface-Files >> >> . >> >> >> >> Cheers, >> >> Moritz >> >> >> >> On Wed, Oct 21, 2020 at 12:06 AM Ben Gamari wrote: >> >> Hi Edward, >> >> While chatting with the ghc-ide folks recently I realized that it would >> be useful to be able to preserve Core such that compilation can be >> restarted (e.g. to be pushed down the bytecode pipeline to evaluate TH >> splices). >> >> As I recall this is precisely what you implemented in your "fat >> interface file" work. Do you recall what the state of this work was? Do >> you have a branch with last-known-good work? Do you recall any tricky >> questions that remained outstanding? >> >> Cheers, >> >> - Ben >> _______________________________________________ >> ghc-devs mailing list >> ghc-devs at haskell.org >> http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs >> >> >> -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 488 bytes Desc: not available URL: From ryan.gl.scott at gmail.com Thu Oct 22 11:34:25 2020 From: ryan.gl.scott at gmail.com (Ryan Scott) Date: Thu, 22 Oct 2020 07:34:25 -0400 Subject: GitLab e-mails: down for everyone, or just me? Message-ID: Since yesterday, I haven't been receiving any e-mail notifications from GitLab. Are other people having this problem too, or is it a problem on my end? Ryan S. -------------- next part -------------- An HTML attachment was scrubbed... URL: From rae at richarde.dev Thu Oct 22 12:30:56 2020 From: rae at richarde.dev (Richard Eisenberg) Date: Thu, 22 Oct 2020 12:30:56 +0000 Subject: GitLab e-mails: down for everyone, or just me? In-Reply-To: References: Message-ID: <010f0175504a6d5f-696d4fb6-0051-43fc-a5fd-fc208687ee85-000000@us-east-2.amazonses.com> I, too, have received no notifications at all from GitLab. Richard > On Oct 22, 2020, at 7:34 AM, Ryan Scott wrote: > > Since yesterday, I haven't been receiving any e-mail notifications from GitLab. Are other people having this problem too, or is it a problem on my end? > > Ryan S. > _______________________________________________ > ghc-devs mailing list > ghc-devs at haskell.org > http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs From moritz.angermann at gmail.com Thu Oct 22 13:45:26 2020 From: moritz.angermann at gmail.com (Moritz Angermann) Date: Thu, 22 Oct 2020 21:45:26 +0800 Subject: GHC's internal confusion about Ints and Words In-Reply-To: References: Message-ID: Hi *, so, after some discussion with Simon and Simon, as well as Ben, we are all in agreement that using sized hints is a band-aid solution for the real underlying problem. Where the underlying problem is that we have CInt ~ Int32, and we represent Int32 as I32# Int#. And the proper solution would not likely be to represent Int32 as I32# Int32#. After some trial and error (mostly be being too aggressive on changing Ints to sized ones, unnecessarily -- thanks Ben for helping me stay on course!), I've produce what mostly amounts to this patch[1]. It also requires some additional narrow/extend calls to a few Data.Array.Base signatures to make them typecheck. However I've got plenty of failures in the testsuite now. Hooray! Most of them are of this form: *** Core Lint errors : in result of Desugar (before optimization) *** T12010.hsc:34:1: warning: Argument value doesn't match argument type: Fun type: Int# -> Int# Arg type: Int32# Arg: ds_d1B3 In the RHS of c_socket :: CInt -> CInt -> CInt -> IO CInt In the body of lambda with binder ds_d1AU :: Int32 In the body of lambda with binder ds_d1AV :: Int32 In the body of lambda with binder ds_d1AW :: Int32 In a case alternative: (I32# ds_d1AY :: Int32#) In a case alternative: (I32# ds_d1B0 :: Int32#) In a case alternative: (I32# ds_d1B2 :: Int32#) In the body of lambda with binder ds_d1B5 :: State# RealWorld In a case alternative: ((#,#) ds_d1B4 :: State# RealWorld, ds_d1B3 :: Int32#) Substitution: [TCvSubst In scope: InScope {} Type env: [] Co env: []] (full log at https://gist.github.com/angerman/3d6e1e3da5299b9365125ee9e0a2c40f) Some other minor ones are test that now need explicit narrow/extending where it didn't need before. As well as this beauty: -- RHS size: {terms: 16, types: 0, coercions: 0, joins: 0/0} i32 :: Int32 [GblId, Cpr=m1, Unf=Unf{Src=, TopLvl=True, Value=True, ConLike=True, WorkFree=False, Expandable=False, Guidance=IF_ARGS [] 23 10}] i32 = GHC.Int.I32# (GHC.Prim.narrowInt32# (GHC.Prim.andI# (GHC.Prim.extendInt32# (GHC.Prim.narrowInt32# (GHC.Prim.extendInt32# (GHC.Prim.narrowInt32# 1#)))) (GHC.Prim.extendInt32# (GHC.Prim.narrowInt32# (GHC.Prim.notI# (GHC.Prim.extendInt32# (GHC.Prim.narrowInt32# (GHC.Prim.extendInt32# (GHC.Prim.narrowInt32# 1#))))))))) This clearly needs some clean up. Apart from that the rest seems to be mostly working. Any input would be appreciated. I'll need to do the same for Word as well I'm afraid. Cheers, Moritz -- [1]: https://gitlab.haskell.org/ghc/ghc/-/commit/acb5ce792806bc3c1e1730c6bdae853d2755de16?merge_request_iid=3641 On Tue, Oct 20, 2020 at 10:34 PM Cheng Shao wrote: > Indeed STG to Cmm lowering drops the correct size information for > ccall arguments, there's even a TODO comment that has been around for > quite a few years: > > https://gitlab.haskell.org/ghc/ghc/-/blob/master/compiler/GHC/StgToCmm/Foreign.hs#L83 > > This has been an annoyance for Asterius as well. When we try to > translate a CmmUnsafeForeignCall node to a wasm function call, a CInt > argument (which should be i32 in wasm) can be mistyped as i64 which > causes a validation error. We have to insert wrap/extend opcodes based > on the callee function signature, but if we preserve correct argument > size in Cmm (or at least enrich the hints to include it), we won't > need such a hack. > > On Tue, Oct 20, 2020 at 4:05 PM Moritz Angermann > wrote: > > > > Yes, that's right. I'm not sure it's in core though, as the width > information still seems to be available in Stg. However the lowering from > > stg into cmm widens it. > > > > On Tue, Oct 20, 2020 at 9:57 PM Carter Schonwald < > carter.schonwald at gmail.com> wrote: > >> > >> ... are you talking about Haskell Int and word? Those are always the > same size in bits and should match native point size. That is definitely an > assumption of ghc > >> > >> It sounds like some information that is dropped after core is needed to > correctly do something in stg/cmm in the context of the ARM64 ncg that was > recently added to handle cint being 32bit in this context ? > >> > >> > >> On Tue, Oct 20, 2020 at 5:49 AM Moritz Angermann < > moritz.angermann at gmail.com> wrote: > >>> > >>> Alright, let me expand a bit. I've been looking at aarch64 NCG for > ghc. The Linux side of things is looking really good, > >>> so I've moved onto the macOS side (I'm afraid I don't have any Windows > aarch64 hardware, nor much windows knowledge > >>> to even attempt a Windows version yet). > >>> > >>> When calling C functions, the usual approach is to pass the first few > arguments in registers, and then arguments that exceed > >>> the argument passing slots on the stack. The Arm AArch64 Procedure > Call Standard (aapcs) for C does this by assigning 8byte > >>> slots to each overflow argument on the stack. A company I won't name, > has decided to implement a slightly different variation of > >>> the Procedure Call Standard, often referred to as darwinpcs. This > deviates from the aapcs for vargs, as well as for handling of > >>> spilled arguments on the stack. > >>> > >>> The aapcs allows us to generate calls to C functions without knowing > the actual prototype of the function, as all arguments are > >>> simply spilled into 8byte slots on the stack. The darwinpcs however > requires us to know the size of the arguments, so we can > >>> properly pack them onto the stack. Ints have 4 bytes, so we need to > pack them into 4byte slots. > >>> > >>> In the process library we have this rather fun foreign import: > >>> foreign import ccall unsafe "runInteractiveProcess" > >>> c_runInteractiveProcess > >>> :: Ptr CString > >>> -> CString > >>> -> Ptr CString > >>> -> FD > >>> -> FD > >>> -> FD > >>> -> Ptr FD > >>> -> Ptr FD > >>> -> Ptr FD > >>> -> Ptr CGid > >>> -> Ptr CUid > >>> -> CInt -- reset child's SIGINT & > SIGQUIT handlers > >>> -> CInt -- flags > >>> -> Ptr CString > >>> -> IO PHANDLE > >>> > >>> with the corresponding C declaration: > >>> > >>> extern ProcHandle runInteractiveProcess( char *const args[], > >>> char *workingDirectory, > >>> char **environment, > >>> int fdStdIn, > >>> int fdStdOut, > >>> int fdStdErr, > >>> int *pfdStdInput, > >>> int *pfdStdOutput, > >>> int *pfdStdError, > >>> gid_t *childGroup, > >>> uid_t *childUser, > >>> int reset_int_quit_handlers, > >>> int flags, > >>> char **failed_doing); > >>> This function thus takes 14 arguments. We pass only the first 8 > arguments in registers, and the others on the stack. > >>> Argument 12 and 13 are of type int. On linux using the aapcs, we can > pass those in 8byte slots on the stack. That is > >>> both of them are effectively 64bits wide when passed. However for > darwinpcs, it is expected that these adhere to their > >>> size and are packed as such. Therefore Argument 12 and 13 need to be > passed as 4byte slots each on the stack. > >>> > >>> This yields a moderate 8byte saving on the stack for the same function > call on darwinpcs compared to aapcs. > >>> > >>> Now onto GHC. When we generate function calls for foreign C > functions, we deal with something like: > >>> > >>> genCCall > >>> :: ForeignTarget -- function to call > >>> -> [CmmFormal] -- where to put the result > >>> -> [CmmActual] -- arguments (of mixed type) > >>> -> BlockId -- The block we are in > >>> -> NatM (InstrBlock, Maybe BlockId) > >>> > >>> based on Cmm Nodes of the form CmmUnsafeForeignCall target result_regs > args > >>> > >>> The CmmActual in the care of runInteractiveProcess hold the arguments > for the function, however contrary to the function > >>> declaration, it contains I64 slots for Argument 12 and 13. Thus > computing the space needed for them based on their Cmm > >>> Representations yields 8bytes, when they should really be 32bit and > consume only 4 byte. > >>> > >>> To illustrate this a bit better: here is what we see in the pretty > printed cmm: > >>> > >>> (_s6w3::I64) = call "ccall" arg hints: [PtrHint, PtrHint, PtrHint, > signed, signed, signed, PtrHint, PtrHint, PtrHint, PtrHint, PtrHint, > signed, signed, PtrHint] result hints: [signed] > _runInteractiveProcess(I64[Sp + 96], I64[Sp + 88], I64[Sp + 104], I64[Sp + > 112], I64[Sp + 120], I64[Sp + 56], I64[Sp + 64], I64[Sp + 72], I64[Sp + > 24], 0, 0, I64[Sp + 8], I64[Sp + 40] | I64[Sp + 48] | I64[Sp + 80] | 3, > I64[R1 + 7]); > >>> > >>> I've added size information to the ForeignHints (NoHint, AddrHint, > SignedHint) we have, and computed both, which yields: > >>> > >>> [(CmmReg (CmmLocal (LocalReg s6Gi (CmmType BitsCat W64))),AddrHint) > >>> ,(CmmReg (CmmLocal (LocalReg s6Gk (CmmType BitsCat W64))),AddrHint) > >>> ,(CmmReg (CmmLocal (LocalReg s6Gm (CmmType BitsCat W64))),AddrHint) > >>> ,(CmmReg (CmmLocal (LocalReg s6Go (CmmType BitsCat W64))),SignedHint > W32) > >>> ,(CmmReg (CmmLocal (LocalReg s6Gq (CmmType BitsCat W64))),SignedHint > W32) > >>> ,(CmmReg (CmmLocal (LocalReg s6Gs (CmmType BitsCat W64))),SignedHint > W32) > >>> ,(CmmReg (CmmLocal (LocalReg s6Gu (CmmType BitsCat W64))),AddrHint) > >>> ,(CmmReg (CmmLocal (LocalReg s6Gw (CmmType BitsCat W64))),AddrHint) > >>> ,(CmmReg (CmmLocal (LocalReg s6Gy (CmmType BitsCat W64))),AddrHint) > >>> ,(CmmReg (CmmLocal (LocalReg s6Cp (CmmType BitsCat W64))),AddrHint) > >>> ,(CmmReg (CmmLocal (LocalReg s6FU (CmmType BitsCat W64))),AddrHint) > >>> ,(CmmReg (CmmLocal (LocalReg s6GA (CmmType BitsCat W64))),SignedHint > W32) > >>> ,(CmmReg (CmmLocal (LocalReg s6GR (CmmType BitsCat W64))),SignedHint > W32) > >>> ,(CmmReg (CmmLocal (LocalReg s6GM (CmmType BitsCat W64))),AddrHint)] > >>> > >>> Thus, while we *do* know the right size from STG (which is what the > Hints are computed from), we loose this information when lowering > >>> into Cmm, where we represent them with W64. This is what I was > alluding to in the previous email. In primRepCmmType, and mkIntCLit, we set > their type to 64bit for Ints; which on this platform does not hold. > >>> > >>> Now I've gone ahead and effectively assume Cmm is lying to me when > generating Foreign Function Calls, and rely on the (new) sized > >>> hints to produce the appropriate argument packing on the stack. > However I believe the correct way would be for GHC not to conflate Ints > >>> and Words in Cmm; implicitly assuming they are the same width. Sadly > it's not as simple as having primRepCmmType and mkIntCLit produce 32bit > types. I fear GHC internally assumes "Int" means 64bit Integer, and then > just happens to make the Int ~ CInt assumption. > >>> > >>> Cheers, > >>> Moritz > >>> > >>> On Tue, Oct 20, 2020 at 3:33 PM Simon Peyton Jones < > simonpj at microsoft.com> wrote: > >>>> > >>>> Moritz > >>>> > >>>> > >>>> > >>>> I’m afraid I don’t understand any of this. Not your fault, but I > just don’t have enough context to know what you mean. > >>>> > >>>> > >>>> > >>>> Is there a current bug? If so, can you demonstrate it? If not, > what is the problem you want to solve? Examples are always helpful. > >>>> > >>>> > >>>> > >>>> Maybe it’s worth opening a ticket too? > >>>> > >>>> > >>>> > >>>> Thanks! > >>>> > >>>> > >>>> > >>>> Simon > >>>> > >>>> > >>>> > >>>> From: ghc-devs On Behalf Of Moritz > Angermann > >>>> Sent: 20 October 2020 02:51 > >>>> To: ghc-devs > >>>> Subject: GHC's internal confusion about Ints and Words > >>>> > >>>> > >>>> > >>>> Hi there! > >>>> > >>>> > >>>> > >>>> So there is a procedure calling convention that for reasons I did not > fully understand, but seem to be historically grown, uses packed arguments > for those that are spilled onto the stack. On top of that, CInt is 32bit, > Word is 64bits. This provides the following spectacle: > >>>> > >>>> > >>>> > >>>> While we know in STG that the CInt is 32bits wide, when lowered into > Cmm, it's represented as I64 in the arguments to the C function. Thus > packing based on the format of the Cmm type would yield 8 bytes. And now, > all further packed arguments have the wrong offset (by four). > >>>> > >>>> > >>>> > >>>> Specifically in GHC.Cmm.Utils we find: > >>>> > >>>> primRepCmmType :: Platform -> PrimRep -> CmmType > >>>> > >>>> primRepCmmType platform IntRep = bWord platform > >>>> > >>>> > >>>> > >>>> mkIntCLit :: Platform -> Int -> CmmLit > >>>> mkIntCLit platform i = CmmInt (toInteger i) (wordWidth platform) > >>>> > >>>> > >>>> > >>>> The naive idea to just fix this and make them return cIntWidth > instead, seemingly produces the correct Cmm expressions at a local level, > but produces a broken compiler. > >>>> > >>>> > >>>> > >>>> A second approach could be to extend the Hints into providing sizes, > and using those during the foreign call generation to pack spilled > arguments. This however appears to be more of a patching up of some > fundamental underlying issue, instead of rectifying it properly. > >>>> > >>>> > >>>> > >>>> Maybe I'll have to go down the Hint path, it does however break > current Eq assumptions, as they are sized now, and what was equal before, > is only equal now if they represent the same size. > >>>> > >>>> > >>>> > >>>> From a cursory glance at the issues with naively fixing the width for > Int, it seems that GHC internally assumes sizeof(Int) = sizeof(Word). > Maybe there is a whole level of HsInt vs CInt discrimination missing? > >>>> > >>>> > >>>> > >>>> Cheers, > >>>> > >>>> Moritz > >>> > >>> _______________________________________________ > >>> ghc-devs mailing list > >>> ghc-devs at haskell.org > >>> http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs > > > > _______________________________________________ > > ghc-devs mailing list > > ghc-devs at haskell.org > > http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs > -------------- next part -------------- An HTML attachment was scrubbed... URL: From john.ericson at obsidian.systems Thu Oct 22 20:12:02 2020 From: john.ericson at obsidian.systems (John Ericson) Date: Thu, 22 Oct 2020 16:12:02 -0400 Subject: GHC's internal confusion about Ints and Words In-Reply-To: References: Message-ID: <1681b15b-e293-f536-b00b-79c5ff99d5a0@obsidian.systems> That sounds like a great change, and beautifully dovetails with !3658. (In fact an earlier version of that PR also attempted the Int32 change.) I would just try to finish that and/or reuse the techniques. Sylvain has been doing all the work lately as I've been starved for time/ideas, so talk to him. John On 10/22/20 9:45 AM, Moritz Angermann wrote: > Hi *, > > so, after some discussion with Simon and Simon, as well as Ben, we are > all in agreement that using sized hints > is a band-aid solution for the real underlying problem. Where the > underlying problem is that we have CInt ~ Int32, > and we represent Int32 as I32# Int#.  And the proper solution would > not likely be to represent Int32 as I32# Int32#. > > After some trial and error (mostly be being too aggressive on changing > Ints to sized ones, unnecessarily -- thanks > Ben for helping me stay on course!), I've produce what mostly amounts > to this patch[1]. > > It also requires some additional narrow/extend calls to a few > Data.Array.Base signatures to make them typecheck. > > However I've got plenty of failures in the testsuite now. Hooray! > > Most of them are of this form: > > *** Core Lint errors : in result of Desugar (before optimization) *** > T12010.hsc:34:1: warning: >     Argument value doesn't match argument type: >     Fun type: Int# -> Int# >     Arg type: Int32# >     Arg: ds_d1B3 >     In the RHS of c_socket :: CInt -> CInt -> CInt -> IO CInt >     In the body of lambda with binder ds_d1AU :: Int32 >     In the body of lambda with binder ds_d1AV :: Int32 >     In the body of lambda with binder ds_d1AW :: Int32 >     In a case alternative: (I32# ds_d1AY :: Int32#) >     In a case alternative: (I32# ds_d1B0 :: Int32#) >     In a case alternative: (I32# ds_d1B2 :: Int32#) >     In the body of lambda with binder ds_d1B5 :: State# RealWorld >     In a case alternative: ((#,#) ds_d1B4 :: State# RealWorld, >                                   ds_d1B3 :: Int32#) >     Substitution: [TCvSubst >                      In scope: InScope {} >                      Type env: [] >                      Co env: []] > > (full log at > https://gist.github.com/angerman/3d6e1e3da5299b9365125ee9e0a2c40f) > > Some other minor ones are test that now need explicit narrow/extending > where it didn't need before. > > As well as this beauty: > > -- RHS size: {terms: 16, types: 0, coercions: 0, joins: 0/0} > i32 :: Int32 > [GblId, >  Cpr=m1, >  Unf=Unf{Src=, TopLvl=True, Value=True, ConLike=True, >          WorkFree=False, Expandable=False, Guidance=IF_ARGS [] 23 10}] > i32 >   = GHC.Int.I32# >       (GHC.Prim.narrowInt32# >          (GHC.Prim.andI# >             (GHC.Prim.extendInt32# >                (GHC.Prim.narrowInt32# >                   (GHC.Prim.extendInt32# (GHC.Prim.narrowInt32# 1#)))) >             (GHC.Prim.extendInt32# >                (GHC.Prim.narrowInt32# >                   (GHC.Prim.notI# >                      (GHC.Prim.extendInt32# >                         (GHC.Prim.narrowInt32# >                            (GHC.Prim.extendInt32# > (GHC.Prim.narrowInt32# 1#))))))))) > > This clearly needs some clean up. > > Apart from that the rest seems to be mostly working. Any input would > be appreciated. I'll need to do the same for > Word as well I'm afraid. > > Cheers, >  Moritz > -- > [1]: > https://gitlab.haskell.org/ghc/ghc/-/commit/acb5ce792806bc3c1e1730c6bdae853d2755de16?merge_request_iid=3641 > > On Tue, Oct 20, 2020 at 10:34 PM Cheng Shao > wrote: > > Indeed STG to Cmm lowering drops the correct size information for > ccall arguments, there's even a TODO comment that has been around for > quite a few years: > https://gitlab.haskell.org/ghc/ghc/-/blob/master/compiler/GHC/StgToCmm/Foreign.hs#L83 > > This has been an annoyance for Asterius as well. When we try to > translate a CmmUnsafeForeignCall node to a wasm function call, a CInt > argument (which should be i32 in wasm) can be mistyped as i64 which > causes a validation error. We have to insert wrap/extend opcodes based > on the callee function signature, but if we preserve correct argument > size in Cmm (or at least enrich the hints to include it), we won't > need such a hack. > > On Tue, Oct 20, 2020 at 4:05 PM Moritz Angermann > > > wrote: > > > > Yes, that's right. I'm not sure it's in core though, as the > width information still seems to be available in Stg. However the > lowering from > > stg into cmm widens it. > > > > On Tue, Oct 20, 2020 at 9:57 PM Carter Schonwald > > > wrote: > >> > >> ... are you talking about Haskell Int and word? Those are > always the same size in bits and should match native point size. > That is definitely an assumption of ghc > >> > >> It sounds like some information that is dropped after core is > needed to correctly do something in stg/cmm in the context of the > ARM64 ncg that was recently added to handle cint being 32bit in > this context ? > >> > >> > >> On Tue, Oct 20, 2020 at 5:49 AM Moritz Angermann > > > wrote: > >>> > >>> Alright, let me expand a bit.  I've been looking at aarch64 > NCG for ghc.  The Linux side of things is looking really good, > >>> so I've moved onto the macOS side (I'm afraid I don't have any > Windows aarch64 hardware, nor much windows knowledge > >>> to even attempt a Windows version yet). > >>> > >>> When calling C functions, the usual approach is to pass the > first few arguments in registers, and then arguments that exceed > >>> the argument passing slots on the stack.  The Arm AArch64 > Procedure Call Standard (aapcs) for C does this by assigning 8byte > >>> slots to each overflow argument on the stack.  A company I > won't name, has decided to implement a slightly different variation of > >>> the Procedure Call Standard, often referred to as darwinpcs.  > This deviates from the aapcs for vargs, as well as for handling of > >>> spilled arguments on the stack. > >>> > >>> The aapcs allows us to generate calls to C functions without > knowing the actual prototype of the function, as all arguments are > >>> simply spilled into 8byte slots on the stack. The darwinpcs > however requires us to know the size of the arguments, so we can > >>> properly pack them onto the stack.  Ints have 4 bytes, so we > need to pack them into 4byte slots. > >>> > >>> In the process library we have this rather fun foreign import: > >>> foreign import ccall unsafe "runInteractiveProcess" > >>>   c_runInteractiveProcess > >>>         ::  Ptr CString > >>>         -> CString > >>>         -> Ptr CString > >>>         -> FD > >>>         -> FD > >>>         -> FD > >>>         -> Ptr FD > >>>         -> Ptr FD > >>>         -> Ptr FD > >>>         -> Ptr CGid > >>>         -> Ptr CUid > >>>         -> CInt                         -- reset child's > SIGINT & SIGQUIT handlers > >>>         -> CInt                         -- flags > >>>         -> Ptr CString > >>>         -> IO PHANDLE > >>> > >>> with the corresponding C declaration: > >>> > >>> extern ProcHandle runInteractiveProcess( char *const args[], > >>>                                          char *workingDirectory, > >>>                                          char **environment, > >>>                                          int fdStdIn, > >>>                                          int fdStdOut, > >>>                                          int fdStdErr, > >>>                                          int *pfdStdInput, > >>>                                          int *pfdStdOutput, > >>>                                          int *pfdStdError, > >>>                                          gid_t *childGroup, > >>>                                          uid_t *childUser, > >>>                                          int > reset_int_quit_handlers, > >>>                                          int flags, > >>>                                          char **failed_doing); > >>> This function thus takes 14 arguments. We pass only the first > 8 arguments in registers, and the others on the stack. > >>> Argument 12 and 13 are of type int.  On linux using the aapcs, > we can pass those in 8byte slots on the stack. That is > >>> both of them are effectively 64bits wide when passed.  However > for darwinpcs, it is expected that these adhere to their > >>> size and are packed as such. Therefore Argument 12 and 13 need > to be passed as 4byte slots each on the stack. > >>> > >>> This yields a moderate 8byte saving on the stack for the same > function call on darwinpcs compared to aapcs. > >>> > >>> Now onto GHC.  When we generate function calls for foreign C > functions, we deal with something like: > >>> > >>> genCCall > >>>     :: ForeignTarget      -- function to call > >>>     -> [CmmFormal]        -- where to put the result > >>>     -> [CmmActual]        -- arguments (of mixed type) > >>>     -> BlockId            -- The block we are in > >>>     -> NatM (InstrBlock, Maybe BlockId) > >>> > >>> based on Cmm Nodes of the form CmmUnsafeForeignCall target > result_regs args > >>> > >>> The CmmActual in the care of runInteractiveProcess hold the > arguments for the function, however contrary to the function > >>> declaration, it contains I64 slots for Argument 12 and 13. > Thus computing the space needed for them based on their Cmm > >>> Representations yields 8bytes, when they should really be > 32bit and consume only 4 byte. > >>> > >>> To illustrate this a bit better: here is what we see in the > pretty printed cmm: > >>> > >>> (_s6w3::I64) = call "ccall" arg hints:  [PtrHint, PtrHint, > PtrHint, signed, signed, signed, PtrHint, PtrHint, PtrHint, > PtrHint, PtrHint, signed, signed, PtrHint]  result hints:  > [signed] _runInteractiveProcess(I64[Sp + 96], I64[Sp + 88], I64[Sp > + 104], I64[Sp + 112], I64[Sp + 120], I64[Sp + 56], I64[Sp + 64], > I64[Sp + 72], I64[Sp + 24], 0, 0, I64[Sp + 8], I64[Sp + 40] | > I64[Sp + 48] | I64[Sp + 80] | 3, I64[R1 + 7]); > >>> > >>> I've added size information to the ForeignHints (NoHint, > AddrHint, SignedHint) we have, and computed both, which yields: > >>> > >>> [(CmmReg (CmmLocal (LocalReg s6Gi (CmmType BitsCat > W64))),AddrHint) > >>> ,(CmmReg (CmmLocal (LocalReg s6Gk (CmmType BitsCat > W64))),AddrHint) > >>> ,(CmmReg (CmmLocal (LocalReg s6Gm (CmmType BitsCat > W64))),AddrHint) > >>> ,(CmmReg (CmmLocal (LocalReg s6Go (CmmType BitsCat > W64))),SignedHint W32) > >>> ,(CmmReg (CmmLocal (LocalReg s6Gq (CmmType BitsCat > W64))),SignedHint W32) > >>> ,(CmmReg (CmmLocal (LocalReg s6Gs (CmmType BitsCat > W64))),SignedHint W32) > >>> ,(CmmReg (CmmLocal (LocalReg s6Gu (CmmType BitsCat > W64))),AddrHint) > >>> ,(CmmReg (CmmLocal (LocalReg s6Gw (CmmType BitsCat > W64))),AddrHint) > >>> ,(CmmReg (CmmLocal (LocalReg s6Gy (CmmType BitsCat > W64))),AddrHint) > >>> ,(CmmReg (CmmLocal (LocalReg s6Cp (CmmType BitsCat > W64))),AddrHint) > >>> ,(CmmReg (CmmLocal (LocalReg s6FU (CmmType BitsCat > W64))),AddrHint) > >>> ,(CmmReg (CmmLocal (LocalReg s6GA (CmmType BitsCat > W64))),SignedHint W32) > >>> ,(CmmReg (CmmLocal (LocalReg s6GR (CmmType BitsCat > W64))),SignedHint W32) > >>> ,(CmmReg (CmmLocal (LocalReg s6GM (CmmType BitsCat > W64))),AddrHint)] > >>> > >>> Thus, while we *do* know the right size from STG (which is > what the Hints are computed from), we loose this information when > lowering > >>> into Cmm, where we represent them with W64. This is what I was > alluding to in the previous email. In primRepCmmType, and > mkIntCLit, we set their type to 64bit for Ints; which on this > platform does not hold. > >>> > >>> Now I've gone ahead and effectively assume Cmm is lying to me > when generating Foreign Function Calls, and rely on the (new) sized > >>> hints to produce the appropriate argument packing on the > stack.  However I believe the correct way would be for GHC not to > conflate Ints > >>> and Words in Cmm; implicitly assuming they are the same > width.  Sadly it's not as simple as having primRepCmmType and > mkIntCLit produce 32bit types. I fear GHC internally assumes "Int" > means 64bit Integer, and then just happens to make the Int ~ CInt > assumption. > >>> > >>> Cheers, > >>>  Moritz > >>> > >>> On Tue, Oct 20, 2020 at 3:33 PM Simon Peyton Jones > > wrote: > >>>> > >>>> Moritz > >>>> > >>>> > >>>> > >>>> I’m afraid I don’t understand any of this. Not your fault, > but  I just don’t have enough context to know what you mean. > >>>> > >>>> > >>>> > >>>> Is there a current bug?  If so, can you demonstrate it?   If > not, what is the problem you want to solve?  Examples are always > helpful. > >>>> > >>>> > >>>> > >>>> Maybe it’s worth opening a ticket too? > >>>> > >>>> > >>>> > >>>> Thanks! > >>>> > >>>> > >>>> > >>>> Simon > >>>> > >>>> > >>>> > >>>> From: ghc-devs > On Behalf Of Moritz Angermann > >>>> Sent: 20 October 2020 02:51 > >>>> To: ghc-devs > > >>>> Subject: GHC's internal confusion about Ints and Words > >>>> > >>>> > >>>> > >>>> Hi there! > >>>> > >>>> > >>>> > >>>> So there is a procedure calling convention that for reasons I > did not fully understand, but seem to be historically grown, uses > packed arguments for those that are spilled onto the stack. On top > of that, CInt is 32bit, Word is 64bits. This provides the > following spectacle: > >>>> > >>>> > >>>> > >>>> While we know in STG that the CInt is 32bits wide, when > lowered into Cmm, it's represented as I64 in the arguments to the > C function.  Thus packing based on the format of the Cmm type > would yield 8 bytes. And now, all further packed arguments have > the wrong offset (by four). > >>>> > >>>> > >>>> > >>>> Specifically in GHC.Cmm.Utils we find: > >>>> > >>>> primRepCmmType :: Platform -> PrimRep -> CmmType > >>>> > >>>> primRepCmmType platform IntRep = bWord platform > >>>> > >>>> > >>>> > >>>> mkIntCLit :: Platform -> Int -> CmmLit > >>>> mkIntCLit platform i = CmmInt (toInteger i) (wordWidth platform) > >>>> > >>>> > >>>> > >>>> The naive idea to just fix this and make them return > cIntWidth instead, seemingly produces the correct Cmm expressions > at a local level, but produces a broken compiler. > >>>> > >>>> > >>>> > >>>> A second approach could be to extend the Hints into providing > sizes, and using those during the foreign call generation to pack > spilled arguments.  This however appears to be more of a patching > up of some fundamental underlying issue, instead of rectifying it > properly. > >>>> > >>>> > >>>> > >>>> Maybe I'll have to go down the Hint path, it does however > break current Eq assumptions, as they are sized now, and what was > equal before, is only equal now if they represent the same size. > >>>> > >>>> > >>>> > >>>> From a cursory glance at the issues with naively fixing the > width for Int, it seems that GHC internally assumes sizeof(Int) = > sizeof(Word).  Maybe there is a whole level of HsInt vs CInt > discrimination missing? > >>>> > >>>> > >>>> > >>>> Cheers, > >>>> > >>>>  Moritz > >>> > >>> _______________________________________________ > >>> ghc-devs mailing list > >>> ghc-devs at haskell.org > >>> http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs > > > > _______________________________________________ > > ghc-devs mailing list > > ghc-devs at haskell.org > > http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs > > > _______________________________________________ > ghc-devs mailing list > ghc-devs at haskell.org > http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs -------------- next part -------------- An HTML attachment was scrubbed... URL: From ben at well-typed.com Fri Oct 23 13:34:49 2020 From: ben at well-typed.com (Ben Gamari) Date: Fri, 23 Oct 2020 09:34:49 -0400 Subject: GitLab down Message-ID: <871rhpoykq.fsf@smart-cactus.org> Hi all, GitLab will be down for a bit while I do a bit of maintenance. Cheers, - Ben -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 487 bytes Desc: not available URL: From ben at well-typed.com Fri Oct 23 14:05:03 2020 From: ben at well-typed.com (Ben Gamari) Date: Fri, 23 Oct 2020 10:05:03 -0400 Subject: GitLab down In-Reply-To: <871rhpoykq.fsf@smart-cactus.org> References: <871rhpoykq.fsf@smart-cactus.org> Message-ID: <87y2jxnilx.fsf@smart-cactus.org> Ben Gamari writes: > Hi all, > > GitLab will be down for a bit while I do a bit of maintenance. > GitLab is back up and notification email has been restored. Cheers, - Ben -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 487 bytes Desc: not available URL: From rae at richarde.dev Fri Oct 23 14:35:51 2020 From: rae at richarde.dev (Richard Eisenberg) Date: Fri, 23 Oct 2020 14:35:51 +0000 Subject: GitLab down In-Reply-To: <87y2jxnilx.fsf@smart-cactus.org> References: <871rhpoykq.fsf@smart-cactus.org> <87y2jxnilx.fsf@smart-cactus.org> Message-ID: <010f017555e3278d-3d649d39-ea88-4e4e-9f7b-5b86531bbf15-000000@us-east-2.amazonses.com> > On Oct 23, 2020, at 10:05 AM, Ben Gamari wrote: > > GitLab is back up and notification email has been restored. I'm seeing new emails -- thanks. Should we expect to get the notifications that weren't delivered yesterday? Thanks! Richard -------------- next part -------------- An HTML attachment was scrubbed... URL: From ben at well-typed.com Mon Oct 26 21:25:50 2020 From: ben at well-typed.com (Ben Gamari) Date: Mon, 26 Oct 2020 17:25:50 -0400 Subject: GitLab down In-Reply-To: <010f017555e3278d-3d649d39-ea88-4e4e-9f7b-5b86531bbf15-000000@us-east-2.amazonses.com> References: <871rhpoykq.fsf@smart-cactus.org> <87y2jxnilx.fsf@smart-cactus.org> <010f017555e3278d-3d649d39-ea88-4e4e-9f7b-5b86531bbf15-000000@us-east-2.amazonses.com> Message-ID: <87h7qgof1g.fsf@smart-cactus.org> Richard Eisenberg writes: >> On Oct 23, 2020, at 10:05 AM, Ben Gamari wrote: >> >> GitLab is back up and notification email has been restored. > > I'm seeing new emails -- thanks. Should we expect to get the notifications that weren't delivered yesterday? > I'm afraid not. It looks like GitLab believes that the messages were sent and I don't see any easy way to reconstruct which were lost. Cheers, - Ben -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 487 bytes Desc: not available URL: