From lexi.lambda at gmail.com  Sat Feb  1 00:23:19 2020
From: lexi.lambda at gmail.com (Alexis King)
Date: Fri, 31 Jan 2020 18:23:19 -0600
Subject: Feasibility of native RTS support for continuations?
In-Reply-To: <CANsrgP+ueYtSMKkuzh4nFdgnPH-3N+Q+LcSHHyg2QEVeu7fOxw@mail.gmail.com>
References: <C5F2E66E-E91C-4DB6-B88C-EE24C33B4B80@gmail.com>
 <VI1PR8303MB0093F04A891A00CEC953043AAD0A0@VI1PR8303MB0093.EURPRD83.prod.outlook.com>
 <D9E11FA1-2724-4835-B1F3-0BACDAC366E4@gmail.com>
 <VI1PR8303MB0093926BBCAAD612A3D76E5AAD050@VI1PR8303MB0093.EURPRD83.prod.outlook.com>
 <72859DAE-06D5-473F-BA92-AD6A40543C97@gmail.com>
 <CANsrgP+ueYtSMKkuzh4nFdgnPH-3N+Q+LcSHHyg2QEVeu7fOxw@mail.gmail.com>
Message-ID: <B786EFAF-59E1-4F85-B593-1B02D8081645@gmail.com>

> On Jan 30, 2020, at 02:35, Simon Marlow <marlowsd at gmail.com> wrote:
> 
> My guess is you can almost do what you want with asynchronous exceptions but some changes to the RTS would be needed.
> 
> There's a bit of code in the IO library that literally looks like this (https://gitlab.haskell.org/ghc/ghc/blob/master/libraries%2Fbase%2FGHC%2FIO%2FHandle%2FInternals.hs#L175)

Thanks for the pointer, I definitely had not discovered that! That is an interesting trick. I think your explanation paired with the Note is enough for it to make sense to me, though I don’t yet understand all the implementation subtleties of raiseAsync itself.

> Also you might want to optimise the implementation so that it doesn't actually tear down the stack as it copies it into the heap, so that you could avoid the need to copy it back from the heap again in shift#.

I don’t fully understand this point — do you mean “avoid the need to copy it back on continuation application”? shift# just copies the stack slice to the heap, so it doesn’t need to copy it back.

If I was right, and you were referring to continuation application rather than shift#, I agree with you there. It seems as though it ought to be possible to represent the stack slice as a stack itself, so if the stack looks like

    ┌───────────┐
    │ RET_SMALL │
    ├───────────┤
    │ CATCH     │
    ├───────────┤
    │ RESET     │
    ├───────────┤

then the captured continuation could itself essentially be a stack like

    ┌───────────┐
    │ RET_SMALL │
    ├───────────┤
    │ CATCH     │
    ├───────────┤
    │ UNDERFLOW │
    └───────────┘

where restoring a continuation just copies the captured stack and updates its underflow frame to point at the top of the current stack. If the caller promises not to use the continuation again after applying it, the copying could be skipped entirely, and the captured stack could just become the new stack.

However, I don’t understand enough about the way the RTS currently works to know if this is viable. For example, what if I write this:

    reset (mask_ (shift f))

Now presumably I want to ensure the masking state is restored when I invoke the continuation. But it can’t just be unconditionally restored, since if I write

    mask_ (reset (shift f >>= g))

then the mask frame isn’t included on the stack, so applying the continuation shouldn’t affect the masking state. Presumably this means a continuation restore can’t be as simple as copying the captured stack frames onto the current stack, since restoring certain frames affects other parts of the RTS state.

(Tangentially, it seems like this particular example is not handled properly in the existing capture/restore code, as this comment in Exception.cmm notes:

     NB. there's a bug in here.  If a thread is inside an
     unsafePerformIO, and inside maskAsyncExceptions# (there is an
     unmaskAsyncExceptions_ret on the stack), and it is blocked in an
     interruptible operation, and it receives an exception, then the
     unsafePerformIO thunk will be updated with a stack object
     containing the unmaskAsyncExceptions_ret frame.  Later, when
     someone else evaluates this thunk, the original masking state is
     not restored.

I think getting this right probably matters more if continuations are added, so that’s something else to worry about.)

> So that's shift#. What about reset#? I expect it's something like `unsafeInterleaveIO`, that is it creates a thunk to name the continuation. You probably also want a `catch` in there, so that we don't tear down more of the stack than we need to.

It would be nice to be able to capture slices of the stack that include catch frames, though theoretically it isn’t necessary — it would be possible to arrange for a continuation that wants to capture through a catch to do something like

    reset (... (catch (reset ...) ...))

instead, then call shift twice and compose the two continuations by hand (inserting another call to catch in between). I don’t know enough yet to understand all the tradeoffs involved, but I’ll see if it’s possible to get away with the userland emulation, since I figure the less code in the RTS the better!

> Hope this is helpful.

Very much so, thank you!

> On Jan 30, 2020, at 10:31, Ben Gamari <ben at smart-cactus.org> wrote:
> 
> For the record, runtime system captures the stack state in an AP_STACK
> closure. This is done in rts/RaiseAsync.c:raiseAsync and some of this is
> described in the comment attached to that function.
> 
> As Simon PJ points out, this is all very tricky stuff, especially in a
> concurrent context. If you make any changes in this area do be sure to
> keep in mind the considerations described in Note [AP_STACKs must be
> eagerly blackholed], which arose out of the very-nast #13615.

Thank you for the pointers — I did manage to find raiseAsync, but I hadn’t seen that Note. I will try my best to be suitably wary, though I imagine I’m in far enough over my head that I don’t yet know half the things I need to be wary of. :)

Alexis

From matthewtpickering at gmail.com  Sat Feb  1 09:41:10 2020
From: matthewtpickering at gmail.com (Matthew Pickering)
Date: Sat, 1 Feb 2020 09:41:10 +0000
Subject: How to run the testsuite with an out of tree compiler?
Message-ID: <CALuQ0m-=VVco02pEWXEG7zN5b-G-wQuuhO8c4J2CbB5rka9iUA@mail.gmail.com>

I am trying to run `ghcide` on GHC's tests.

The documentation says that I should pass `--test-compiler=..` to
hadrian in order to do this. When I do this however it starts trying
to configure a bunch of packages and starting to build stuff.

I don't want hadrian to build anything, I just want it to use my
patched version of GHC when running certain tests!

How can I achieve this?

Cheers,

Matt

From matthewtpickering at gmail.com  Sat Feb  1 14:20:22 2020
From: matthewtpickering at gmail.com (Matthew Pickering)
Date: Sat, 1 Feb 2020 14:20:22 +0000
Subject: How to run the testsuite with an out of tree compiler?
In-Reply-To: <CALuQ0m-=VVco02pEWXEG7zN5b-G-wQuuhO8c4J2CbB5rka9iUA@mail.gmail.com>
References: <CALuQ0m-=VVco02pEWXEG7zN5b-G-wQuuhO8c4J2CbB5rka9iUA@mail.gmail.com>
Message-ID: <CALuQ0m9geZ=E=w+LppJ5XQUJ=CSe=RyK_juKOAkzfywTmh72mA@mail.gmail.com>

I managed to hack something together by directly using `make` and
hardcoding some paths. Now happily running the testsuite..

Cheers,

Matt

On Sat, Feb 1, 2020 at 9:41 AM Matthew Pickering
<matthewtpickering at gmail.com> wrote:
>
> I am trying to run `ghcide` on GHC's tests.
>
> The documentation says that I should pass `--test-compiler=..` to
> hadrian in order to do this. When I do this however it starts trying
> to configure a bunch of packages and starting to build stuff.
>
> I don't want hadrian to build anything, I just want it to use my
> patched version of GHC when running certain tests!
>
> How can I achieve this?
>
> Cheers,
>
> Matt

From lexi.lambda at gmail.com  Sun Feb  2 04:26:27 2020
From: lexi.lambda at gmail.com (Alexis King)
Date: Sat, 1 Feb 2020 22:26:27 -0600
Subject: Feasibility of native RTS support for continuations?
In-Reply-To: <B786EFAF-59E1-4F85-B593-1B02D8081645@gmail.com>
References: <C5F2E66E-E91C-4DB6-B88C-EE24C33B4B80@gmail.com>
 <VI1PR8303MB0093F04A891A00CEC953043AAD0A0@VI1PR8303MB0093.EURPRD83.prod.outlook.com>
 <D9E11FA1-2724-4835-B1F3-0BACDAC366E4@gmail.com>
 <VI1PR8303MB0093926BBCAAD612A3D76E5AAD050@VI1PR8303MB0093.EURPRD83.prod.outlook.com>
 <72859DAE-06D5-473F-BA92-AD6A40543C97@gmail.com>
 <CANsrgP+ueYtSMKkuzh4nFdgnPH-3N+Q+LcSHHyg2QEVeu7fOxw@mail.gmail.com>
 <B786EFAF-59E1-4F85-B593-1B02D8081645@gmail.com>
Message-ID: <3BDD2CB6-5170-4C37-AB0E-24E7317AE1C7@gmail.com>

I took a stab at implementing this today, using the “continuation is a
stack” implementation strategy I described in my previous email. I
haven’t tried very hard to break it yet, but this tiny test program
works:

    {-# LANGUAGE BangPatterns, BlockArguments, MagicHash,
                 ScopedTypeVariables, UnboxedTuples #-}

    import GHC.Prim
    import GHC.Types

    data Continuation a b = Continuation# (Continuation# RealWorld a b)

    reset :: IO a -> IO a
    reset (IO m) = IO (reset# m)

    shift :: (Continuation a b -> IO b) -> IO a
    shift f = IO (shift# \k -> let !(IO m) = f (Continuation# k) in m)

    applyContinuation :: Continuation a b -> a -> IO b
    applyContinuation (Continuation# k) a = IO (applyContinuation# k a)

    main :: IO ()
    main = do
      ns <- reset do
        n <- shift \(k :: Continuation Integer [Integer]) -> do
          a <- applyContinuation k 2
          b <- applyContinuation k 3
          pure (a ++ b)
        pure [n]
      print ns

The output of this program is [2, 3], as expected.

My implementation doesn’t share any code with raiseAsync. Currently, it
isn’t very clever:

    * reset# pushes a RET_SMALL frame with a well-known info pointer,
      &stg_reset_frame_info.

    * shift# walks the stack and copies it up to the nearest reset
      frame. If the stack consists of several chunks, it only copies the
      chunk that contains the reset frame, and it just repurposes the
      other chunks as the continuation (since the stack is unwinding
      anyway).

    * applyContinuation# copies the captured stack and updates the
      UNDERFLOW frames as needed to point to the current stack.

    * I haven’t implemented it yet, but it would be straightforward to
      implement an applyContinuationOneShot# operation that works like
      applyContinuation#, but doesn’t actually copy anything and just
      updates the UNDERFLOW frames in the captured stack itself.

This seems to work in my very simple examples, but there are also things
I know it doesn’t handle properly:

    * It doesn’t make any attempt to handle modifications to the
      interrupt masking state properly. The right thing to do here is
      probably to look for mask/unmask frames on the stack during
      unwinding and to stash that information somewhere.

    * It doesn’t do anything special for UPDATE_FRAMEs, so if there’s an
      UPDATE_FRAME that owns a blackhole on the stack, things will go
      quite wrong.

      I haven’t been worrying about this because I don’t think there
      should ever be any update frames between shift# and reset#. In the
      case of raiseAsync, the location of the “prompt” is well-defined:
      it’s the update frame. But shift# captures up to an explicit
      prompt, so using shift# when there’s an update frame on the stack
      can surely only lead to nonsense... right?

    * It doesn’t do anything special for STM frames, so trying to
      capture a continuation through those will be similarly broken.

There are also probably bugs I don’t know about — I haven’t exercised
the implementation very hard yet — but I’ll keep playing with it. If
anyone is at all interested, I’ve pushed the code to a branch here:

    https://gitlab.haskell.org/lexi.lambda/ghc/compare/master...first-class-continuations

My thanks again to everyone’s help!

Alexis

From simonpj at microsoft.com  Sun Feb  2 18:38:48 2020
From: simonpj at microsoft.com (Simon Peyton Jones)
Date: Sun, 2 Feb 2020 18:38:48 +0000
Subject: Simplifying subsumption
Message-ID: <VI1PR8303MB0093832A7F102609722F8FADAD010@VI1PR8303MB0093.EURPRD83.prod.outlook.com>

Jurriaan, Ben
I have a MR up for #17775, which implements the now-accepted GHC proposal to simplify subsumption.
https://gitlab.haskell.org/ghc/ghc/merge_requests/2600
However, it requires some eta expansions in haskeline, and maybe other packages.  I don't know how to do that.  Could you, or another member of the GHC support team, help do this?  We want a MR we can validate on its own before committing, of course.
Jurriaan has all the necessary library patches.
We should not commit to HEAD until there has been some code review, though.
Moreover as we know, when committed, it'll break a number of packages, so we need to broadcast a warning, along with the patches Jurriaan has to fix them.  I'm not sure when and how to do that.
Thanks!
Simon
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.haskell.org/pipermail/ghc-devs/attachments/20200202/c48dab65/attachment.html>

From harendra.kumar at gmail.com  Sun Feb  2 22:25:41 2020
From: harendra.kumar at gmail.com (Harendra Kumar)
Date: Mon, 3 Feb 2020 03:55:41 +0530
Subject: Random crashes with memory corruption symptoms
Message-ID: <CAPW+kkZJ=Rj4LchpLMqCA7GCqe0jUePrPonPBJXth36St0ENSg@mail.gmail.com>

Hi,

While running a test-suite for the streaming library streamly I am
encountering a crash which seems to happen at random places at different
times. The common messages are:

* Segmentation fault: 11
*  internal error: scavenge_mark_stack: unimplemented/strange closure type
24792696 @ 0x4200a623e0
* internal error: update_fwd: unknown/strange object  223743520

and several other such messages. Prima facie this looks like the memory is
getting corrupted/scribbled somehow. My first suspicion was that this could
be a problem in the streamly library code. But I have stripped down the
code to bare minimum and there is no C FFI code or no poking to memory
pointers.

My next suspicion was the hspec/quickcheck testing code that is being used
in this test. I checked the hspec code to ensure that there is no C
code/pointer poking in any of the code involved. But no luck there as well,
still looking to further strip down that code.

My suspicion now is moving more towards the GHC RTS. This issue only shows
when the following conditions are met:

* hspec "parallel" combinator is used to run tests in parallel
* streamly concurrent code is being tested which can create many threads
* The GHC heap size is restricted to a small size ~32MB using "-M32M"
rts option.
* It is consistently seen with GHC 8.6.5 as well as GHC 8.8.1

It never occurs when the heap size is not restricted. I have seen random
crashes before as well with a "IO manager die" message, when using
concurrent networking IO with streamly. Though earlier it was not easily
reproducible, I stopped chasing it. But now it looks like that issue might
also be a manifestation of the same underlying problem.

My guess is it could be something in the RTS concurrency/threading related
code. Let me know if the symptoms ring a bell or if you can point to
something specific based on the symptoms. Also, what are the usual
tools/methods/debugging aids/flags to debug such issues in GHC? If not a
GHC issue what are the possible ways in which such problem can be induced
by application code?

Meanwhile, I am also trying to simplify the reproducing code further to
remove other factors as much as possible. The current code is at
https://github.com/composewell/streamly on the ghc-segfault branch. Run "$
while true; do cabal run properties || break; done" in the shell and if you
are lucky it may crash soon. The test code is in "test/Prop.hs" - here
https://github.com/composewell/streamly/blob/ghc-segfault/test/Prop.hs .

-harendra
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.haskell.org/pipermail/ghc-devs/attachments/20200203/ac918fb5/attachment.html>

From matthewtpickering at gmail.com  Sun Feb  2 22:54:49 2020
From: matthewtpickering at gmail.com (Matthew Pickering)
Date: Sun, 2 Feb 2020 22:54:49 +0000
Subject: Random crashes with memory corruption symptoms
In-Reply-To: <CAPW+kkZJ=Rj4LchpLMqCA7GCqe0jUePrPonPBJXth36St0ENSg@mail.gmail.com>
References: <CAPW+kkZJ=Rj4LchpLMqCA7GCqe0jUePrPonPBJXth36St0ENSg@mail.gmail.com>
Message-ID: <CALuQ0m_Lss9kTUwwTctpiFzc0mJNyww9g7p7peLu6AsJAwyHLg@mail.gmail.com>

The way to debug these kinds of issues is to use gdb.

The best clue in your message seems to be "The GHC heap size is
restricted to a small size ~32MB using "-M32M" rts option.".

Good luck!

Matt

On Sun, Feb 2, 2020 at 10:26 PM Harendra Kumar <harendra.kumar at gmail.com> wrote:
>
> Hi,
>
> While running a test-suite for the streaming library streamly I am encountering a crash which seems to happen at random places at different times. The common messages are:
>
> * Segmentation fault: 11
> *  internal error: scavenge_mark_stack: unimplemented/strange closure type 24792696 @ 0x4200a623e0
> * internal error: update_fwd: unknown/strange object  223743520
>
> and several other such messages. Prima facie this looks like the memory is getting corrupted/scribbled somehow. My first suspicion was that this could be a problem in the streamly library code. But I have stripped down the code to bare minimum and there is no C FFI code or no poking to memory pointers.
>
> My next suspicion was the hspec/quickcheck testing code that is being used in this test. I checked the hspec code to ensure that there is no C code/pointer poking in any of the code involved. But no luck there as well, still looking to further strip down that code.
>
> My suspicion now is moving more towards the GHC RTS. This issue only shows when the following conditions are met:
>
> * hspec "parallel" combinator is used to run tests in parallel
> * streamly concurrent code is being tested which can create many threads
> * The GHC heap size is restricted to a small size ~32MB using "-M32M" rts option.
> * It is consistently seen with GHC 8.6.5 as well as GHC 8.8.1
>
> It never occurs when the heap size is not restricted. I have seen random crashes before as well with a "IO manager die" message, when using concurrent networking IO with streamly. Though earlier it was not easily reproducible, I stopped chasing it. But now it looks like that issue might also be a manifestation of the same underlying problem.
>
> My guess is it could be something in the RTS concurrency/threading related code. Let me know if the symptoms ring a bell or if you can point to something specific based on the symptoms. Also, what are the usual tools/methods/debugging aids/flags to debug such issues in GHC? If not a GHC issue what are the possible ways in which such problem can be induced by application code?
>
> Meanwhile, I am also trying to simplify the reproducing code further to remove other factors as much as possible. The current code is at https://github.com/composewell/streamly on the ghc-segfault branch. Run "$ while true; do cabal run properties || break; done" in the shell and if you are lucky it may crash soon. The test code is in "test/Prop.hs" - here https://github.com/composewell/streamly/blob/ghc-segfault/test/Prop.hs .
>
> -harendra
> _______________________________________________
> ghc-devs mailing list
> ghc-devs at haskell.org
> http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs

From liuyiyun at terpmail.umd.edu  Mon Feb  3 02:03:04 2020
From: liuyiyun at terpmail.umd.edu (Yiyun Liu)
Date: Sun, 2 Feb 2020 21:03:04 -0500
Subject: Free dictionary variables in elaborated core expressions
Message-ID: <9be2d40e-ad9c-9a1a-0f21-9ad0bad17dc5@terpmail.umd.edu>

Hi ghc-devs,

About 10 days ago, I made a thread about defining a function called 
elaborateExpr which turns a string into a core expression 
<https://github.com/yiyunliu/ghc-elaboration-test/blob/8f362ad92dc6601b4cb7e4c76f0a42bc6b64480f/src/Main.hs#L55> 
within an interactive context. Now here's an unexpected behavior which 
I'm not sure how to deal with.

Given the expression:

(\x -> x + 1) :: Int -> Int

I expect to get something that looks like:

\ (x :: Int) -> + @ Int GHC.Num.$fNumInt x (GHC.Types.I# 1#)

where GHC.Num.$fNumInt is the exported dictionary.


What I actually get is something like this:

\ (x :: Int) -> + @ Int $dNum_someuniqueid x (GHC.Types.I# 1#)

where $dNum_someuniqueid is a free dictionary variable within the 
expression.


I was confused by the occurrence of the free variable $dNum at first, 
but after running the command: "ghc -ddump-ds-preopt somefile.hs" to 
dump the core bindings, I found that the dictionary variables like 
$dNum_ are actually local variables defined at the top-level.

My objective is to inline those top-level dictionary definitions into 
the core expression using let bindings, but it seems tricky since I'm 
doing everything within an interactive context.  Calling getBindings 
<https://hackage.haskell.org/package/ghc-8.6.5/docs/GHC.html#v:getBindings> 
only gives me the expression I elaborated, but the dictionary is no 
where to be found.

Interestingly, when I turn on flags such as "DeferTypedHoles" or 
"DeferOutOfScopeVariables", all the dictionaries are defined locally in 
let bindings. However, I can't replicate that behavior even with the 
flags on in the interactive context. How do I find the dictionaries?

Thanks,

- Yiyun

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.haskell.org/pipermail/ghc-devs/attachments/20200202/5aaeda20/attachment.html>

From omeragacan at gmail.com  Mon Feb  3 05:52:15 2020
From: omeragacan at gmail.com (=?UTF-8?Q?=C3=96mer_Sinan_A=C4=9Facan?=)
Date: Mon, 3 Feb 2020 08:52:15 +0300
Subject: Random crashes with memory corruption symptoms
In-Reply-To: <CAPW+kkZJ=Rj4LchpLMqCA7GCqe0jUePrPonPBJXth36St0ENSg@mail.gmail.com>
References: <CAPW+kkZJ=Rj4LchpLMqCA7GCqe0jUePrPonPBJXth36St0ENSg@mail.gmail.com>
Message-ID: <CAMQQO3mUWWbnCSbTGHgAwNncMCGnz=YPVJHje6v1dEGfuVojxg@mail.gmail.com>

You should try with 8.8.2 which fixes a bug in the compacting GC (#17088).

When debugging it's a good idea to use the latest minor release of your GHC
version (8.8.2 in your case), as minor releases fix bugs and usually do not
introduce new ones as they don't ship new features.

If the problem still exists, unless you're interested in GHC hacking I think
most productive use of the time would be to make the reproduer smaller, and
collect as many data as possible, like which flags trigger/hide the bug.

Some of the things you could check:

- Build your program with `-dcore-lint -dstg-lint -dcmm-lint` and see if it
  builds.
- Build your program with `-debug` and run it, see if it crahes.
- Build your program with `-debug` and run it with `+RTS -DS` and see if the
  error message changes.

But really you should try with 8.8.2 as first thing. It's possible that this is
another manifestation of #17088.

Ömer

Harendra Kumar <harendra.kumar at gmail.com>, 3 Şub 2020 Pzt, 01:26
tarihinde şunu yazdı:
>
> Hi,
>
> While running a test-suite for the streaming library streamly I am encountering a crash which seems to happen at random places at different times. The common messages are:
>
> * Segmentation fault: 11
> *  internal error: scavenge_mark_stack: unimplemented/strange closure type 24792696 @ 0x4200a623e0
> * internal error: update_fwd: unknown/strange object  223743520
>
> and several other such messages. Prima facie this looks like the memory is getting corrupted/scribbled somehow. My first suspicion was that this could be a problem in the streamly library code. But I have stripped down the code to bare minimum and there is no C FFI code or no poking to memory pointers.
>
> My next suspicion was the hspec/quickcheck testing code that is being used in this test. I checked the hspec code to ensure that there is no C code/pointer poking in any of the code involved. But no luck there as well, still looking to further strip down that code.
>
> My suspicion now is moving more towards the GHC RTS. This issue only shows when the following conditions are met:
>
> * hspec "parallel" combinator is used to run tests in parallel
> * streamly concurrent code is being tested which can create many threads
> * The GHC heap size is restricted to a small size ~32MB using "-M32M" rts option.
> * It is consistently seen with GHC 8.6.5 as well as GHC 8.8.1
>
> It never occurs when the heap size is not restricted. I have seen random crashes before as well with a "IO manager die" message, when using concurrent networking IO with streamly. Though earlier it was not easily reproducible, I stopped chasing it. But now it looks like that issue might also be a manifestation of the same underlying problem.
>
> My guess is it could be something in the RTS concurrency/threading related code. Let me know if the symptoms ring a bell or if you can point to something specific based on the symptoms. Also, what are the usual tools/methods/debugging aids/flags to debug such issues in GHC? If not a GHC issue what are the possible ways in which such problem can be induced by application code?
>
> Meanwhile, I am also trying to simplify the reproducing code further to remove other factors as much as possible. The current code is at https://github.com/composewell/streamly on the ghc-segfault branch. Run "$ while true; do cabal run properties || break; done" in the shell and if you are lucky it may crash soon. The test code is in "test/Prop.hs" - here https://github.com/composewell/streamly/blob/ghc-segfault/test/Prop.hs .
>
> -harendra
> _______________________________________________
> ghc-devs mailing list
> ghc-devs at haskell.org
> http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs

From harendra.kumar at gmail.com  Mon Feb  3 07:01:30 2020
From: harendra.kumar at gmail.com (Harendra Kumar)
Date: Mon, 3 Feb 2020 12:31:30 +0530
Subject: Random crashes with memory corruption symptoms
In-Reply-To: <CAMQQO3mUWWbnCSbTGHgAwNncMCGnz=YPVJHje6v1dEGfuVojxg@mail.gmail.com>
References: <CAPW+kkZJ=Rj4LchpLMqCA7GCqe0jUePrPonPBJXth36St0ENSg@mail.gmail.com>
 <CAMQQO3mUWWbnCSbTGHgAwNncMCGnz=YPVJHje6v1dEGfuVojxg@mail.gmail.com>
Message-ID: <CAPW+kkYBKL8=k8fqV5Spnp_Vofhkew8c_47fQ6KA4JEthh7+EA@mail.gmail.com>

Unfortunately, it is present in 8.8.2 as well.

On Mon, 3 Feb 2020 at 11:22, Ömer Sinan Ağacan <omeragacan at gmail.com> wrote:

> You should try with 8.8.2 which fixes a bug in the compacting GC (#17088).
>
> When debugging it's a good idea to use the latest minor release of your GHC
> version (8.8.2 in your case), as minor releases fix bugs and usually do not
> introduce new ones as they don't ship new features.
>
> If the problem still exists, unless you're interested in GHC hacking I
> think
> most productive use of the time would be to make the reproduer smaller, and
> collect as many data as possible, like which flags trigger/hide the bug.
>
> Some of the things you could check:
>
> - Build your program with `-dcore-lint -dstg-lint -dcmm-lint` and see if it
>   builds.
> - Build your program with `-debug` and run it, see if it crahes.
> - Build your program with `-debug` and run it with `+RTS -DS` and see if
> the
>   error message changes.
>
> But really you should try with 8.8.2 as first thing. It's possible that
> this is
> another manifestation of #17088.
>
> Ömer
>
> Harendra Kumar <harendra.kumar at gmail.com>, 3 Şub 2020 Pzt, 01:26
> tarihinde şunu yazdı:
> >
> > Hi,
> >
> > While running a test-suite for the streaming library streamly I am
> encountering a crash which seems to happen at random places at different
> times. The common messages are:
> >
> > * Segmentation fault: 11
> > *  internal error: scavenge_mark_stack: unimplemented/strange closure
> type 24792696 @ 0x4200a623e0
> > * internal error: update_fwd: unknown/strange object  223743520
> >
> > and several other such messages. Prima facie this looks like the memory
> is getting corrupted/scribbled somehow. My first suspicion was that this
> could be a problem in the streamly library code. But I have stripped down
> the code to bare minimum and there is no C FFI code or no poking to memory
> pointers.
> >
> > My next suspicion was the hspec/quickcheck testing code that is being
> used in this test. I checked the hspec code to ensure that there is no C
> code/pointer poking in any of the code involved. But no luck there as well,
> still looking to further strip down that code.
> >
> > My suspicion now is moving more towards the GHC RTS. This issue only
> shows when the following conditions are met:
> >
> > * hspec "parallel" combinator is used to run tests in parallel
> > * streamly concurrent code is being tested which can create many threads
> > * The GHC heap size is restricted to a small size ~32MB using "-M32M"
> rts option.
> > * It is consistently seen with GHC 8.6.5 as well as GHC 8.8.1
> >
> > It never occurs when the heap size is not restricted. I have seen random
> crashes before as well with a "IO manager die" message, when using
> concurrent networking IO with streamly. Though earlier it was not easily
> reproducible, I stopped chasing it. But now it looks like that issue might
> also be a manifestation of the same underlying problem.
> >
> > My guess is it could be something in the RTS concurrency/threading
> related code. Let me know if the symptoms ring a bell or if you can point
> to something specific based on the symptoms. Also, what are the usual
> tools/methods/debugging aids/flags to debug such issues in GHC? If not a
> GHC issue what are the possible ways in which such problem can be induced
> by application code?
> >
> > Meanwhile, I am also trying to simplify the reproducing code further to
> remove other factors as much as possible. The current code is at
> https://github.com/composewell/streamly on the ghc-segfault branch. Run
> "$ while true; do cabal run properties || break; done" in the shell and if
> you are lucky it may crash soon. The test code is in "test/Prop.hs" - here
> https://github.com/composewell/streamly/blob/ghc-segfault/test/Prop.hs .
> >
> > -harendra
> > _______________________________________________
> > ghc-devs mailing list
> > ghc-devs at haskell.org
> > http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.haskell.org/pipermail/ghc-devs/attachments/20200203/42a07f71/attachment.html>

From omeragacan at gmail.com  Mon Feb  3 08:38:59 2020
From: omeragacan at gmail.com (=?UTF-8?Q?=C3=96mer_Sinan_A=C4=9Facan?=)
Date: Mon, 3 Feb 2020 11:38:59 +0300
Subject: Random crashes with memory corruption symptoms
In-Reply-To: <CAPW+kkYBKL8=k8fqV5Spnp_Vofhkew8c_47fQ6KA4JEthh7+EA@mail.gmail.com>
References: <CAPW+kkZJ=Rj4LchpLMqCA7GCqe0jUePrPonPBJXth36St0ENSg@mail.gmail.com>
 <CAMQQO3mUWWbnCSbTGHgAwNncMCGnz=YPVJHje6v1dEGfuVojxg@mail.gmail.com>
 <CAPW+kkYBKL8=k8fqV5Spnp_Vofhkew8c_47fQ6KA4JEthh7+EA@mail.gmail.com>
Message-ID: <CAMQQO3kT8HwMrwFJK_5FJCzXDSEuSAjy_gkHrNrV8Uh=nE0DiA@mail.gmail.com>

In that case it'd be good to move the discussion to Gitlab. Could you file an
issue?

I was able to reproduce on GHC HEAD. With debug runtime I consistently get this
assertion error:

    internal error: ASSERTION FAILED: file rts/Messages.c, line 95

        (GHC version 8.11.0.20200201 for x86_64_unknown_linux)
        Please report this as a GHC bug:  https://www.haskell.org/ghc/reportabug

In non-debug runtime it works fine maybe half of the time, in others I get a
panic in the GC.

Ömer

Harendra Kumar <harendra.kumar at gmail.com>, 3 Şub 2020 Pzt, 10:01
tarihinde şunu yazdı:
>
> Unfortunately, it is present in 8.8.2 as well.
>
> On Mon, 3 Feb 2020 at 11:22, Ömer Sinan Ağacan <omeragacan at gmail.com> wrote:
>>
>> You should try with 8.8.2 which fixes a bug in the compacting GC (#17088).
>>
>> When debugging it's a good idea to use the latest minor release of your GHC
>> version (8.8.2 in your case), as minor releases fix bugs and usually do not
>> introduce new ones as they don't ship new features.
>>
>> If the problem still exists, unless you're interested in GHC hacking I think
>> most productive use of the time would be to make the reproduer smaller, and
>> collect as many data as possible, like which flags trigger/hide the bug.
>>
>> Some of the things you could check:
>>
>> - Build your program with `-dcore-lint -dstg-lint -dcmm-lint` and see if it
>>   builds.
>> - Build your program with `-debug` and run it, see if it crahes.
>> - Build your program with `-debug` and run it with `+RTS -DS` and see if the
>>   error message changes.
>>
>> But really you should try with 8.8.2 as first thing. It's possible that this is
>> another manifestation of #17088.
>>
>> Ömer
>>
>> Harendra Kumar <harendra.kumar at gmail.com>, 3 Şub 2020 Pzt, 01:26
>> tarihinde şunu yazdı:
>> >
>> > Hi,
>> >
>> > While running a test-suite for the streaming library streamly I am encountering a crash which seems to happen at random places at different times. The common messages are:
>> >
>> > * Segmentation fault: 11
>> > *  internal error: scavenge_mark_stack: unimplemented/strange closure type 24792696 @ 0x4200a623e0
>> > * internal error: update_fwd: unknown/strange object  223743520
>> >
>> > and several other such messages. Prima facie this looks like the memory is getting corrupted/scribbled somehow. My first suspicion was that this could be a problem in the streamly library code. But I have stripped down the code to bare minimum and there is no C FFI code or no poking to memory pointers.
>> >
>> > My next suspicion was the hspec/quickcheck testing code that is being used in this test. I checked the hspec code to ensure that there is no C code/pointer poking in any of the code involved. But no luck there as well, still looking to further strip down that code.
>> >
>> > My suspicion now is moving more towards the GHC RTS. This issue only shows when the following conditions are met:
>> >
>> > * hspec "parallel" combinator is used to run tests in parallel
>> > * streamly concurrent code is being tested which can create many threads
>> > * The GHC heap size is restricted to a small size ~32MB using "-M32M" rts option.
>> > * It is consistently seen with GHC 8.6.5 as well as GHC 8.8.1
>> >
>> > It never occurs when the heap size is not restricted. I have seen random crashes before as well with a "IO manager die" message, when using concurrent networking IO with streamly. Though earlier it was not easily reproducible, I stopped chasing it. But now it looks like that issue might also be a manifestation of the same underlying problem.
>> >
>> > My guess is it could be something in the RTS concurrency/threading related code. Let me know if the symptoms ring a bell or if you can point to something specific based on the symptoms. Also, what are the usual tools/methods/debugging aids/flags to debug such issues in GHC? If not a GHC issue what are the possible ways in which such problem can be induced by application code?
>> >
>> > Meanwhile, I am also trying to simplify the reproducing code further to remove other factors as much as possible. The current code is at https://github.com/composewell/streamly on the ghc-segfault branch. Run "$ while true; do cabal run properties || break; done" in the shell and if you are lucky it may crash soon. The test code is in "test/Prop.hs" - here https://github.com/composewell/streamly/blob/ghc-segfault/test/Prop.hs .
>> >
>> > -harendra
>> > _______________________________________________
>> > ghc-devs mailing list
>> > ghc-devs at haskell.org
>> > http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs

From simonpj at microsoft.com  Mon Feb  3 08:55:19 2020
From: simonpj at microsoft.com (Simon Peyton Jones)
Date: Mon, 3 Feb 2020 08:55:19 +0000
Subject: Simplifying subsumption
In-Reply-To: <181C77B7-5ED4-418E-BE48-A545488154AC@uu.nl>
References: <VI1PR8303MB0093832A7F102609722F8FADAD010@VI1PR8303MB0093.EURPRD83.prod.outlook.com>
 <181C77B7-5ED4-418E-BE48-A545488154AC@uu.nl>
Message-ID: <AM5PR8303MB008283ECFB7B0E2A8B4902BFAD000@AM5PR8303MB0082.EURPRD83.prod.outlook.com>

Jurriaan,

It would be good to dig out the patches that affect packages without which GHC cannot build. (E.g. haskeline.)   Would that take long? You have them already...   It doesn't have to be perfect or git-tracked; just send a patch file in an email.  I'm just trying to avoid re-inventing the wheel.

A more systematic set of patches, for all the hackage packages you looked at, could certainly wait a week or three.

Thanks

Simon

|  -----Original Message-----
|  From: Jurriaan Hage <J.Hage at uu.nl>
|  Sent: 02 February 2020 19:00
|  To: Simon Peyton Jones <simonpj at microsoft.com>
|  Cc: Jurriaan Hage <jur at cs.uu.nl>; Ben Gamari <ben at well-typed.com>; ghc-
|  devs <ghc-devs at haskell.org>
|  Subject: Re: Simplifying subsumption
|  
|  Hi Simon,
|  
|  Fine by me.
|  What kind of timeline are we looking at? I have to grade exams the coming
|  week, since I was ill last week,
|  and the  week afterwards I am in Poland (but maybe I do have time for this
|  kind of thing then).
|  
|  Jur
|  
|  > On 2Feb, 2020, at 19:38, Simon Peyton Jones <simonpj at microsoft.com>
|  wrote:
|  >
|  > Jurriaan, Ben
|  >
|  > I have a MR up for #17775, which implements the now-accepted GHC
|  proposal to simplify subsumption.
|  >
|  >
|  https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgitlab.h
|  askell.org%2Fghc%2Fghc%2Fmerge_requests%2F2600&amp;data=02%7C01%7Csimonpj%
|  40microsoft.com%7C3d05874428304c245e9d08d7a8121882%7C72f988bf86f141af91ab2
|  d7cd011db47%7C1%7C0%7C637162667999082070&amp;sdata=Y7s3L%2F1xrCSxdF0EfRNik
|  YJnOU2Y6RtVlR92irMg7cU%3D&amp;reserved=0
|  >
|  > However, it requires some eta expansions in haskeline, and maybe other
|  packages.  I don’t know how to do that.  Could you, or another member of
|  the GHC support team, help do this?  We want a MR we can validate on its
|  own before committing, of course.
|  >
|  > Jurriaan has all the necessary library patches.
|  >
|  > We should not commit to HEAD until there has been some code review,
|  though.
|  >
|  > Moreover as we know, when committed, it’ll break a number of packages,
|  so we need to broadcast a warning, along with the patches Jurriaan has to
|  fix them.  I’m not sure when and how to do that.
|  >
|  > Thanks!
|  >
|  > Simon
|  >


From simonpj at microsoft.com  Mon Feb  3 09:18:33 2020
From: simonpj at microsoft.com (Simon Peyton Jones)
Date: Mon, 3 Feb 2020 09:18:33 +0000
Subject: Free dictionary variables in elaborated core expressions
In-Reply-To: <9be2d40e-ad9c-9a1a-0f21-9ad0bad17dc5@terpmail.umd.edu>
References: <9be2d40e-ad9c-9a1a-0f21-9ad0bad17dc5@terpmail.umd.edu>
Message-ID: <AM5PR8303MB0082F5EBC2A9A450F232A5B6AD000@AM5PR8303MB0082.EURPRD83.prod.outlook.com>

In your code (elabRnExpr) you have


_ <- perhaps_disable_default_warnings $ simplifyInteractive residual

You’ll notice that

simplifyInteractive :: WantedConstraints -> TcM (Bag EvBind)

So you are discarding the “evidence bindings” returned by simplifyInteractive.  Those are precisely the bindings of the dictionaries (dictionaries are a form of “evidence”) that you need.  Don’t discard them.

Untested:

ev_binds <- perhaps_disable_default_warnings $ simplifyInteractive residual

let full_expr = mkHsDictLet (EvBinds ev_binds) tc_expr

zonkTopLExpr full_expr

Simon

From: ghc-devs <ghc-devs-bounces at haskell.org> On Behalf Of Yiyun Liu
Sent: 03 February 2020 02:03
To: ghc-devs at haskell.org
Cc: James Parker <jp at jamesparker.me>
Subject: Free dictionary variables in elaborated core expressions


Hi ghc-devs,

About 10 days ago, I made a thread about defining a function called elaborateExpr which turns a string into a core expression<https://github.com/yiyunliu/ghc-elaboration-test/blob/8f362ad92dc6601b4cb7e4c76f0a42bc6b64480f/src/Main.hs#L55> within an interactive context. Now here's an unexpected behavior which I'm not sure how to deal with.

Given the expression:

(\x -> x + 1) :: Int -> Int

I expect to get something that looks like:

\ (x :: Int) -> + @ Int GHC.Num.$fNumInt x (GHC.Types.I# 1#)

where GHC.Num.$fNumInt is the exported dictionary.



What I actually get is something like this:

\ (x :: Int) -> + @ Int $dNum_someuniqueid x (GHC.Types.I# 1#)

where $dNum_someuniqueid is a free dictionary variable within the expression.



I was confused by the occurrence of the free variable $dNum at first, but after running the command: "ghc -ddump-ds-preopt somefile.hs" to dump the core bindings, I found that the dictionary variables like $dNum_ are actually local variables defined at the top-level.

My objective is to inline those top-level dictionary definitions into the core expression using let bindings, but it seems tricky since I'm doing everything within an interactive context.  Calling getBindings<https://hackage.haskell.org/package/ghc-8.6.5/docs/GHC.html#v:getBindings> only gives me the expression I elaborated, but the dictionary is no where to be found.

Interestingly, when I turn on flags such as "DeferTypedHoles" or "DeferOutOfScopeVariables", all the dictionaries are defined locally in let bindings. However, I can't replicate that behavior even with the flags on in the interactive context. How do I find the dictionaries?

Thanks,

- Yiyun
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.haskell.org/pipermail/ghc-devs/attachments/20200203/e862ca79/attachment.html>

From harendra.kumar at gmail.com  Mon Feb  3 09:29:40 2020
From: harendra.kumar at gmail.com (Harendra Kumar)
Date: Mon, 3 Feb 2020 14:59:40 +0530
Subject: Random crashes with memory corruption symptoms
In-Reply-To: <CAMQQO3kT8HwMrwFJK_5FJCzXDSEuSAjy_gkHrNrV8Uh=nE0DiA@mail.gmail.com>
References: <CAPW+kkZJ=Rj4LchpLMqCA7GCqe0jUePrPonPBJXth36St0ENSg@mail.gmail.com>
 <CAMQQO3mUWWbnCSbTGHgAwNncMCGnz=YPVJHje6v1dEGfuVojxg@mail.gmail.com>
 <CAPW+kkYBKL8=k8fqV5Spnp_Vofhkew8c_47fQ6KA4JEthh7+EA@mail.gmail.com>
 <CAMQQO3kT8HwMrwFJK_5FJCzXDSEuSAjy_gkHrNrV8Uh=nE0DiA@mail.gmail.com>
Message-ID: <CAPW+kkY7Gm4x2Qhryvb66MzXGkDzaVKe8=cOpifDEGypyNDX6Q@mail.gmail.com>

Ok, I will file an issue. I just wanted to rule out any application level
issues first. Did you try it on the code I sent or did you use some other
test case? Is there a test-suite in GHC that stresses the threaded runtime?

-harendra

On Mon, 3 Feb 2020 at 14:09, Ömer Sinan Ağacan <omeragacan at gmail.com> wrote:

> In that case it'd be good to move the discussion to Gitlab. Could you file
> an
> issue?
>
> I was able to reproduce on GHC HEAD. With debug runtime I consistently get
> this
> assertion error:
>
>     internal error: ASSERTION FAILED: file rts/Messages.c, line 95
>
>         (GHC version 8.11.0.20200201 for x86_64_unknown_linux)
>         Please report this as a GHC bug:
> https://www.haskell.org/ghc/reportabug
>
> In non-debug runtime it works fine maybe half of the time, in others I get
> a
> panic in the GC.
>
> Ömer
>
> Harendra Kumar <harendra.kumar at gmail.com>, 3 Şub 2020 Pzt, 10:01
> tarihinde şunu yazdı:
> >
> > Unfortunately, it is present in 8.8.2 as well.
> >
> > On Mon, 3 Feb 2020 at 11:22, Ömer Sinan Ağacan <omeragacan at gmail.com>
> wrote:
> >>
> >> You should try with 8.8.2 which fixes a bug in the compacting GC
> (#17088).
> >>
> >> When debugging it's a good idea to use the latest minor release of your
> GHC
> >> version (8.8.2 in your case), as minor releases fix bugs and usually do
> not
> >> introduce new ones as they don't ship new features.
> >>
> >> If the problem still exists, unless you're interested in GHC hacking I
> think
> >> most productive use of the time would be to make the reproduer smaller,
> and
> >> collect as many data as possible, like which flags trigger/hide the bug.
> >>
> >> Some of the things you could check:
> >>
> >> - Build your program with `-dcore-lint -dstg-lint -dcmm-lint` and see
> if it
> >>   builds.
> >> - Build your program with `-debug` and run it, see if it crahes.
> >> - Build your program with `-debug` and run it with `+RTS -DS` and see
> if the
> >>   error message changes.
> >>
> >> But really you should try with 8.8.2 as first thing. It's possible that
> this is
> >> another manifestation of #17088.
> >>
> >> Ömer
> >>
> >> Harendra Kumar <harendra.kumar at gmail.com>, 3 Şub 2020 Pzt, 01:26
> >> tarihinde şunu yazdı:
> >> >
> >> > Hi,
> >> >
> >> > While running a test-suite for the streaming library streamly I am
> encountering a crash which seems to happen at random places at different
> times. The common messages are:
> >> >
> >> > * Segmentation fault: 11
> >> > *  internal error: scavenge_mark_stack: unimplemented/strange closure
> type 24792696 @ 0x4200a623e0
> >> > * internal error: update_fwd: unknown/strange object  223743520
> >> >
> >> > and several other such messages. Prima facie this looks like the
> memory is getting corrupted/scribbled somehow. My first suspicion was that
> this could be a problem in the streamly library code. But I have stripped
> down the code to bare minimum and there is no C FFI code or no poking to
> memory pointers.
> >> >
> >> > My next suspicion was the hspec/quickcheck testing code that is being
> used in this test. I checked the hspec code to ensure that there is no C
> code/pointer poking in any of the code involved. But no luck there as well,
> still looking to further strip down that code.
> >> >
> >> > My suspicion now is moving more towards the GHC RTS. This issue only
> shows when the following conditions are met:
> >> >
> >> > * hspec "parallel" combinator is used to run tests in parallel
> >> > * streamly concurrent code is being tested which can create many
> threads
> >> > * The GHC heap size is restricted to a small size ~32MB using "-M32M"
> rts option.
> >> > * It is consistently seen with GHC 8.6.5 as well as GHC 8.8.1
> >> >
> >> > It never occurs when the heap size is not restricted. I have seen
> random crashes before as well with a "IO manager die" message, when using
> concurrent networking IO with streamly. Though earlier it was not easily
> reproducible, I stopped chasing it. But now it looks like that issue might
> also be a manifestation of the same underlying problem.
> >> >
> >> > My guess is it could be something in the RTS concurrency/threading
> related code. Let me know if the symptoms ring a bell or if you can point
> to something specific based on the symptoms. Also, what are the usual
> tools/methods/debugging aids/flags to debug such issues in GHC? If not a
> GHC issue what are the possible ways in which such problem can be induced
> by application code?
> >> >
> >> > Meanwhile, I am also trying to simplify the reproducing code further
> to remove other factors as much as possible. The current code is at
> https://github.com/composewell/streamly on the ghc-segfault branch. Run
> "$ while true; do cabal run properties || break; done" in the shell and if
> you are lucky it may crash soon. The test code is in "test/Prop.hs" - here
> https://github.com/composewell/streamly/blob/ghc-segfault/test/Prop.hs .
> >> >
> >> > -harendra
> >> > _______________________________________________
> >> > ghc-devs mailing list
> >> > ghc-devs at haskell.org
> >> > http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.haskell.org/pipermail/ghc-devs/attachments/20200203/7a689131/attachment-0001.html>

From omeragacan at gmail.com  Mon Feb  3 09:35:14 2020
From: omeragacan at gmail.com (=?UTF-8?Q?=C3=96mer_Sinan_A=C4=9Facan?=)
Date: Mon, 3 Feb 2020 12:35:14 +0300
Subject: Random crashes with memory corruption symptoms
In-Reply-To: <CAPW+kkY7Gm4x2Qhryvb66MzXGkDzaVKe8=cOpifDEGypyNDX6Q@mail.gmail.com>
References: <CAPW+kkZJ=Rj4LchpLMqCA7GCqe0jUePrPonPBJXth36St0ENSg@mail.gmail.com>
 <CAMQQO3mUWWbnCSbTGHgAwNncMCGnz=YPVJHje6v1dEGfuVojxg@mail.gmail.com>
 <CAPW+kkYBKL8=k8fqV5Spnp_Vofhkew8c_47fQ6KA4JEthh7+EA@mail.gmail.com>
 <CAMQQO3kT8HwMrwFJK_5FJCzXDSEuSAjy_gkHrNrV8Uh=nE0DiA@mail.gmail.com>
 <CAPW+kkY7Gm4x2Qhryvb66MzXGkDzaVKe8=cOpifDEGypyNDX6Q@mail.gmail.com>
Message-ID: <CAMQQO3mXdCTK7kxu9g5vWLn9X5ebpzhq1EHGYRHt6At1dAdrVQ@mail.gmail.com>

> Did you try it on the code I sent or did you use some other test case?

I tried your code.

It's still possible that this is an application bug, of course (maybe in one of
the dependencies, if not in your application).

> Is there a test-suite in GHC that stresses the threaded runtime?

Some of the tests in the test suite uses threaded runtime. Other than those not
really.

Ömer

Harendra Kumar <harendra.kumar at gmail.com>, 3 Şub 2020 Pzt, 12:29
tarihinde şunu yazdı:
>
> Ok, I will file an issue. I just wanted to rule out any application level issues first. Did you try it on the code I sent or did you use some other test case? Is there a test-suite in GHC that stresses the threaded runtime?
>
> -harendra
>
> On Mon, 3 Feb 2020 at 14:09, Ömer Sinan Ağacan <omeragacan at gmail.com> wrote:
>>
>> In that case it'd be good to move the discussion to Gitlab. Could you file an
>> issue?
>>
>> I was able to reproduce on GHC HEAD. With debug runtime I consistently get this
>> assertion error:
>>
>>     internal error: ASSERTION FAILED: file rts/Messages.c, line 95
>>
>>         (GHC version 8.11.0.20200201 for x86_64_unknown_linux)
>>         Please report this as a GHC bug:  https://www.haskell.org/ghc/reportabug
>>
>> In non-debug runtime it works fine maybe half of the time, in others I get a
>> panic in the GC.
>>
>> Ömer
>>
>> Harendra Kumar <harendra.kumar at gmail.com>, 3 Şub 2020 Pzt, 10:01
>> tarihinde şunu yazdı:
>> >
>> > Unfortunately, it is present in 8.8.2 as well.
>> >
>> > On Mon, 3 Feb 2020 at 11:22, Ömer Sinan Ağacan <omeragacan at gmail.com> wrote:
>> >>
>> >> You should try with 8.8.2 which fixes a bug in the compacting GC (#17088).
>> >>
>> >> When debugging it's a good idea to use the latest minor release of your GHC
>> >> version (8.8.2 in your case), as minor releases fix bugs and usually do not
>> >> introduce new ones as they don't ship new features.
>> >>
>> >> If the problem still exists, unless you're interested in GHC hacking I think
>> >> most productive use of the time would be to make the reproduer smaller, and
>> >> collect as many data as possible, like which flags trigger/hide the bug.
>> >>
>> >> Some of the things you could check:
>> >>
>> >> - Build your program with `-dcore-lint -dstg-lint -dcmm-lint` and see if it
>> >>   builds.
>> >> - Build your program with `-debug` and run it, see if it crahes.
>> >> - Build your program with `-debug` and run it with `+RTS -DS` and see if the
>> >>   error message changes.
>> >>
>> >> But really you should try with 8.8.2 as first thing. It's possible that this is
>> >> another manifestation of #17088.
>> >>
>> >> Ömer
>> >>
>> >> Harendra Kumar <harendra.kumar at gmail.com>, 3 Şub 2020 Pzt, 01:26
>> >> tarihinde şunu yazdı:
>> >> >
>> >> > Hi,
>> >> >
>> >> > While running a test-suite for the streaming library streamly I am encountering a crash which seems to happen at random places at different times. The common messages are:
>> >> >
>> >> > * Segmentation fault: 11
>> >> > *  internal error: scavenge_mark_stack: unimplemented/strange closure type 24792696 @ 0x4200a623e0
>> >> > * internal error: update_fwd: unknown/strange object  223743520
>> >> >
>> >> > and several other such messages. Prima facie this looks like the memory is getting corrupted/scribbled somehow. My first suspicion was that this could be a problem in the streamly library code. But I have stripped down the code to bare minimum and there is no C FFI code or no poking to memory pointers.
>> >> >
>> >> > My next suspicion was the hspec/quickcheck testing code that is being used in this test. I checked the hspec code to ensure that there is no C code/pointer poking in any of the code involved. But no luck there as well, still looking to further strip down that code.
>> >> >
>> >> > My suspicion now is moving more towards the GHC RTS. This issue only shows when the following conditions are met:
>> >> >
>> >> > * hspec "parallel" combinator is used to run tests in parallel
>> >> > * streamly concurrent code is being tested which can create many threads
>> >> > * The GHC heap size is restricted to a small size ~32MB using "-M32M" rts option.
>> >> > * It is consistently seen with GHC 8.6.5 as well as GHC 8.8.1
>> >> >
>> >> > It never occurs when the heap size is not restricted. I have seen random crashes before as well with a "IO manager die" message, when using concurrent networking IO with streamly. Though earlier it was not easily reproducible, I stopped chasing it. But now it looks like that issue might also be a manifestation of the same underlying problem.
>> >> >
>> >> > My guess is it could be something in the RTS concurrency/threading related code. Let me know if the symptoms ring a bell or if you can point to something specific based on the symptoms. Also, what are the usual tools/methods/debugging aids/flags to debug such issues in GHC? If not a GHC issue what are the possible ways in which such problem can be induced by application code?
>> >> >
>> >> > Meanwhile, I am also trying to simplify the reproducing code further to remove other factors as much as possible. The current code is at https://github.com/composewell/streamly on the ghc-segfault branch. Run "$ while true; do cabal run properties || break; done" in the shell and if you are lucky it may crash soon. The test code is in "test/Prop.hs" - here https://github.com/composewell/streamly/blob/ghc-segfault/test/Prop.hs .
>> >> >
>> >> > -harendra
>> >> > _______________________________________________
>> >> > ghc-devs mailing list
>> >> > ghc-devs at haskell.org
>> >> > http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs

From compl.yue at icloud.com  Mon Feb  3 14:28:12 2020
From: compl.yue at icloud.com (Compl Yue)
Date: Mon, 3 Feb 2020 22:28:12 +0800
Subject: testing ... ping
Message-ID: <b9c45981-e4be-9fad-0951-d9635ea744d4@icloud.com>

Sorry if this bothered you, I have problem sending email to the list, 
sending this for bgamari to further diagnostic the situation.


-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.haskell.org/pipermail/ghc-devs/attachments/20200203/51300147/attachment.html>

From ben at well-typed.com  Mon Feb  3 16:02:32 2020
From: ben at well-typed.com (Ben Gamari)
Date: Mon, 03 Feb 2020 11:02:32 -0500
Subject: Allowing Windows CI to fail
Message-ID: <87wo93veuk.fsf@smart-cactus.org>

Hi everyone,

After multiple weeks of effort struggling to get Windows CI into a stable
condition I'm sorry to say that we're going to need to revert to
allowing it to fail for a bit longer. The status quo is essentially
holding up the entire merge queue and we still seem quite far from
resolving the issues.

I have summarised the current state-of-play in #17777. In short, the
gcc toolchain likely can't be used reliably on Windows due to its
ubiquitous use of `exec`, which cannot be reliably implemented on
Windows.

Switching to LLVM as our native toolchain was my (initially promising)
last-ditch attempt at avoiding this issue but sadly this looks to be a
long road. My current attempt is stuck on an inscrutable loader error.

For the short-term, I am afraid I have run out of time for this effort.
My current plan is to merge what I can from my wip/windows-ci branch but
again enable the Windows CI jobs' allow_failure flag so that its
unreliable nature doesn't hold up otherwise-passing CI jobs.

While it's unfortunately that we still lack reliable CI on Windows,
I think the effords of the last few weeks were quite worthwhile. We now
have:

 * A much better understanding of the issues affecting us on Windows
 * Significantly better documentation and automation for producing our
   mingw toolchain artifacts
 * better scripting for setting up Windows CI runners
 * fixed several bugs in the ghc-jailbreak library used to work
   around the Windows MAX_PATH limitation

Many thanks to Tamar Christina for his many hours of patient help.
Without him, GHC's Windows support would be in significantly worse shape
than it is.

Users of GHC should note that the CI issues we are struggling with *do
not* affect compiled code. These bugs manifest only as (rare) failed
compilations (particularly when building GHC itself); however, once
compilation succeeds the program that results is correct and reliable.

Cheers,

- Ben
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 487 bytes
Desc: not available
URL: <http://mail.haskell.org/pipermail/ghc-devs/attachments/20200203/14adab9a/attachment.sig>

From simonpj at microsoft.com  Mon Feb  3 16:18:04 2020
From: simonpj at microsoft.com (Simon Peyton Jones)
Date: Mon, 3 Feb 2020 16:18:04 +0000
Subject: Allowing Windows CI to fail
In-Reply-To: <87wo93veuk.fsf@smart-cactus.org>
References: <87wo93veuk.fsf@smart-cactus.org>
Message-ID: <VI1PR8303MB00936162EC5A32792EDB5564AD000@VI1PR8303MB0093.EURPRD83.prod.outlook.com>

Ben

This sounds like a good decision to me, thanks.

Is there a possibility to have a slow CI-on-windows job (not part of the "this must pass before merging" step), which will slowly, but reliably, fail if the Windows build fails. E.g. does it help to make the build be 100% sequential?

Or is there currently no way to build GHC at all on Windows in a way that won't fail?  (That would be surprising to me.  Until relatively recently I was *only* building on Windows.)

Simon

| -----Original Message-----
| From: ghc-devs <ghc-devs-bounces at haskell.org> On Behalf Of Ben Gamari
| Sent: 03 February 2020 16:03
| To: GHC developers <ghc-devs at haskell.org>
| Subject: Allowing Windows CI to fail
| 
| Hi everyone,
| 
| After multiple weeks of effort struggling to get Windows CI into a stable
| condition I'm sorry to say that we're going to need to revert to allowing
| it to fail for a bit longer. The status quo is essentially holding up the
| entire merge queue and we still seem quite far from resolving the issues.
| 
| I have summarised the current state-of-play in #17777. In short, the gcc
| toolchain likely can't be used reliably on Windows due to its ubiquitous
| use of `exec`, which cannot be reliably implemented on Windows.
| 
| Switching to LLVM as our native toolchain was my (initially promising)
| last-ditch attempt at avoiding this issue but sadly this looks to be a long
| road. My current attempt is stuck on an inscrutable loader error.
| 
| For the short-term, I am afraid I have run out of time for this effort.
| My current plan is to merge what I can from my wip/windows-ci branch but
| again enable the Windows CI jobs' allow_failure flag so that its unreliable
| nature doesn't hold up otherwise-passing CI jobs.
| 
| While it's unfortunately that we still lack reliable CI on Windows, I think
| the effords of the last few weeks were quite worthwhile. We now
| have:
| 
|  * A much better understanding of the issues affecting us on Windows
|  * Significantly better documentation and automation for producing our
|    mingw toolchain artifacts
|  * better scripting for setting up Windows CI runners
|  * fixed several bugs in the ghc-jailbreak library used to work
|    around the Windows MAX_PATH limitation
| 
| Many thanks to Tamar Christina for his many hours of patient help.
| Without him, GHC's Windows support would be in significantly worse shape
| than it is.
| 
| Users of GHC should note that the CI issues we are struggling with *do
| not* affect compiled code. These bugs manifest only as (rare) failed
| compilations (particularly when building GHC itself); however, once
| compilation succeeds the program that results is correct and reliable.
| 
| Cheers,
| 
| - Ben

From ben at well-typed.com  Mon Feb  3 17:13:25 2020
From: ben at well-typed.com (Ben Gamari)
Date: Mon, 03 Feb 2020 12:13:25 -0500
Subject: Allowing Windows CI to fail
In-Reply-To: <VI1PR8303MB00936162EC5A32792EDB5564AD000@VI1PR8303MB0093.EURPRD83.prod.outlook.com>
References: <87wo93veuk.fsf@smart-cactus.org>
 <VI1PR8303MB00936162EC5A32792EDB5564AD000@VI1PR8303MB0093.EURPRD83.prod.outlook.com>
Message-ID: <87tv47vbki.fsf@smart-cactus.org>

Simon Peyton Jones via ghc-devs <ghc-devs at haskell.org> writes:

> Ben
>
> This sounds like a good decision to me, thanks.
>
> Is there a possibility to have a slow CI-on-windows job (not part of
> the "this must pass before merging" step), which will slowly, but
> reliably, fail if the Windows build fails. E.g. does it help to make
> the build be 100% sequential?
>
Sadly that won't fix the underlying problem.

> Or is there currently no way to build GHC at all on Windows in a way
> that won't fail? (That would be surprising to me. Until relatively
> recently I was *only* building on Windows.)
>
There is no way to build GHC that won't have a chance of failing. Indeed
Phyx and I also find it quite surprising how the probability of failure
seems to be higher now than in the past. However, we also both agree
that the status quo, when it works, works only accidentally (if the
win32 API documentation is to be believed).

What is especially intriguing is the fact that mingw32 gnu make should
also be affected by the same `exec` issue that we are struggling with,
does none of the job object headstands that we are doing, and yet
*appears* to be quite reliable. Tamar had a hypothesis for why this
might be that he will test when he has time.

Cheers,

- Ben

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 487 bytes
Desc: not available
URL: <http://mail.haskell.org/pipermail/ghc-devs/attachments/20200203/7da11e1e/attachment.sig>

From liuyiyun at terpmail.umd.edu  Mon Feb  3 18:27:31 2020
From: liuyiyun at terpmail.umd.edu (Yiyun Liu)
Date: Mon, 3 Feb 2020 13:27:31 -0500
Subject: Free dictionary variables in elaborated core expressions
In-Reply-To: <AM5PR8303MB0082F5EBC2A9A450F232A5B6AD000@AM5PR8303MB0082.EURPRD83.prod.outlook.com>
References: <9be2d40e-ad9c-9a1a-0f21-9ad0bad17dc5@terpmail.umd.edu>
 <AM5PR8303MB0082F5EBC2A9A450F232A5B6AD000@AM5PR8303MB0082.EURPRD83.prod.outlook.com>
Message-ID: <97f0e34c-e7ff-4474-7e90-86eb63802beb@terpmail.umd.edu>

Thanks for having the patience to read through my code! That's exactly 
what I was missing. I did print out the return value in my debug code at 
some point, but the prettyprinter only shows the suffix of the 
dictionary variable without the rhs so I totally missed it.

Also, I noticed that simplifyInteractive actually returns an empty bag 
in the (\x -> x + 1) example I gave, and I actually ended up using the 
EvBinds returned by simplifyInfer to add the let binding:

*    (_, _, evbs, residual, _) <- simplifyInfer tclvl**
**                                            infer_mode**
**                                            []**
**                                            [(fresh_it, res_ty)]**
**                                            lie**
**    evbs' <- perhaps_disable_default_warnings $ simplifyInteractive 
residual*

*    let full_expr = ***(mkHsDictLet (EvBinds evbs') (mkHsDictLet evbs 
tc_expr))**

*    zonkTopLExpr  full_expr
*

**

- Yiyun



On 2/3/20 4:18 AM, Simon Peyton Jones wrote:
>
> In your code (elabRnExpr) you have
>
> _ <- perhaps_disable_default_warnings $ simplifyInteractive residual
>
> You’ll notice that
>
> simplifyInteractive :: WantedConstraints -> TcM (Bag EvBind)
>
> So you are discarding the “evidence bindings” returned by 
> simplifyInteractive.  Those are precisely the bindings of the 
> dictionaries (dictionaries are a form of “evidence”) that you need.  
> Don’t discard them.
>
> Untested:
>
> ev_binds <- perhaps_disable_default_warnings $ simplifyInteractive 
> residual
>
> let full_expr = mkHsDictLet (EvBinds ev_binds) tc_expr
>
> zonkTopLExpr full_expr
>
> Simon
>
> *From:*ghc-devs <ghc-devs-bounces at haskell.org> *On Behalf Of *Yiyun Liu
> *Sent:* 03 February 2020 02:03
> *To:* ghc-devs at haskell.org
> *Cc:* James Parker <jp at jamesparker.me>
> *Subject:* Free dictionary variables in elaborated core expressions
>
> Hi ghc-devs,
>
> About 10 days ago, I made a thread about defining a function called 
> elaborateExpr which turns a string into a core expression 
> <https://github.com/yiyunliu/ghc-elaboration-test/blob/8f362ad92dc6601b4cb7e4c76f0a42bc6b64480f/src/Main.hs#L55> 
> within an interactive context. Now here's an unexpected behavior which 
> I'm not sure how to deal with.
>
> Given the expression:
>
> (\x -> x + 1) :: Int -> Int
>
> I expect to get something that looks like:
>
> \ (x :: Int) -> + @ Int GHC.Num.$fNumInt x (GHC.Types.I# 1#)
>
> where GHC.Num.$fNumInt is the exported dictionary.
>
> What I actually get is something like this:
>
> \ (x :: Int) -> + @ Int $dNum_someuniqueid x (GHC.Types.I# 1#)
>
> where $dNum_someuniqueid is a free dictionary variable within the 
> expression.
>
> I was confused by the occurrence of the free variable $dNum at first, 
> but after running the command: "ghc -ddump-ds-preopt somefile.hs" to 
> dump the core bindings, I found that the dictionary variables like 
> $dNum_ are actually local variables defined at the top-level.
>
> My objective is to inline those top-level dictionary definitions into 
> the core expression using let bindings, but it seems tricky since I'm 
> doing everything within an interactive context.  Calling getBindings 
> <https://hackage.haskell.org/package/ghc-8.6.5/docs/GHC.html#v:getBindings> 
> only gives me the expression I elaborated, but the dictionary is no 
> where to be found.
>
> Interestingly, when I turn on flags such as "DeferTypedHoles" or 
> "DeferOutOfScopeVariables", all the dictionaries are defined locally 
> in let bindings. However, I can't replicate that behavior even with 
> the flags on in the interactive context. How do I find the dictionaries?
>
> Thanks,
>
> - Yiyun
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.haskell.org/pipermail/ghc-devs/attachments/20200203/d185626d/attachment.html>

From compl.yue at icloud.com  Tue Feb  4 11:54:48 2020
From: compl.yue at icloud.com (Compl Yue)
Date: Tue, 4 Feb 2020 19:54:48 +0800
Subject: reporting my progress in building ghc 8 on latest smartos
Message-ID: <1c18e7b6-c8ba-a88d-9730-5577779425a9@icloud.com>

freenode server seems in trouble for this while, and I'd like to take 
this chance to report my progress to the mailing lists.

  I'm thankful to folks at #ghc and #smartos who helped me so much in 
this journey, especially bgamari, without his hints and bleeding edge 
patches this won't be possible at all.

And 
https://www.mail-archive.com/smartos-discuss at lists.smartos.org/msg05016.html 
provided the information to get me started at all.

So far what's done:

     :: installed 7.6.3 -> booted 7.10.3 -> booted 8.2.2 -> booted 8.4.4 
-> booted 8.6.5 -> booted 8.8.2

Notable issues as I remembered: (I don't remember the details clearly, 
anyway am to redo it all over again to have everything logged for reprod)

  * `gmake -i` is used to build 8.2.2 to workaround the stage1 ghc crash
    after target file generated

  * 8.6.5's ghci won't start

| [root at hswander /build/ghc8]# /opt/local/ghc8.6.5/bin/ghci|||
||| GHCi, version 8.6.5: http://www.haskell.org/ghc/  :? for help|||
||| ghc: loadArchive: Not an archive: `/usr/lib/iconv'|||
||| ghc: panic! (the 'impossible' happened)|||
|||   (GHC version 8.6.5 for x86_64-unknown-solaris2):|||
|||         loadArchive "/usr/lib/iconv": failed|||
||| Please report this as a GHC bug: 
http://www.haskell.org/ghc/reportabug|||

  *    worth to mention that I needed to put "SplitSections=YES" into
    mk/build.mk even for 8.8.2 or it'll fail, might be unexpected.

  And dtrace compiler on latest SmartOS don't support C++ style line 
comment, so I applied this patch to 8.8.2 
https://gitlab.haskell.org/snippets/1549


Meanwhile I'm starting a fresh smartos vm to repeat (and log) all steps 
to make sure others and future me can follow such a reproducible 
procedure to build later versions.

Cheers,

Compl


-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.haskell.org/pipermail/ghc-devs/attachments/20200204/71fb2199/attachment.html>

From ben at well-typed.com  Tue Feb  4 14:22:37 2020
From: ben at well-typed.com (Ben Gamari)
Date: Tue, 04 Feb 2020 09:22:37 -0500
Subject: reporting my progress in building ghc 8 on latest smartos
In-Reply-To: <1c18e7b6-c8ba-a88d-9730-5577779425a9@icloud.com>
References: <1c18e7b6-c8ba-a88d-9730-5577779425a9@icloud.com>
Message-ID: <9B1102E2-30D6-45F6-9BE8-5C3322F99B37@well-typed.com>

Thanks for writing this down!

The dtrace patch is certainly something that we can merge as-is. Do let us know if there are any other patches that you find are necessary.

- Ben 

On February 4, 2020 6:54:48 AM EST, Compl Yue via ghc-devs <ghc-devs at haskell.org> wrote:
>freenode server seems in trouble for this while, and I'd like to take 
>this chance to report my progress to the mailing lists.
>
>  I'm thankful to folks at #ghc and #smartos who helped me so much in 
>this journey, especially bgamari, without his hints and bleeding edge 
>patches this won't be possible at all.
>
>And 
>https://www.mail-archive.com/smartos-discuss at lists.smartos.org/msg05016.html
>
>provided the information to get me started at all.
>
>So far what's done:
>
>    :: installed 7.6.3 -> booted 7.10.3 -> booted 8.2.2 -> booted 8.4.4
>
>-> booted 8.6.5 -> booted 8.8.2
>
>Notable issues as I remembered: (I don't remember the details clearly, 
>anyway am to redo it all over again to have everything logged for
>reprod)
>
> * `gmake -i` is used to build 8.2.2 to workaround the stage1 ghc crash
>    after target file generated
>
>  * 8.6.5's ghci won't start
>
>| [root at hswander /build/ghc8]# /opt/local/ghc8.6.5/bin/ghci|||
>||| GHCi, version 8.6.5: http://www.haskell.org/ghc/  :? for help|||
>||| ghc: loadArchive: Not an archive: `/usr/lib/iconv'|||
>||| ghc: panic! (the 'impossible' happened)|||
>|||   (GHC version 8.6.5 for x86_64-unknown-solaris2):|||
>|||         loadArchive "/usr/lib/iconv": failed|||
>||| Please report this as a GHC bug: 
>http://www.haskell.org/ghc/reportabug|||
>
>  *    worth to mention that I needed to put "SplitSections=YES" into
>    mk/build.mk even for 8.8.2 or it'll fail, might be unexpected.
>
>  And dtrace compiler on latest SmartOS don't support C++ style line 
>comment, so I applied this patch to 8.8.2 
>https://gitlab.haskell.org/snippets/1549
>
>
>Meanwhile I'm starting a fresh smartos vm to repeat (and log) all steps
>
>to make sure others and future me can follow such a reproducible 
>procedure to build later versions.
>
>Cheers,
>
>Compl

-- 
Sent from my Android device with K-9 Mail. Please excuse my brevity.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.haskell.org/pipermail/ghc-devs/attachments/20200204/c9a6f784/attachment.html>

From marlowsd at gmail.com  Thu Feb  6 08:28:23 2020
From: marlowsd at gmail.com (Simon Marlow)
Date: Thu, 6 Feb 2020 08:28:23 +0000
Subject: Feasibility of native RTS support for continuations?
In-Reply-To: <B786EFAF-59E1-4F85-B593-1B02D8081645@gmail.com>
References: <C5F2E66E-E91C-4DB6-B88C-EE24C33B4B80@gmail.com>
 <VI1PR8303MB0093F04A891A00CEC953043AAD0A0@VI1PR8303MB0093.EURPRD83.prod.outlook.com>
 <D9E11FA1-2724-4835-B1F3-0BACDAC366E4@gmail.com>
 <VI1PR8303MB0093926BBCAAD612A3D76E5AAD050@VI1PR8303MB0093.EURPRD83.prod.outlook.com>
 <72859DAE-06D5-473F-BA92-AD6A40543C97@gmail.com>
 <CANsrgP+ueYtSMKkuzh4nFdgnPH-3N+Q+LcSHHyg2QEVeu7fOxw@mail.gmail.com>
 <B786EFAF-59E1-4F85-B593-1B02D8081645@gmail.com>
Message-ID: <CANsrgP+zGhZELmvdZyz-GUyMiYp=mLfZwFj2LZXzgDv4A-eTTA@mail.gmail.com>

On Sat, 1 Feb 2020 at 00:23, Alexis King <lexi.lambda at gmail.com> wrote:

> > On Jan 30, 2020, at 02:35, Simon Marlow <marlowsd at gmail.com> wrote:
>
> > Also you might want to optimise the implementation so that it doesn't
> actually tear down the stack as it copies it into the heap, so that you
> could avoid the need to copy it back from the heap again in shift#.
>
> I don’t fully understand this point — do you mean “avoid the need to copy
> it back on continuation application”? shift# just copies the stack slice to
> the heap, so it doesn’t need to copy it back.
>

The issue here is that raiseAsync is destructive - it *moves* the stack to
the heap, rather than copying it. So if you want to continue execution
where you left off, for shift#, you would have to copy it back onto the
stack again. That's the point I was trying to highlight here.

 Cheers
Simon


> If I was right, and you were referring to continuation application rather
> than shift#, I agree with you there. It seems as though it ought to be
> possible to represent the stack slice as a stack itself, so if the stack
> looks like
>
>     ┌───────────┐
>     │ RET_SMALL │
>     ├───────────┤
>     │ CATCH     │
>     ├───────────┤
>     │ RESET     │
>     ├───────────┤
>
> then the captured continuation could itself essentially be a stack like
>
>     ┌───────────┐
>     │ RET_SMALL │
>     ├───────────┤
>     │ CATCH     │
>     ├───────────┤
>     │ UNDERFLOW │
>     └───────────┘
>
> where restoring a continuation just copies the captured stack and updates
> its underflow frame to point at the top of the current stack. If the caller
> promises not to use the continuation again after applying it, the copying
> could be skipped entirely, and the captured stack could just become the new
> stack.
>
> However, I don’t understand enough about the way the RTS currently works
> to know if this is viable. For example, what if I write this:
>
>     reset (mask_ (shift f))
>
> Now presumably I want to ensure the masking state is restored when I
> invoke the continuation. But it can’t just be unconditionally restored,
> since if I write
>
>     mask_ (reset (shift f >>= g))
>
> then the mask frame isn’t included on the stack, so applying the
> continuation shouldn’t affect the masking state. Presumably this means a
> continuation restore can’t be as simple as copying the captured stack
> frames onto the current stack, since restoring certain frames affects other
> parts of the RTS state.
>
> (Tangentially, it seems like this particular example is not handled
> properly in the existing capture/restore code, as this comment in
> Exception.cmm notes:
>
>      NB. there's a bug in here.  If a thread is inside an
>      unsafePerformIO, and inside maskAsyncExceptions# (there is an
>      unmaskAsyncExceptions_ret on the stack), and it is blocked in an
>      interruptible operation, and it receives an exception, then the
>      unsafePerformIO thunk will be updated with a stack object
>      containing the unmaskAsyncExceptions_ret frame.  Later, when
>      someone else evaluates this thunk, the original masking state is
>      not restored.
>
> I think getting this right probably matters more if continuations are
> added, so that’s something else to worry about.)
>
> > So that's shift#. What about reset#? I expect it's something like
> `unsafeInterleaveIO`, that is it creates a thunk to name the continuation.
> You probably also want a `catch` in there, so that we don't tear down more
> of the stack than we need to.
>
> It would be nice to be able to capture slices of the stack that include
> catch frames, though theoretically it isn’t necessary — it would be
> possible to arrange for a continuation that wants to capture through a
> catch to do something like
>
>     reset (... (catch (reset ...) ...))
>
> instead, then call shift twice and compose the two continuations by hand
> (inserting another call to catch in between). I don’t know enough yet to
> understand all the tradeoffs involved, but I’ll see if it’s possible to get
> away with the userland emulation, since I figure the less code in the RTS
> the better!
>
> > Hope this is helpful.
>
> Very much so, thank you!
>
> > On Jan 30, 2020, at 10:31, Ben Gamari <ben at smart-cactus.org> wrote:
> >
> > For the record, runtime system captures the stack state in an AP_STACK
> > closure. This is done in rts/RaiseAsync.c:raiseAsync and some of this is
> > described in the comment attached to that function.
> >
> > As Simon PJ points out, this is all very tricky stuff, especially in a
> > concurrent context. If you make any changes in this area do be sure to
> > keep in mind the considerations described in Note [AP_STACKs must be
> > eagerly blackholed], which arose out of the very-nast #13615.
>
> Thank you for the pointers — I did manage to find raiseAsync, but I hadn’t
> seen that Note. I will try my best to be suitably wary, though I imagine
> I’m in far enough over my head that I don’t yet know half the things I need
> to be wary of. :)
>
> Alexis
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.haskell.org/pipermail/ghc-devs/attachments/20200206/e21c80ab/attachment.html>

From marlowsd at gmail.com  Thu Feb  6 08:33:55 2020
From: marlowsd at gmail.com (Simon Marlow)
Date: Thu, 6 Feb 2020 08:33:55 +0000
Subject: Feasibility of native RTS support for continuations?
In-Reply-To: <3BDD2CB6-5170-4C37-AB0E-24E7317AE1C7@gmail.com>
References: <C5F2E66E-E91C-4DB6-B88C-EE24C33B4B80@gmail.com>
 <VI1PR8303MB0093F04A891A00CEC953043AAD0A0@VI1PR8303MB0093.EURPRD83.prod.outlook.com>
 <D9E11FA1-2724-4835-B1F3-0BACDAC366E4@gmail.com>
 <VI1PR8303MB0093926BBCAAD612A3D76E5AAD050@VI1PR8303MB0093.EURPRD83.prod.outlook.com>
 <72859DAE-06D5-473F-BA92-AD6A40543C97@gmail.com>
 <CANsrgP+ueYtSMKkuzh4nFdgnPH-3N+Q+LcSHHyg2QEVeu7fOxw@mail.gmail.com>
 <B786EFAF-59E1-4F85-B593-1B02D8081645@gmail.com>
 <3BDD2CB6-5170-4C37-AB0E-24E7317AE1C7@gmail.com>
Message-ID: <CANsrgPLB=cxGpYz+kHqbFv=1WH10njaDj9yrCN4Z2QLmwKhKqQ@mail.gmail.com>

On Sun, 2 Feb 2020 at 04:26, Alexis King <lexi.lambda at gmail.com> wrote:

> I took a stab at implementing this today, using the “continuation is a
> stack” implementation strategy I described in my previous email. I
> haven’t tried very hard to break it yet, but this tiny test program
> works:
>
>     {-# LANGUAGE BangPatterns, BlockArguments, MagicHash,
>                  ScopedTypeVariables, UnboxedTuples #-}
>
>     import GHC.Prim
>     import GHC.Types
>
>     data Continuation a b = Continuation# (Continuation# RealWorld a b)
>
>     reset :: IO a -> IO a
>     reset (IO m) = IO (reset# m)
>
>     shift :: (Continuation a b -> IO b) -> IO a
>     shift f = IO (shift# \k -> let !(IO m) = f (Continuation# k) in m)
>
>     applyContinuation :: Continuation a b -> a -> IO b
>     applyContinuation (Continuation# k) a = IO (applyContinuation# k a)
>
>     main :: IO ()
>     main = do
>       ns <- reset do
>         n <- shift \(k :: Continuation Integer [Integer]) -> do
>           a <- applyContinuation k 2
>           b <- applyContinuation k 3
>           pure (a ++ b)
>         pure [n]
>       print ns
>
> The output of this program is [2, 3], as expected.
>

That's impressive!


>
> My implementation doesn’t share any code with raiseAsync. Currently, it
> isn’t very clever:
>
>     * reset# pushes a RET_SMALL frame with a well-known info pointer,
>       &stg_reset_frame_info.
>
>     * shift# walks the stack and copies it up to the nearest reset
>       frame. If the stack consists of several chunks, it only copies the
>       chunk that contains the reset frame, and it just repurposes the
>       other chunks as the continuation (since the stack is unwinding
>       anyway).
>
>     * applyContinuation# copies the captured stack and updates the
>       UNDERFLOW frames as needed to point to the current stack.
>
>     * I haven’t implemented it yet, but it would be straightforward to
>       implement an applyContinuationOneShot# operation that works like
>       applyContinuation#, but doesn’t actually copy anything and just
>       updates the UNDERFLOW frames in the captured stack itself.
>
> This seems to work in my very simple examples, but there are also things
> I know it doesn’t handle properly:
>
>     * It doesn’t make any attempt to handle modifications to the
>       interrupt masking state properly. The right thing to do here is
>       probably to look for mask/unmask frames on the stack during
>       unwinding and to stash that information somewhere.
>
>     * It doesn’t do anything special for UPDATE_FRAMEs, so if there’s an
>       UPDATE_FRAME that owns a blackhole on the stack, things will go
>       quite wrong.
>
>       I haven’t been worrying about this because I don’t think there
>       should ever be any update frames between shift# and reset#. In the
>       case of raiseAsync, the location of the “prompt” is well-defined:
>       it’s the update frame. But shift# captures up to an explicit
>       prompt, so using shift# when there’s an update frame on the stack
>       can surely only lead to nonsense... right?
>
>     * It doesn’t do anything special for STM frames, so trying to
>       capture a continuation through those will be similarly broken.
>

Yes, these are all the things that make raiseAsync tricky! You can either
copy what raiseAsync does (but be warned, it has taken a lot of iteration
to get right) or try to use raiseAsync and/or modify it to do what you want.

Cheers
Simon



> There are also probably bugs I don’t know about — I haven’t exercised
> the implementation very hard yet — but I’ll keep playing with it. If
> anyone is at all interested, I’ve pushed the code to a branch here:
>
>
> https://gitlab.haskell.org/lexi.lambda/ghc/compare/master...first-class-continuations
>
> My thanks again to everyone’s help!
>
> Alexis
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.haskell.org/pipermail/ghc-devs/attachments/20200206/d62b28b1/attachment.html>

From klebinger.andreas at gmx.at  Fri Feb  7 16:42:25 2020
From: klebinger.andreas at gmx.at (Andreas Klebinger)
Date: Fri, 7 Feb 2020 17:42:25 +0100
Subject: Request for two new MR-labels: "review-needed" and "changes-required"
Message-ID: <e2f708d4-60f3-9f61-5a0b-fddfcf961bf7@gmx.at>

Hello devs,

Recently in a MR discussion the topic came up of contributors being
frustrated about lack of reviews.

As a contributor and as a reviewer the lack of tools to find MR's in
need of attention also frustrated me often.

I suggest two new MR labels:

Label: "review-needed":  If a contributor has an MR which he considers
to be blocked on (lack of) review he can assign the label to clearly
signal this.
Label: "changes-required": This is the inverse - if a reviewer looked at
a patch and it's clear that the patch needs more work from the
contributor this label can signal this.

What are the benefits?

* As a reviewer I can look for MR's tagged with review-needed and can be
sure that time spent reviewing is not wasted.
* As a contributor I can mark a patch as changes-required if I think my
patch is ready to land, or if I need input on the design.
* Core maintainers can more easily identify patches which have been
stalled for a long time, making their review a priority.

The obvious downside is more labels.

Cheers
Andreas

From ben at well-typed.com  Fri Feb  7 21:32:16 2020
From: ben at well-typed.com (Ben Gamari)
Date: Fri, 07 Feb 2020 16:32:16 -0500
Subject: Request for two new MR-labels: "review-needed" and
 "changes-required"
In-Reply-To: <e2f708d4-60f3-9f61-5a0b-fddfcf961bf7@gmx.at>
References: <e2f708d4-60f3-9f61-5a0b-fddfcf961bf7@gmx.at>
Message-ID: <8736bmxew4.fsf@smart-cactus.org>

Andreas Klebinger <klebinger.andreas at gmx.at> writes:

> Hello devs,
>
> Recently in a MR discussion the topic came up of contributors being
> frustrated about lack of reviews.
>
> As a contributor and as a reviewer the lack of tools to find MR's in
> need of attention also frustrated me often.
>
> I suggest two new MR labels:
>
I believe we already have a solution to this problem, which is outlined
here [1]. I have created the necessary labels however haven't updated
our contributor documentation to reflect the new workflow.

Cheers,

- Ben


[1] https://docs.google.com/document/d/1-xPFaYdwhAlwOO0PLGwN3PxFLpwsYwjtWsVVALbOREc/edit
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 487 bytes
Desc: not available
URL: <http://mail.haskell.org/pipermail/ghc-devs/attachments/20200207/a95f3332/attachment.sig>

From carter.schonwald at gmail.com  Sat Feb  8 00:37:04 2020
From: carter.schonwald at gmail.com (Carter Schonwald)
Date: Fri, 7 Feb 2020 19:37:04 -0500
Subject: I'm concerned about the affine-types extension and its impact on ghc
 and haskell at large
Message-ID: <CAHYVw0whtfJAYzxq3zYySpZgmbAFqUktuFnNP7feTZxsnSFhTg@mail.gmail.com>

https://github.com/ghc-proposals/ghc-proposals/pull/111#issuecomment-583664586

https://www.reddit.com/r/haskell/comments/f0jigv/im_concerned_about_the_longterm_impact_of_the/

As current maintainer of vector, and a member of the CLC, both of which
roles really should be AFFIRMATIONAL stakeholders in this, I only see costs
and concerns.

As someone who's spent the past few years doing a LOT of modelling and
prototyping around marrying linear logic, formal methods, and functional
programming in an applied industrial setting, i should be an Affirmational
stakeholder. yet again I am not.

theres very real costs in the complexity and scope of impact that impact
EVERY single user and stakeholder of ghc and haskell. And I do not see any
concrete population that benefits.


cale even makes a very constructive and articulate point

> I don't know how much my opinion matters at this point, but I'd really
like to see the non-toy real-world use cases of this before I can even
consider thinking that it would be a good idea to merge to the main
compiler. It has such a huge potential impact on so many people in the
Haskell community:
>
>     * Library maintainers who start getting PRs that make "improvements"
to linearity while making their libraries harder to maintain because their
interface becomes more rigid, and harder to understand because the types
are more complicated.
>
>     * Beginners, or simply ordinary users of the language who have to pay
the mental overhead of living with the linear types and polymorphism
spreading everywhere as types are adjusted to make terms more usable in
places where one is concerned with linearity.
>
>     * Commercial users of the language who pay for the additional time
taken by all their employees waiting for the compiler to run, regardless of
whether or not they're using the extension. If Carter's 6-7% slowdown is
real even in cases where one doesn't care about the extension, I can
imagine wanting to make a fork without the extension. The compiler is
already 2 orders of magnitude slower than I'd like. If that weren't the
case, maybe 6-7% wouldn't be a huge deal. While ghci is often helpful at
shortening the feedback loop, it's not always a viable solution.
>
>
> But really, I just want to know how anyone would put this to practical
use in a real setting -- it needs to be _really_ compelling to make up for
the social cost. It can't be that hard for Tweag, or someone enthusiastic,
to live on a fork until they have a decent case study to show the world, so
we can say "okay, that's actually really cool, maybe we actually want that
everywhere".
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.haskell.org/pipermail/ghc-devs/attachments/20200207/9a2c8990/attachment.html>

From lexi.lambda at gmail.com  Mon Feb 10 08:10:26 2020
From: lexi.lambda at gmail.com (Alexis King)
Date: Mon, 10 Feb 2020 02:10:26 -0600
Subject: Feasibility of native RTS support for continuations?
In-Reply-To: <CANsrgPLB=cxGpYz+kHqbFv=1WH10njaDj9yrCN4Z2QLmwKhKqQ@mail.gmail.com>
References: <C5F2E66E-E91C-4DB6-B88C-EE24C33B4B80@gmail.com>
 <VI1PR8303MB0093F04A891A00CEC953043AAD0A0@VI1PR8303MB0093.EURPRD83.prod.outlook.com>
 <D9E11FA1-2724-4835-B1F3-0BACDAC366E4@gmail.com>
 <VI1PR8303MB0093926BBCAAD612A3D76E5AAD050@VI1PR8303MB0093.EURPRD83.prod.outlook.com>
 <72859DAE-06D5-473F-BA92-AD6A40543C97@gmail.com>
 <CANsrgP+ueYtSMKkuzh4nFdgnPH-3N+Q+LcSHHyg2QEVeu7fOxw@mail.gmail.com>
 <B786EFAF-59E1-4F85-B593-1B02D8081645@gmail.com>
 <3BDD2CB6-5170-4C37-AB0E-24E7317AE1C7@gmail.com>
 <CANsrgPLB=cxGpYz+kHqbFv=1WH10njaDj9yrCN4Z2QLmwKhKqQ@mail.gmail.com>
Message-ID: <EE113BEE-36C0-4FCB-B70B-6DE2B6A7A0E0@gmail.com>

> On Feb 6, 2020, at 02:28, Simon Marlow <marlowsd at gmail.com> wrote:
> 
> The issue here is that raiseAsync is destructive - it *moves* the stack to the heap, rather than copying it. So if you want to continue execution where you left off, for shift#, you would have to copy it back onto the stack again. That's the point I was trying to highlight here.

Ah, yes, I see what you mean! It happens that for my use case I actually do want to unwind the stack when I capture a continuation, so that isn’t a problem for me.

> Yes, these are all the things that make raiseAsync tricky! You can either copy what raiseAsync does (but be warned, it has taken a lot of iteration to get right) or try to use raiseAsync and/or modify it to do what you want.

My point was more that I’m unsure that shift# should handle most of those cases. For raiseAsync, it makes sense, since asynchronous interrupts can, by their nature, occur at any time, even during pure code. But my shift# operation lives in IO, and the intent is to only capture up to a reset# in the same state thread.

My justification for this is that if you could use shift# in pure code, it would be ill-defined what you’d even be capturing. Suppose you return a thunk containing a call to shift#. When the thunk is evaluated, you capture up to the nearest reset#… but who knows what that is now? This opens you up to all sorts of general badness.

Therefore, I don’t think there should ever be an UPDATE_FRAME in the captured continuation—if there is, it’s probably a bug. So unless someone can think of any valid use cases, I’ll make that more explicit by modifying the continuation-capturing code to add some assertions that those frames never appear in the captured stack.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.haskell.org/pipermail/ghc-devs/attachments/20200210/db95652a/attachment.html>

From marlowsd at gmail.com  Mon Feb 10 08:17:15 2020
From: marlowsd at gmail.com (Simon Marlow)
Date: Mon, 10 Feb 2020 08:17:15 +0000
Subject: Feasibility of native RTS support for continuations?
In-Reply-To: <EE113BEE-36C0-4FCB-B70B-6DE2B6A7A0E0@gmail.com>
References: <C5F2E66E-E91C-4DB6-B88C-EE24C33B4B80@gmail.com>
 <VI1PR8303MB0093F04A891A00CEC953043AAD0A0@VI1PR8303MB0093.EURPRD83.prod.outlook.com>
 <D9E11FA1-2724-4835-B1F3-0BACDAC366E4@gmail.com>
 <VI1PR8303MB0093926BBCAAD612A3D76E5AAD050@VI1PR8303MB0093.EURPRD83.prod.outlook.com>
 <72859DAE-06D5-473F-BA92-AD6A40543C97@gmail.com>
 <CANsrgP+ueYtSMKkuzh4nFdgnPH-3N+Q+LcSHHyg2QEVeu7fOxw@mail.gmail.com>
 <B786EFAF-59E1-4F85-B593-1B02D8081645@gmail.com>
 <3BDD2CB6-5170-4C37-AB0E-24E7317AE1C7@gmail.com>
 <CANsrgPLB=cxGpYz+kHqbFv=1WH10njaDj9yrCN4Z2QLmwKhKqQ@mail.gmail.com>
 <EE113BEE-36C0-4FCB-B70B-6DE2B6A7A0E0@gmail.com>
Message-ID: <CANsrgPL=Aj-59pyPk_WLen-dbBjJsNwu5iP3r59CqkH=yJ1jdA@mail.gmail.com>

On Mon, 10 Feb 2020 at 08:10, Alexis King <lexi.lambda at gmail.com> wrote:

> On Feb 6, 2020, at 02:28, Simon Marlow <marlowsd at gmail.com> wrote:
>
> The issue here is that raiseAsync is destructive - it *moves* the stack to
> the heap, rather than copying it. So if you want to continue execution
> where you left off, for shift#, you would have to copy it back onto the
> stack again. That's the point I was trying to highlight here.
>
>
> Ah, yes, I see what you mean! It happens that for my use case I actually
> do want to unwind the stack when I capture a continuation, so that isn’t a
> problem for me.
>
> Yes, these are all the things that make raiseAsync tricky! You can either
> copy what raiseAsync does (but be warned, it has taken a lot of iteration
> to get right) or try to use raiseAsync and/or modify it to do what you want.
>
>
> My point was more that I’m unsure that shift# *should* handle most of
> those cases. For raiseAsync, it makes sense, since asynchronous interrupts
> can, by their nature, occur at any time, even during pure code. But my
> shift# operation lives in IO, and the intent is to only capture up to a
> reset# in the same state thread.
>
> My justification for this is that if you could use shift# in pure code, it
> would be ill-defined what you’d even be capturing. Suppose you return a
> thunk containing a call to shift#. When the thunk is evaluated, you capture
> up to the nearest reset#… but who knows what that is now? This opens you up
> to all sorts of general badness.
>
> Therefore, I don’t think there should ever be an UPDATE_FRAME in the
> captured continuation—if there is, it’s probably a bug. So unless someone
> can think of any valid use cases, I’ll make that more explicit by modifying
> the continuation-capturing code to add some assertions that those frames
> never appear in the captured stack.
>

Let me just say "unsafePerformIO" :)  You probably want to at least ensure
that things don't crash in that case, even if you can't give a sensible
semantics to what actually happens. We have a similar situation with
unsafeIOToST - we can't tell you exactly what it does in general, except
that it doesn't crash (I hope!).

Cheers
Simon
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.haskell.org/pipermail/ghc-devs/attachments/20200210/ebbbad0c/attachment.html>

From marlowsd at gmail.com  Mon Feb 10 08:18:34 2020
From: marlowsd at gmail.com (Simon Marlow)
Date: Mon, 10 Feb 2020 08:18:34 +0000
Subject: Feasibility of native RTS support for continuations?
In-Reply-To: <CANsrgPL=Aj-59pyPk_WLen-dbBjJsNwu5iP3r59CqkH=yJ1jdA@mail.gmail.com>
References: <C5F2E66E-E91C-4DB6-B88C-EE24C33B4B80@gmail.com>
 <VI1PR8303MB0093F04A891A00CEC953043AAD0A0@VI1PR8303MB0093.EURPRD83.prod.outlook.com>
 <D9E11FA1-2724-4835-B1F3-0BACDAC366E4@gmail.com>
 <VI1PR8303MB0093926BBCAAD612A3D76E5AAD050@VI1PR8303MB0093.EURPRD83.prod.outlook.com>
 <72859DAE-06D5-473F-BA92-AD6A40543C97@gmail.com>
 <CANsrgP+ueYtSMKkuzh4nFdgnPH-3N+Q+LcSHHyg2QEVeu7fOxw@mail.gmail.com>
 <B786EFAF-59E1-4F85-B593-1B02D8081645@gmail.com>
 <3BDD2CB6-5170-4C37-AB0E-24E7317AE1C7@gmail.com>
 <CANsrgPLB=cxGpYz+kHqbFv=1WH10njaDj9yrCN4Z2QLmwKhKqQ@mail.gmail.com>
 <EE113BEE-36C0-4FCB-B70B-6DE2B6A7A0E0@gmail.com>
 <CANsrgPL=Aj-59pyPk_WLen-dbBjJsNwu5iP3r59CqkH=yJ1jdA@mail.gmail.com>
Message-ID: <CANsrgPJ+bNyTcrkpY1M_01sa5K7XXgwkD0t3-SCJ+6iKR6tZmQ@mail.gmail.com>

On Mon, 10 Feb 2020 at 08:17, Simon Marlow <marlowsd at gmail.com> wrote:

> On Mon, 10 Feb 2020 at 08:10, Alexis King <lexi.lambda at gmail.com> wrote:
>
>> On Feb 6, 2020, at 02:28, Simon Marlow <marlowsd at gmail.com> wrote:
>>
>> The issue here is that raiseAsync is destructive - it *moves* the stack
>> to the heap, rather than copying it. So if you want to continue execution
>> where you left off, for shift#, you would have to copy it back onto the
>> stack again. That's the point I was trying to highlight here.
>>
>>
>> Ah, yes, I see what you mean! It happens that for my use case I actually
>> do want to unwind the stack when I capture a continuation, so that isn’t a
>> problem for me.
>>
>> Yes, these are all the things that make raiseAsync tricky! You can either
>> copy what raiseAsync does (but be warned, it has taken a lot of iteration
>> to get right) or try to use raiseAsync and/or modify it to do what you want.
>>
>>
>> My point was more that I’m unsure that shift# *should* handle most of
>> those cases. For raiseAsync, it makes sense, since asynchronous interrupts
>> can, by their nature, occur at any time, even during pure code. But my
>> shift# operation lives in IO, and the intent is to only capture up to a
>> reset# in the same state thread.
>>
>> My justification for this is that if you could use shift# in pure code,
>> it would be ill-defined what you’d even be capturing. Suppose you return a
>> thunk containing a call to shift#. When the thunk is evaluated, you capture
>> up to the nearest reset#… but who knows what that is now? This opens you up
>> to all sorts of general badness.
>>
>> Therefore, I don’t think there should ever be an UPDATE_FRAME in the
>> captured continuation—if there is, it’s probably a bug. So unless someone
>> can think of any valid use cases, I’ll make that more explicit by modifying
>> the continuation-capturing code to add some assertions that those frames
>> never appear in the captured stack.
>>
>
> Let me just say "unsafePerformIO" :)  You probably want to at least ensure
> that things don't crash in that case, even if you can't give a sensible
> semantics to what actually happens. We have a similar situation with
> unsafeIOToST - we can't tell you exactly what it does in general, except
> that it doesn't crash (I hope!).
>

Typo - I meant unsafeIOToSTM here.


>
> Cheers
> Simon
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.haskell.org/pipermail/ghc-devs/attachments/20200210/578f0318/attachment.html>

From compl.yue at icloud.com  Mon Feb 10 08:25:33 2020
From: compl.yue at icloud.com (Compl Yue)
Date: Mon, 10 Feb 2020 16:25:33 +0800
Subject: GHC 8.8.2 + cabal-install 3.0.0 finally working on SmartOS x64
 20200117 (yet GHC 8.6.5 left less usable)
Message-ID: <f1d4b7bf-5398-ed9e-aeee-90b14a99ea1e@icloud.com>

Folks,

I'm glad to report that I finally get GHC 8.8.2 with cabal-install 
3.0.0.0 working on SmartOS x64, you can checkout detailed steps and all 
the patches applied at https://gitlab.haskell.org/complyue/smart-ghc8

The only pity so far, is that the built GHC 8.6.5 has its `ghci` not 
working with a strange iconv error, and failing building one of my 
Haskell applications. I myself am going to use 8.8.2 where possible  
from now on. But given not all great libs have been updated being 8.8 
compatible, I feel it still worth some further effort to get 8.6.5 working.

And regarding stack, as it's focused on managing binary distributions of 
GHC (build-from-src not working very well per my experience), it'll need 
a trusted binary repository for stack to work with on SmartOS. A private 
http server hosting the bindists should be straight forward, but I'm not 
interested to host a publicly available binary repository with those 
bindist tarballs digitally signed (or there're security concerns). If 
anyone is interested, I'd like to help as far as I can.

I would particular like to see Nix be ported to SmartOS/Illumos/Solaris, 
so I could contribute to nixpkgs instead, which should save more asses 
in tool dependency resolution, but seems that's not an option in near 
future.

I'm thankful to folks at #ghc and #smartos who helped me so much in this 
journey. Especially bgamari, without his hints and bleeding edge patches 
this won't be possible at all. And thanks to geekosaur for his 
professional insights into the linker situation on SmartOS, quick and 
precise identification of the problem from the error messages so 
confusing to me.

https://www.mail-archive.com/smartos-discuss at lists.smartos.org/msg05016.html 
provided the information to get me started at all, and thanks to 
Jonathan Perkin for confirming the bootstrapping path being a viable 
route in the first place.

Cheers,

Compl Yue


-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.haskell.org/pipermail/ghc-devs/attachments/20200210/a1dd3973/attachment.html>

From omeragacan at gmail.com  Mon Feb 10 08:45:48 2020
From: omeragacan at gmail.com (=?UTF-8?Q?=C3=96mer_Sinan_A=C4=9Facan?=)
Date: Mon, 10 Feb 2020 11:45:48 +0300
Subject: Missing symbols in ghc-stage2 executable
Message-ID: <CAMQQO3mVP_+rjKfxuN26kigHt7dtkaa_rNXCKuxQtdbXZGhcTw@mail.gmail.com>

Does anyone know why I get so little symbols in stage 2 compiler compared to
programs built with the same stage 2 compiler?

This is the build.mk I'm using

    BuildFlavour = quick

    ifneq "$(BuildFlavour)" ""
    include mk/flavours/$(BuildFlavour).mk
    endif

    GhcRtsHcOpts += -O0 -g3
    GhcStage2HcOpts += -debug -g5
    GhcLibHcOpts += -g5

    GhcThreaded = NO

    BUILD_PROF_LIBS = NO

    HADDOCK_DOCS = NO
    BUILD_SPHINX_HTML = NO
    BUILD_SPHINX_PDF = NO
    BUILD_MAN = NO

    STRIP_CMD = :

Symbols in stage 2 compiler built with this build.mk:

    $ nm /home/omer/haskell/ghc/inplace/lib/bin/ghc-stage2 | wc -l
    9202

If I build `main = return ()` with the same compiler:

    $ cat test.hs
    main = return ()

    $ /home/omer/haskell/ghc/inplace/bin/ghc-stage2 -g5 -O0 test -fforce-recomp
    [1 of 1] Compiling Main             ( test.hs, test.o )
    Linking test ...

    $ nm test | wc -l
    35266

So it seems like a lot of symbols in the libraries are missing in the stage 2
executable.

I tried setting `DYNAMIC_GHC_PROGRAMS = NO` but I got even less number of
symbols in the stage 2 compiler.

I also confirmed this using a debugger program I wrote which can find info table
labels of objects in the heap when I run it on `test.hs` above but not when I
run it on the stage 2 executable.

Ömer

From omeragacan at gmail.com  Mon Feb 10 08:57:27 2020
From: omeragacan at gmail.com (=?UTF-8?Q?=C3=96mer_Sinan_A=C4=9Facan?=)
Date: Mon, 10 Feb 2020 11:57:27 +0300
Subject: Missing symbols in ghc-stage2 executable
In-Reply-To: <CAMQQO3mVP_+rjKfxuN26kigHt7dtkaa_rNXCKuxQtdbXZGhcTw@mail.gmail.com>
References: <CAMQQO3mVP_+rjKfxuN26kigHt7dtkaa_rNXCKuxQtdbXZGhcTw@mail.gmail.com>
Message-ID: <CAMQQO3mvmPj_kGDJxyQf_dxpL43B=iZ2ie010t-aMLOvVXJHpw@mail.gmail.com>

If I build with hadrian using "quickest" setting I get a more reasonable number:

    $ nm ./_build/stage1/bin/ghc | wc -l
    102174

It'd be good to have these symbols in make-generated stage 2 executables too.

Ömer

Ömer Sinan Ağacan <omeragacan at gmail.com>, 10 Şub 2020 Pzt, 11:45
tarihinde şunu yazdı:
>
> Does anyone know why I get so little symbols in stage 2 compiler compared to
> programs built with the same stage 2 compiler?
>
> This is the build.mk I'm using
>
>     BuildFlavour = quick
>
>     ifneq "$(BuildFlavour)" ""
>     include mk/flavours/$(BuildFlavour).mk
>     endif
>
>     GhcRtsHcOpts += -O0 -g3
>     GhcStage2HcOpts += -debug -g5
>     GhcLibHcOpts += -g5
>
>     GhcThreaded = NO
>
>     BUILD_PROF_LIBS = NO
>
>     HADDOCK_DOCS = NO
>     BUILD_SPHINX_HTML = NO
>     BUILD_SPHINX_PDF = NO
>     BUILD_MAN = NO
>
>     STRIP_CMD = :
>
> Symbols in stage 2 compiler built with this build.mk:
>
>     $ nm /home/omer/haskell/ghc/inplace/lib/bin/ghc-stage2 | wc -l
>     9202
>
> If I build `main = return ()` with the same compiler:
>
>     $ cat test.hs
>     main = return ()
>
>     $ /home/omer/haskell/ghc/inplace/bin/ghc-stage2 -g5 -O0 test -fforce-recomp
>     [1 of 1] Compiling Main             ( test.hs, test.o )
>     Linking test ...
>
>     $ nm test | wc -l
>     35266
>
> So it seems like a lot of symbols in the libraries are missing in the stage 2
> executable.
>
> I tried setting `DYNAMIC_GHC_PROGRAMS = NO` but I got even less number of
> symbols in the stage 2 compiler.
>
> I also confirmed this using a debugger program I wrote which can find info table
> labels of objects in the heap when I run it on `test.hs` above but not when I
> run it on the stage 2 executable.
>
> Ömer

From omeragacan at gmail.com  Mon Feb 10 12:57:27 2020
From: omeragacan at gmail.com (=?UTF-8?Q?=C3=96mer_Sinan_A=C4=9Facan?=)
Date: Mon, 10 Feb 2020 15:57:27 +0300
Subject: Adding extra C compiler arguments when building with hadrian?
Message-ID: <CAMQQO3m=3HNEAsaOqFskJaUNdfpr1F4zZ0jfbYwYiJSYryz6fQ@mail.gmail.com>

In make build system I can add extra C compiler arguments to any flavor using
something like

    BuildFlavour = ...

    ifneq "$(BuildFlavour)" ""
    include mk/flavours/$(BuildFlavour).mk
    endif

    GhcRtsHcOpts += -O0 -g3

How do I do the same in hadrian when defining a flavor?

Thanks,

Ömer

From compl.yue at icloud.com  Mon Feb 10 13:00:58 2020
From: compl.yue at icloud.com (Compl Yue)
Date: Mon, 10 Feb 2020 21:00:58 +0800
Subject: reporting my progress in building ghc 8 on latest smartos
In-Reply-To: <9B1102E2-30D6-45F6-9BE8-5C3322F99B37@well-typed.com>
References: <1c18e7b6-c8ba-a88d-9730-5577779425a9@icloud.com>
 <9B1102E2-30D6-45F6-9BE8-5C3322F99B37@well-typed.com>
Message-ID: <5dc89a7f-7124-226e-d710-f9dd005a7da7@icloud.com>

The reproducible logs and patches are done as I reported in another 
email, and now I can confirm the dtrace patch is the only one eligible 
to be merged, all other patches for SmartOS builds only do backporting. FYI

https://gitlab.haskell.org/complyue/smart-ghc8/blob/master/ghc-arts/ghc-8.8.2_includes_HsFFI.h.patch

And thank you again for being excellently helpful!

Cheers,

Compl


On 2020/2/4 下午10:22, Ben Gamari wrote:
> Thanks for writing this down!
>
> The dtrace patch is certainly something that we can merge as-is. Do 
> let us know if there are any other patches that you find are necessary.
>
> - Ben
>
> On February 4, 2020 6:54:48 AM EST, Compl Yue via ghc-devs 
> <ghc-devs at haskell.org> wrote:
>
>     freenode server seems in trouble for this while, and I'd like to
>     take this chance to report my progress to the mailing lists.
>
>      I'm thankful to folks at #ghc and #smartos who helped me so much
>     in this journey, especially bgamari, without his hints and
>     bleeding edge patches this won't be possible at all.
>
>     And
>     https://www.mail-archive.com/smartos-discuss at lists.smartos.org/msg05016.html
>     provided the information to get me started at all.
>
>     So far what's done:
>
>         :: installed 7.6.3 -> booted 7.10.3 -> booted 8.2.2 -> booted
>     8.4.4 -> booted 8.6.5 -> booted 8.8.2
>
>     Notable issues as I remembered: (I don't remember the details
>     clearly, anyway am to redo it all over again to have everything
>     logged for reprod)
>
>       * `gmake -i` is used to build 8.2.2 to workaround the stage1 ghc
>         crash after target file generated
>
>       * 8.6.5's ghci won't start
>
>     | [root at hswander /build/ghc8]# /opt/local/ghc8.6.5/bin/ghci|||
>     ||| GHCi, version 8.6.5: http://www.haskell.org/ghc/ :? for help|||
>     ||| ghc: loadArchive: Not an archive: `/usr/lib/iconv'|||
>     ||| ghc: panic! (the 'impossible' happened)|||
>     |||   (GHC version 8.6.5 for x86_64-unknown-solaris2):|||
>     |||         loadArchive "/usr/lib/iconv": failed|||
>     ||| Please report this as a GHC bug:
>     http://www.haskell.org/ghc/reportabug|||
>
>       *   worth to mention that I needed to put "SplitSections=YES"
>         into mk/build.mk even for 8.8.2 or it'll fail, might be
>         unexpected.
>
>      And dtrace compiler on latest SmartOS don't support C++ style
>     line comment, so I applied this patch to 8.8.2
>     https://gitlab.haskell.org/snippets/1549
>
>
>     Meanwhile I'm starting a fresh smartos vm to repeat (and log) all
>     steps to make sure others and future me can follow such a
>     reproducible procedure to build later versions.
>
>     Cheers,
>
>     Compl
>
>
>
> -- 
> Sent from my Android device with K-9 Mail. Please excuse my brevity. 
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.haskell.org/pipermail/ghc-devs/attachments/20200210/7c361bae/attachment.html>

From gertjan.bottu at kuleuven.be  Mon Feb 10 08:18:51 2020
From: gertjan.bottu at kuleuven.be (Gert-Jan Bottu)
Date: Mon, 10 Feb 2020 09:18:51 +0100
Subject: GHC Test Failures
Message-ID: <baae0104-1fd8-ba4a-b9f5-e3c6c8304f23@kuleuven.be>

Hi all,

It's my first time working on ghc (currently implementing proposal 99 
[1]), and I have two issues with the ghc tests that I can't seem to 
figure out:

- Running hadrian/build.sh test -j on my machine causes 78 tests to fail 
with the following error:

	collect2: error: ld returned 1 exit status
	`cc' failed in phase `Linker'. (Exit code: 1)

This happens in both my wip branch and on master, both in devel2 and 
quickest flavour. I've tried hadrian/build.sh clean and even recloned 
the repo, but the errors are quite persistent. Running the hadrian tests 
on CI works fine though.

- On CI [2], all tests under build pass. Under full-build however, the 
hadrian tests pass, but a single make test fails :

	Wrong exit code for haddockHtmlTest()(expected 0 , actual 2 )
	Stdout ( haddockHtmlTest ):
	[1 of 6] Compiling Test.Haddock.Process ( /builds/Gertjan423/ghc/testsuite/../utils/haddock/haddock-test/src/Test/Haddock/Process.hs, Test/Haddock/Process.o )
	[2 of 6] Compiling Test.Haddock.Utils ( /builds/Gertjan423/ghc/testsuite/../utils/haddock/haddock-test/src/Test/Haddock/Utils.hs, Test/Haddock/Utils.o )
	[3 of 6] Compiling Test.Haddock.Config ( /builds/Gertjan423/ghc/testsuite/../utils/haddock/haddock-test/src/Test/Haddock/Config.hs, Test/Haddock/Config.o )
	[4 of 6] Compiling Test.Haddock     ( /builds/Gertjan423/ghc/testsuite/../utils/haddock/haddock-test/src/Test/Haddock.hs, Test/Haddock.o )
	[5 of 6] Compiling Test.Haddock.Xhtml ( /builds/Gertjan423/ghc/testsuite/../utils/haddock/haddock-test/src/Test/Haddock/Xhtml.hs, Test/Haddock/Xhtml.o )
	[6 of 6] Compiling Main             ( /builds/Gertjan423/ghc/testsuite/../utils/haddock/html-test/Main.hs, Main.o )
	Linking html-test ...
	Haddock version 2.22.0, (c) Simon Marlow 2006
	Ported to use the GHC API by David Waern 2006-2008
	8.11.0.20200207
	Generating documentation...
	Failed to run Haddock on test package ''
	Haddock output is at 'haddock-out.log'. This file can be set with `--haddock-stdout`.
	Testing output files...
	Some tests crashed.
	Makefile:19: recipe for target 'htmlTest' failed
	Stderr ( haddockHtmlTest ):
	Warning: Unknown/unsupported 'ghc' version detected (Cabal 3.2.0.0 supports
	'ghc' version < 8.10): /builds/Gertjan423/ghc/inplace/bin/ghc-stage2 is
	version 8.11.0.20200207
	make[2]: *** [htmlTest] Error 1
	*** unexpected failure for haddockHtmlTest(normal)

I did make some small Haddock changes, so I did most likely cause this 
behavior. I just find it strange that this error does not happen on my 
machine, and even on CI only occurs in the make tests (and only under 
linux, not under windows).

Any tips regarding what could cause this, and how to tackle debugging 
this would be greatly appreciated :)

Cheers,

Gert-Jan

[1] : 
https://github.com/ghc-proposals/ghc-proposals/blob/master/proposals/0099-explicit-specificity.rst

[2] : https://gitlab.haskell.org/Gertjan423/ghc/pipelines/15509

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.haskell.org/pipermail/ghc-devs/attachments/20200210/6dc8c4f7/attachment.html>

From dxld at darkboxed.org  Mon Feb 10 13:15:55 2020
From: dxld at darkboxed.org (Daniel =?iso-8859-1?Q?Gr=F6ber?=)
Date: Mon, 10 Feb 2020 14:15:55 +0100
Subject: Adding extra C compiler arguments when building with hadrian?
In-Reply-To: <CAMQQO3m=3HNEAsaOqFskJaUNdfpr1F4zZ0jfbYwYiJSYryz6fQ@mail.gmail.com>
References: <CAMQQO3m=3HNEAsaOqFskJaUNdfpr1F4zZ0jfbYwYiJSYryz6fQ@mail.gmail.com>
Message-ID: <20200210131555.GA3548@janet.servers.dxld.at>

Hi,

On Mon, Feb 10, 2020 at 03:57:27PM +0300, Ömer Sinan Ağacan wrote:
> In make build system I can add extra C compiler arguments to any flavor using
> something like
> 
>     BuildFlavour = ...
> 
>     ifneq "$(BuildFlavour)" ""
>     include mk/flavours/$(BuildFlavour).mk
>     endif
> 
>     GhcRtsHcOpts += -O0 -g3
> 
> How do I do the same in hadrian when defining a flavor?

I was experimenting with building ghc with -fsanitize=address and have
this in hadrin/UserSettings.hs still:

    userFlavour = defaultFlavour
      { name = "user"
      , args = mconcat
        [ builder Cc ? arg "-fsanitize=address"
        , builder (Ghc CompileCWithGhc) ? arg "-optc -fsanitize=address"
        , builder (Ghc LinkHs) ? arg "-optl -fsanitize=address"
        ]
      }

I'm not sure when (builder Cc) is used vs. (Ghc CompileCWithGhc) so I
just add flags to both ;)

I think you can restrict these args to just the rts package with
something like `package rts ? ...` on each line, also `import Package`
to get the `rts` binder.

--Daniel
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 833 bytes
Desc: not available
URL: <http://mail.haskell.org/pipermail/ghc-devs/attachments/20200210/37ad25c9/attachment.sig>

From rae at richarde.dev  Mon Feb 10 14:29:25 2020
From: rae at richarde.dev (Richard Eisenberg)
Date: Mon, 10 Feb 2020 14:29:25 +0000
Subject: Request for two new MR-labels: "review-needed" and
 "changes-required"
In-Reply-To: <8736bmxew4.fsf@smart-cactus.org>
References: <e2f708d4-60f3-9f61-5a0b-fddfcf961bf7@gmx.at>
 <8736bmxew4.fsf@smart-cactus.org>
Message-ID: <8D92E8FD-8112-4638-9C0E-14B16E16B3AE@richarde.dev>

Hi Ben,

Thanks for reminding me about that document. Has it been officially ratified as our procedure of record? My guess is "no". What's it waiting on? That document has been around for quite some time now.

Thanks,
Richard

> On Feb 7, 2020, at 9:32 PM, Ben Gamari <ben at well-typed.com> wrote:
> 
> Andreas Klebinger <klebinger.andreas at gmx.at> writes:
> 
>> Hello devs,
>> 
>> Recently in a MR discussion the topic came up of contributors being
>> frustrated about lack of reviews.
>> 
>> As a contributor and as a reviewer the lack of tools to find MR's in
>> need of attention also frustrated me often.
>> 
>> I suggest two new MR labels:
>> 
> I believe we already have a solution to this problem, which is outlined
> here [1]. I have created the necessary labels however haven't updated
> our contributor documentation to reflect the new workflow.
> 
> Cheers,
> 
> - Ben
> 
> 
> [1] https://docs.google.com/document/d/1-xPFaYdwhAlwOO0PLGwN3PxFLpwsYwjtWsVVALbOREc/edit
> _______________________________________________
> ghc-devs mailing list
> ghc-devs at haskell.org
> http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs


From omeragacan at gmail.com  Mon Feb 10 17:13:20 2020
From: omeragacan at gmail.com (=?UTF-8?Q?=C3=96mer_Sinan_A=C4=9Facan?=)
Date: Mon, 10 Feb 2020 20:13:20 +0300
Subject: Adding extra C compiler arguments when building with hadrian?
In-Reply-To: <20200210131555.GA3548@janet.servers.dxld.at>
References: <CAMQQO3m=3HNEAsaOqFskJaUNdfpr1F4zZ0jfbYwYiJSYryz6fQ@mail.gmail.com>
 <20200210131555.GA3548@janet.servers.dxld.at>
Message-ID: <CAMQQO3mya7aNO5YJBTMhoBu7U0BoptC3kap73gNY5=Wn8AMWQA@mail.gmail.com>

Thanks. I'm currently editing the existing "quickest" flavor instead of adding
my own flavor. So far the changes are

    diff --git a/hadrian/src/Settings/Flavours/Quickest.hs
b/hadrian/src/Settings/Flavours/Quickest.hs
    index c0fd72764f..b0f490d35a 100644
    --- a/hadrian/src/Settings/Flavours/Quickest.hs
    +++ b/hadrian/src/Settings/Flavours/Quickest.hs
    @@ -4,15 +4,20 @@ import Expression
     import Flavour
     import {-# SOURCE #-} Settings.Default
     import Settings.Flavours.Common
    +import Packages (rts)

     -- Please update doc/flavours.md when changing this file.
     quickestFlavour :: Flavour
     quickestFlavour = defaultFlavour
         { name        = "quickest"
    -    , args        = defaultBuilderArgs <> quickestArgs <>
defaultPackageArgs
    +    , args        = defaultBuilderArgs <> quickestArgs <>
defaultPackageArgs <> dbgArgs
         , libraryWays = pure [vanilla]
    -    , rtsWays     = pure [vanilla, threaded]
    -    , dynamicGhcPrograms = return False }
    +    , rtsWays     = pure [vanilla, threaded, debug, threadedDebug,
    +                          dynamic, threadedDynamic, debugDynamic,
threadedDebugDynamic]
    +    , dynamicGhcPrograms = return False
    +    , ghcDebugged = True
    +    , ghcThreaded = False
    +    }

     quickestArgs :: Args
     quickestArgs = sourceArgs SourceArgs
    @@ -23,3 +28,6 @@ quickestArgs = sourceArgs SourceArgs
         , hsLibrary  = mempty
         , hsCompiler = stage0 ? arg "-O"
         , hsGhc      = stage0 ? arg "-O" }
    +
    +dbgArgs :: Args
    +dbgArgs = builder Cc ? package rts ? arg "-O0" <> arg "-g3"

This fails with

    Error when running Shake build system:
      at action, called at src/Rules.hs:71:19 in main:Rules
      at need, called at src/Rules.hs:93:5 in main:Rules
    * Depends on: _build/stage1/bin/runghc
      at need, called at src/Rules/Register.hs:73:5 in main:Rules.Register
    * Depends on:
_build/stage1/lib/x86_64-linux-ghc-8.11.0.20200206/libHSrts-1.0-ghc8.11.0.20200206.so
      at error, called at
src/Development/Shake/Internal/Rules/File.hs:179:58 in
shake-0.18.3-593067565aafb558d09b4352b8abc327d8911a39a0e9abab2804b002b1ae536e:Development.Shake.Internal.Rules.File
    * Raised the exception:
    Error, rule finished running but did not produce file:
      _build/stage1/lib/x86_64-linux-ghc-8.11.0.20200206/libHSrts-1.0-ghc8.11.0.20200206.so

If I set `dynamicGhcPrograms = return False` then it works, but it makes things
harder for me so I'd like to dynamically link the RTS.

I thought the `dynamic` way should be generating the requested .so file, no idea
why it doesn't. Is this a bug in hadrian?

Ömer

Daniel Gröber <dxld at darkboxed.org>, 10 Şub 2020 Pzt, 16:17 tarihinde şunu yazdı:
>
> Hi,
>
> On Mon, Feb 10, 2020 at 03:57:27PM +0300, Ömer Sinan Ağacan wrote:
> > In make build system I can add extra C compiler arguments to any flavor using
> > something like
> >
> >     BuildFlavour = ...
> >
> >     ifneq "$(BuildFlavour)" ""
> >     include mk/flavours/$(BuildFlavour).mk
> >     endif
> >
> >     GhcRtsHcOpts += -O0 -g3
> >
> > How do I do the same in hadrian when defining a flavor?
>
> I was experimenting with building ghc with -fsanitize=address and have
> this in hadrin/UserSettings.hs still:
>
>     userFlavour = defaultFlavour
>       { name = "user"
>       , args = mconcat
>         [ builder Cc ? arg "-fsanitize=address"
>         , builder (Ghc CompileCWithGhc) ? arg "-optc -fsanitize=address"
>         , builder (Ghc LinkHs) ? arg "-optl -fsanitize=address"
>         ]
>       }
>
> I'm not sure when (builder Cc) is used vs. (Ghc CompileCWithGhc) so I
> just add flags to both ;)
>
> I think you can restrict these args to just the rts package with
> something like `package rts ? ...` on each line, also `import Package`
> to get the `rts` binder.
>
> --Daniel

From matthewtpickering at gmail.com  Mon Feb 10 17:20:42 2020
From: matthewtpickering at gmail.com (Matthew Pickering)
Date: Mon, 10 Feb 2020 17:20:42 +0000
Subject: Adding extra C compiler arguments when building with hadrian?
In-Reply-To: <CAMQQO3mya7aNO5YJBTMhoBu7U0BoptC3kap73gNY5=Wn8AMWQA@mail.gmail.com>
References: <CAMQQO3m=3HNEAsaOqFskJaUNdfpr1F4zZ0jfbYwYiJSYryz6fQ@mail.gmail.com>
 <20200210131555.GA3548@janet.servers.dxld.at>
 <CAMQQO3mya7aNO5YJBTMhoBu7U0BoptC3kap73gNY5=Wn8AMWQA@mail.gmail.com>
Message-ID: <CALuQ0m9_G5afu+f=pvAwkV4ZdQsLL8gbmq6Ru+zOhTO+n2efcw@mail.gmail.com>

You need to modify the `libraryWays` as well I think. Other flavours
build both by default.

Matt

On Mon, Feb 10, 2020 at 5:14 PM Ömer Sinan Ağacan <omeragacan at gmail.com> wrote:
>
> Thanks. I'm currently editing the existing "quickest" flavor instead of adding
> my own flavor. So far the changes are
>
>     diff --git a/hadrian/src/Settings/Flavours/Quickest.hs
> b/hadrian/src/Settings/Flavours/Quickest.hs
>     index c0fd72764f..b0f490d35a 100644
>     --- a/hadrian/src/Settings/Flavours/Quickest.hs
>     +++ b/hadrian/src/Settings/Flavours/Quickest.hs
>     @@ -4,15 +4,20 @@ import Expression
>      import Flavour
>      import {-# SOURCE #-} Settings.Default
>      import Settings.Flavours.Common
>     +import Packages (rts)
>
>      -- Please update doc/flavours.md when changing this file.
>      quickestFlavour :: Flavour
>      quickestFlavour = defaultFlavour
>          { name        = "quickest"
>     -    , args        = defaultBuilderArgs <> quickestArgs <>
> defaultPackageArgs
>     +    , args        = defaultBuilderArgs <> quickestArgs <>
> defaultPackageArgs <> dbgArgs
>          , libraryWays = pure [vanilla]
>     -    , rtsWays     = pure [vanilla, threaded]
>     -    , dynamicGhcPrograms = return False }
>     +    , rtsWays     = pure [vanilla, threaded, debug, threadedDebug,
>     +                          dynamic, threadedDynamic, debugDynamic,
> threadedDebugDynamic]
>     +    , dynamicGhcPrograms = return False
>     +    , ghcDebugged = True
>     +    , ghcThreaded = False
>     +    }
>
>      quickestArgs :: Args
>      quickestArgs = sourceArgs SourceArgs
>     @@ -23,3 +28,6 @@ quickestArgs = sourceArgs SourceArgs
>          , hsLibrary  = mempty
>          , hsCompiler = stage0 ? arg "-O"
>          , hsGhc      = stage0 ? arg "-O" }
>     +
>     +dbgArgs :: Args
>     +dbgArgs = builder Cc ? package rts ? arg "-O0" <> arg "-g3"
>
> This fails with
>
>     Error when running Shake build system:
>       at action, called at src/Rules.hs:71:19 in main:Rules
>       at need, called at src/Rules.hs:93:5 in main:Rules
>     * Depends on: _build/stage1/bin/runghc
>       at need, called at src/Rules/Register.hs:73:5 in main:Rules.Register
>     * Depends on:
> _build/stage1/lib/x86_64-linux-ghc-8.11.0.20200206/libHSrts-1.0-ghc8.11.0.20200206.so
>       at error, called at
> src/Development/Shake/Internal/Rules/File.hs:179:58 in
> shake-0.18.3-593067565aafb558d09b4352b8abc327d8911a39a0e9abab2804b002b1ae536e:Development.Shake.Internal.Rules.File
>     * Raised the exception:
>     Error, rule finished running but did not produce file:
>       _build/stage1/lib/x86_64-linux-ghc-8.11.0.20200206/libHSrts-1.0-ghc8.11.0.20200206.so
>
> If I set `dynamicGhcPrograms = return False` then it works, but it makes things
> harder for me so I'd like to dynamically link the RTS.
>
> I thought the `dynamic` way should be generating the requested .so file, no idea
> why it doesn't. Is this a bug in hadrian?
>
> Ömer
>
> Daniel Gröber <dxld at darkboxed.org>, 10 Şub 2020 Pzt, 16:17 tarihinde şunu yazdı:
> >
> > Hi,
> >
> > On Mon, Feb 10, 2020 at 03:57:27PM +0300, Ömer Sinan Ağacan wrote:
> > > In make build system I can add extra C compiler arguments to any flavor using
> > > something like
> > >
> > >     BuildFlavour = ...
> > >
> > >     ifneq "$(BuildFlavour)" ""
> > >     include mk/flavours/$(BuildFlavour).mk
> > >     endif
> > >
> > >     GhcRtsHcOpts += -O0 -g3
> > >
> > > How do I do the same in hadrian when defining a flavor?
> >
> > I was experimenting with building ghc with -fsanitize=address and have
> > this in hadrin/UserSettings.hs still:
> >
> >     userFlavour = defaultFlavour
> >       { name = "user"
> >       , args = mconcat
> >         [ builder Cc ? arg "-fsanitize=address"
> >         , builder (Ghc CompileCWithGhc) ? arg "-optc -fsanitize=address"
> >         , builder (Ghc LinkHs) ? arg "-optl -fsanitize=address"
> >         ]
> >       }
> >
> > I'm not sure when (builder Cc) is used vs. (Ghc CompileCWithGhc) so I
> > just add flags to both ;)
> >
> > I think you can restrict these args to just the rts package with
> > something like `package rts ? ...` on each line, also `import Package`
> > to get the `rts` binder.
> >
> > --Daniel
> _______________________________________________
> ghc-devs mailing list
> ghc-devs at haskell.org
> http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs

From omeragacan at gmail.com  Mon Feb 10 18:07:14 2020
From: omeragacan at gmail.com (=?UTF-8?Q?=C3=96mer_Sinan_A=C4=9Facan?=)
Date: Mon, 10 Feb 2020 21:07:14 +0300
Subject: Adding extra C compiler arguments when building with hadrian?
In-Reply-To: <CALuQ0m9_G5afu+f=pvAwkV4ZdQsLL8gbmq6Ru+zOhTO+n2efcw@mail.gmail.com>
References: <CAMQQO3m=3HNEAsaOqFskJaUNdfpr1F4zZ0jfbYwYiJSYryz6fQ@mail.gmail.com>
 <20200210131555.GA3548@janet.servers.dxld.at>
 <CAMQQO3mya7aNO5YJBTMhoBu7U0BoptC3kap73gNY5=Wn8AMWQA@mail.gmail.com>
 <CALuQ0m9_G5afu+f=pvAwkV4ZdQsLL8gbmq6Ru+zOhTO+n2efcw@mail.gmail.com>
Message-ID: <CAMQQO3nQh=t7Ju8asfOSUG2-bM2bXqnE-e=sT8Yu6cKbiVcYow@mail.gmail.com>

That worked, thanks.

I just discovered that dynamicGhcPrograms is not for whether to link stage 2
executable statically or dynamically (with the RTS and Haskell libs). Is there a
way to tell Hadrian to dynamically link stage 2 executable?

Ömer

Matthew Pickering <matthewtpickering at gmail.com>, 10 Şub 2020 Pzt,
20:20 tarihinde şunu yazdı:
>
> You need to modify the `libraryWays` as well I think. Other flavours
> build both by default.
>
> Matt
>
> On Mon, Feb 10, 2020 at 5:14 PM Ömer Sinan Ağacan <omeragacan at gmail.com> wrote:
> >
> > Thanks. I'm currently editing the existing "quickest" flavor instead of adding
> > my own flavor. So far the changes are
> >
> >     diff --git a/hadrian/src/Settings/Flavours/Quickest.hs
> > b/hadrian/src/Settings/Flavours/Quickest.hs
> >     index c0fd72764f..b0f490d35a 100644
> >     --- a/hadrian/src/Settings/Flavours/Quickest.hs
> >     +++ b/hadrian/src/Settings/Flavours/Quickest.hs
> >     @@ -4,15 +4,20 @@ import Expression
> >      import Flavour
> >      import {-# SOURCE #-} Settings.Default
> >      import Settings.Flavours.Common
> >     +import Packages (rts)
> >
> >      -- Please update doc/flavours.md when changing this file.
> >      quickestFlavour :: Flavour
> >      quickestFlavour = defaultFlavour
> >          { name        = "quickest"
> >     -    , args        = defaultBuilderArgs <> quickestArgs <>
> > defaultPackageArgs
> >     +    , args        = defaultBuilderArgs <> quickestArgs <>
> > defaultPackageArgs <> dbgArgs
> >          , libraryWays = pure [vanilla]
> >     -    , rtsWays     = pure [vanilla, threaded]
> >     -    , dynamicGhcPrograms = return False }
> >     +    , rtsWays     = pure [vanilla, threaded, debug, threadedDebug,
> >     +                          dynamic, threadedDynamic, debugDynamic,
> > threadedDebugDynamic]
> >     +    , dynamicGhcPrograms = return False
> >     +    , ghcDebugged = True
> >     +    , ghcThreaded = False
> >     +    }
> >
> >      quickestArgs :: Args
> >      quickestArgs = sourceArgs SourceArgs
> >     @@ -23,3 +28,6 @@ quickestArgs = sourceArgs SourceArgs
> >          , hsLibrary  = mempty
> >          , hsCompiler = stage0 ? arg "-O"
> >          , hsGhc      = stage0 ? arg "-O" }
> >     +
> >     +dbgArgs :: Args
> >     +dbgArgs = builder Cc ? package rts ? arg "-O0" <> arg "-g3"
> >
> > This fails with
> >
> >     Error when running Shake build system:
> >       at action, called at src/Rules.hs:71:19 in main:Rules
> >       at need, called at src/Rules.hs:93:5 in main:Rules
> >     * Depends on: _build/stage1/bin/runghc
> >       at need, called at src/Rules/Register.hs:73:5 in main:Rules.Register
> >     * Depends on:
> > _build/stage1/lib/x86_64-linux-ghc-8.11.0.20200206/libHSrts-1.0-ghc8.11.0.20200206.so
> >       at error, called at
> > src/Development/Shake/Internal/Rules/File.hs:179:58 in
> > shake-0.18.3-593067565aafb558d09b4352b8abc327d8911a39a0e9abab2804b002b1ae536e:Development.Shake.Internal.Rules.File
> >     * Raised the exception:
> >     Error, rule finished running but did not produce file:
> >       _build/stage1/lib/x86_64-linux-ghc-8.11.0.20200206/libHSrts-1.0-ghc8.11.0.20200206.so
> >
> > If I set `dynamicGhcPrograms = return False` then it works, but it makes things
> > harder for me so I'd like to dynamically link the RTS.
> >
> > I thought the `dynamic` way should be generating the requested .so file, no idea
> > why it doesn't. Is this a bug in hadrian?
> >
> > Ömer
> >
> > Daniel Gröber <dxld at darkboxed.org>, 10 Şub 2020 Pzt, 16:17 tarihinde şunu yazdı:
> > >
> > > Hi,
> > >
> > > On Mon, Feb 10, 2020 at 03:57:27PM +0300, Ömer Sinan Ağacan wrote:
> > > > In make build system I can add extra C compiler arguments to any flavor using
> > > > something like
> > > >
> > > >     BuildFlavour = ...
> > > >
> > > >     ifneq "$(BuildFlavour)" ""
> > > >     include mk/flavours/$(BuildFlavour).mk
> > > >     endif
> > > >
> > > >     GhcRtsHcOpts += -O0 -g3
> > > >
> > > > How do I do the same in hadrian when defining a flavor?
> > >
> > > I was experimenting with building ghc with -fsanitize=address and have
> > > this in hadrin/UserSettings.hs still:
> > >
> > >     userFlavour = defaultFlavour
> > >       { name = "user"
> > >       , args = mconcat
> > >         [ builder Cc ? arg "-fsanitize=address"
> > >         , builder (Ghc CompileCWithGhc) ? arg "-optc -fsanitize=address"
> > >         , builder (Ghc LinkHs) ? arg "-optl -fsanitize=address"
> > >         ]
> > >       }
> > >
> > > I'm not sure when (builder Cc) is used vs. (Ghc CompileCWithGhc) so I
> > > just add flags to both ;)
> > >
> > > I think you can restrict these args to just the rts package with
> > > something like `package rts ? ...` on each line, also `import Package`
> > > to get the `rts` binder.
> > >
> > > --Daniel
> > _______________________________________________
> > ghc-devs mailing list
> > ghc-devs at haskell.org
> > http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs

From csaba.hruska at gmail.com  Mon Feb 10 20:48:19 2020
From: csaba.hruska at gmail.com (Csaba Hruska)
Date: Mon, 10 Feb 2020 21:48:19 +0100
Subject: ADT arguments and Levity Polymorphism
Message-ID: <CANjChEpnmY+3VfMUYQhVX1EVDFz-X4cH5jX0CkAT5f87CfWgaw@mail.gmail.com>

Hello,

Are heap stored ADT's arguments always levity (representation) monomorphic?

*(I guess they are otherwise the info table would not be a constant value
instead it would be a function that calculates the representation depending
on some runtime value.)*

Regards,
Csaba Hruska
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.haskell.org/pipermail/ghc-devs/attachments/20200210/7103a73c/attachment.html>

From rae at richarde.dev  Mon Feb 10 21:30:34 2020
From: rae at richarde.dev (Richard Eisenberg)
Date: Mon, 10 Feb 2020 21:30:34 +0000
Subject: ADT arguments and Levity Polymorphism
In-Reply-To: <CANjChEpnmY+3VfMUYQhVX1EVDFz-X4cH5jX0CkAT5f87CfWgaw@mail.gmail.com>
References: <CANjChEpnmY+3VfMUYQhVX1EVDFz-X4cH5jX0CkAT5f87CfWgaw@mail.gmail.com>
Message-ID: <6688B196-FA9D-489D-BD87-BFF2539B76C1@richarde.dev>



> On Feb 10, 2020, at 8:48 PM, Csaba Hruska <csaba.hruska at gmail.com> wrote:
> 
> Hello,
> 
> Are heap stored ADT's arguments always levity (representation) monomorphic?

Yes.

> (I guess they are otherwise the info table would not be a constant value instead it would be a function that calculates the representation depending on some runtime value.)

Yes.

Happy to expand, but it seems you have the number on this one.

Richard

> 
> Regards,
> Csaba Hruska
> _______________________________________________
> ghc-devs mailing list
> ghc-devs at haskell.org
> http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.haskell.org/pipermail/ghc-devs/attachments/20200210/e2c442cb/attachment.html>

From csaba.hruska at gmail.com  Mon Feb 10 21:37:15 2020
From: csaba.hruska at gmail.com (Csaba Hruska)
Date: Mon, 10 Feb 2020 22:37:15 +0100
Subject: ADT arguments and Levity Polymorphism
In-Reply-To: <6688B196-FA9D-489D-BD87-BFF2539B76C1@richarde.dev>
References: <CANjChEpnmY+3VfMUYQhVX1EVDFz-X4cH5jX0CkAT5f87CfWgaw@mail.gmail.com>
 <6688B196-FA9D-489D-BD87-BFF2539B76C1@richarde.dev>
Message-ID: <CANjChEpYRvjjj5_BbcU7M2UqoXU2C4V+kL4Ujntra7WsH44w4A@mail.gmail.com>

Great, thanks!

On Mon, Feb 10, 2020 at 10:30 PM Richard Eisenberg <rae at richarde.dev> wrote:

>
>
> On Feb 10, 2020, at 8:48 PM, Csaba Hruska <csaba.hruska at gmail.com> wrote:
>
> Hello,
>
> Are heap stored ADT's arguments always levity (representation) monomorphic?
>
>
> Yes.
>
>
> *(I guess they are otherwise the info table would not be a constant value
> instead it would be a function that calculates the representation depending
> on some runtime value.)*
>
>
> Yes.
>
> Happy to expand, but it seems you have the number on this one.
>
> Richard
>
>
> Regards,
> Csaba Hruska
> _______________________________________________
> ghc-devs mailing list
> ghc-devs at haskell.org
> http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs
>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.haskell.org/pipermail/ghc-devs/attachments/20200210/f9821342/attachment.html>

From csaba.hruska at gmail.com  Wed Feb 12 18:57:37 2020
From: csaba.hruska at gmail.com (Csaba Hruska)
Date: Wed, 12 Feb 2020 19:57:37 +0100
Subject: DataCon tag value convention
Message-ID: <CANjChErMjoZ-PzDa2x8yhdiC9Z49FJExVoR0UhesLL5Te3sSYQ@mail.gmail.com>

Hello,

In theory could GHC codegen work if every data constructor in the whole
program  have a globally unique tag value instead of starting from 1 for
each algebraic data type?
Would this break any GHC design decision?

Regards,
Csaba
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.haskell.org/pipermail/ghc-devs/attachments/20200212/d6341d73/attachment.html>

From sgraf1337 at gmail.com  Wed Feb 12 19:34:14 2020
From: sgraf1337 at gmail.com (Sebastian Graf)
Date: Wed, 12 Feb 2020 20:34:14 +0100
Subject: DataCon tag value convention
In-Reply-To: <CANjChErMjoZ-PzDa2x8yhdiC9Z49FJExVoR0UhesLL5Te3sSYQ@mail.gmail.com>
References: <CANjChErMjoZ-PzDa2x8yhdiC9Z49FJExVoR0UhesLL5Te3sSYQ@mail.gmail.com>
Message-ID: <CAAS+=P8BpBHqs6Xr6Z70otVSozjdXj=76yyK7jwJDfqN+AVOfw@mail.gmail.com>

You probably couldn't do pointer tagging
<https://gitlab.haskell.org/ghc/ghc/wikis/commentary/rts/haskell-execution/pointer-tagging>
anymore, which is probably a substantial performance loss.

Am Mi., 12. Feb. 2020 um 19:58 Uhr schrieb Csaba Hruska <
csaba.hruska at gmail.com>:

> Hello,
>
> In theory could GHC codegen work if every data constructor in the whole
> program  have a globally unique tag value instead of starting from 1 for
> each algebraic data type?
> Would this break any GHC design decision?
>
> Regards,
> Csaba
> _______________________________________________
> ghc-devs mailing list
> ghc-devs at haskell.org
> http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.haskell.org/pipermail/ghc-devs/attachments/20200212/58bce4b8/attachment.html>

From csaba.hruska at gmail.com  Wed Feb 12 19:48:52 2020
From: csaba.hruska at gmail.com (Csaba Hruska)
Date: Wed, 12 Feb 2020 20:48:52 +0100
Subject: DataCon tag value convention
In-Reply-To: <CAAS+=P8BpBHqs6Xr6Z70otVSozjdXj=76yyK7jwJDfqN+AVOfw@mail.gmail.com>
References: <CANjChErMjoZ-PzDa2x8yhdiC9Z49FJExVoR0UhesLL5Te3sSYQ@mail.gmail.com>
 <CAAS+=P8BpBHqs6Xr6Z70otVSozjdXj=76yyK7jwJDfqN+AVOfw@mail.gmail.com>
Message-ID: <CANjChEotAJd86MznPBUdtRbaL_BFL4texWa+y9SH1pU1SXPn0g@mail.gmail.com>

Ok, fair enough.
Would it break anything else? (i.e. coercions)

On Wed, Feb 12, 2020 at 8:34 PM Sebastian Graf <sgraf1337 at gmail.com> wrote:

> You probably couldn't do pointer tagging
> <https://gitlab.haskell.org/ghc/ghc/wikis/commentary/rts/haskell-execution/pointer-tagging>
> anymore, which is probably a substantial performance loss.
>
> Am Mi., 12. Feb. 2020 um 19:58 Uhr schrieb Csaba Hruska <
> csaba.hruska at gmail.com>:
>
>> Hello,
>>
>> In theory could GHC codegen work if every data constructor in the whole
>> program  have a globally unique tag value instead of starting from 1 for
>> each algebraic data type?
>> Would this break any GHC design decision?
>>
>> Regards,
>> Csaba
>> _______________________________________________
>> ghc-devs mailing list
>> ghc-devs at haskell.org
>> http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.haskell.org/pipermail/ghc-devs/attachments/20200212/878136f6/attachment.html>

From vladislav at serokell.io  Wed Feb 12 20:01:07 2020
From: vladislav at serokell.io (Vladislav Zavialov)
Date: Wed, 12 Feb 2020 23:01:07 +0300
Subject: DataCon tag value convention
In-Reply-To: <CANjChErMjoZ-PzDa2x8yhdiC9Z49FJExVoR0UhesLL5Te3sSYQ@mail.gmail.com>
References: <CANjChErMjoZ-PzDa2x8yhdiC9Z49FJExVoR0UhesLL5Te3sSYQ@mail.gmail.com>
Message-ID: <5F213791-7DE6-47C7-89FD-2A85B8BA589B@serokell.io>

The globally unique tag for data constructors already exists, it’s a pointer to the StgInfoTable. You can observe it using getClosureRaw from GHC.HeapView

Example:

Prelude GHC.HeapView> getClosureRaw Nothing
(0x0000000107fd8e38,[4429024840,4429024752],[])

Here, the globally unique tag is 0x0000000107fd8e38.

Note that newtype constructors do not get their own tag because newtypes guarantee that they do not change the underlying representation of data.

I’ve discovered the existence of such a tag only recently (Alexander Vershilov pointed it out to me), so I cannot say if it’s a reliable way to identify data constructors. For example, it is definitely not stable across several runs of the same binary. However, within a single run, it seems to work.

- Vlad

> On 12 Feb 2020, at 21:57, Csaba Hruska <csaba.hruska at gmail.com> wrote:
> 
> Hello,
> 
> In theory could GHC codegen work if every data constructor in the whole program  have a globally unique tag value instead of starting from 1 for each algebraic data type?
> Would this break any GHC design decision?
> 
> Regards,
> Csaba
> _______________________________________________
> ghc-devs mailing list
> ghc-devs at haskell.org
> http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs


From csaba.hruska at gmail.com  Wed Feb 12 20:05:57 2020
From: csaba.hruska at gmail.com (Csaba Hruska)
Date: Wed, 12 Feb 2020 21:05:57 +0100
Subject: DataCon tag value convention
In-Reply-To: <5F213791-7DE6-47C7-89FD-2A85B8BA589B@serokell.io>
References: <CANjChErMjoZ-PzDa2x8yhdiC9Z49FJExVoR0UhesLL5Te3sSYQ@mail.gmail.com>
 <5F213791-7DE6-47C7-89FD-2A85B8BA589B@serokell.io>
Message-ID: <CANjChEq_HUFvheeiCsSqr8wDT_yhAjwJGJQMCrWRLvyzB7iX+g@mail.gmail.com>

Ok, but what I am curious about if the tag in the info table could be
globally unique?

On Wed, Feb 12, 2020 at 9:01 PM Vladislav Zavialov <vladislav at serokell.io>
wrote:

> The globally unique tag for data constructors already exists, it’s a
> pointer to the StgInfoTable. You can observe it using getClosureRaw from
> GHC.HeapView
>
> Example:
>
> Prelude GHC.HeapView> getClosureRaw Nothing
> (0x0000000107fd8e38,[4429024840,4429024752],[])
>
> Here, the globally unique tag is 0x0000000107fd8e38.
>
> Note that newtype constructors do not get their own tag because newtypes
> guarantee that they do not change the underlying representation of data.
>
> I’ve discovered the existence of such a tag only recently (Alexander
> Vershilov pointed it out to me), so I cannot say if it’s a reliable way to
> identify data constructors. For example, it is definitely not stable across
> several runs of the same binary. However, within a single run, it seems to
> work.
>
> - Vlad
>
> > On 12 Feb 2020, at 21:57, Csaba Hruska <csaba.hruska at gmail.com> wrote:
> >
> > Hello,
> >
> > In theory could GHC codegen work if every data constructor in the whole
> program  have a globally unique tag value instead of starting from 1 for
> each algebraic data type?
> > Would this break any GHC design decision?
> >
> > Regards,
> > Csaba
> > _______________________________________________
> > ghc-devs mailing list
> > ghc-devs at haskell.org
> > http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.haskell.org/pipermail/ghc-devs/attachments/20200212/2043d345/attachment.html>

From lexi.lambda at gmail.com  Thu Feb 13 01:52:00 2020
From: lexi.lambda at gmail.com (Alexis King)
Date: Wed, 12 Feb 2020 19:52:00 -0600
Subject: Feasibility of native RTS support for continuations?
In-Reply-To: <CANsrgPJ+bNyTcrkpY1M_01sa5K7XXgwkD0t3-SCJ+6iKR6tZmQ@mail.gmail.com>
References: <C5F2E66E-E91C-4DB6-B88C-EE24C33B4B80@gmail.com>
 <VI1PR8303MB0093F04A891A00CEC953043AAD0A0@VI1PR8303MB0093.EURPRD83.prod.outlook.com>
 <D9E11FA1-2724-4835-B1F3-0BACDAC366E4@gmail.com>
 <VI1PR8303MB0093926BBCAAD612A3D76E5AAD050@VI1PR8303MB0093.EURPRD83.prod.outlook.com>
 <72859DAE-06D5-473F-BA92-AD6A40543C97@gmail.com>
 <CANsrgP+ueYtSMKkuzh4nFdgnPH-3N+Q+LcSHHyg2QEVeu7fOxw@mail.gmail.com>
 <B786EFAF-59E1-4F85-B593-1B02D8081645@gmail.com>
 <3BDD2CB6-5170-4C37-AB0E-24E7317AE1C7@gmail.com>
 <CANsrgPLB=cxGpYz+kHqbFv=1WH10njaDj9yrCN4Z2QLmwKhKqQ@mail.gmail.com>
 <EE113BEE-36C0-4FCB-B70B-6DE2B6A7A0E0@gmail.com>
 <CANsrgPL=Aj-59pyPk_WLen-dbBjJsNwu5iP3r59CqkH=yJ1jdA@mail.gmail.com>
 <CANsrgPJ+bNyTcrkpY1M_01sa5K7XXgwkD0t3-SCJ+6iKR6tZmQ@mail.gmail.com>
Message-ID: <64005FF6-1AE4-4423-BE66-C10AE54E684B@gmail.com>

> On Feb 10, 2020, at 02:18, Simon Marlow <marlowsd at gmail.com> wrote:
> 
>> On Mon, 10 Feb 2020 at 08:17, Simon Marlow <marlowsd at gmail.com> wrote:
>> 
>> Let me just say "unsafePerformIO" :)  You probably want to at least ensure that things don't crash in that case, even if you can't give a sensible semantics to what actually happens. We have a similar situation with unsafeIOToST - we can't tell you exactly what it does in general, except that it doesn't crash (I hope!).
> 
> Typo - I meant unsafeIOToSTM here.

I’ve been thinking about this. At first I figured you were probably right, and I decided I’d switch to a more raiseAsync-like approach. But once I started trying to implement it, I became less convinced it makes sense.

As its name implies, an AP_STACK is sort of like a saturated application, where the stack itself is the “function.” Extending the analogy, a continuation would be a PAP_STACK, since the stack is awaiting a value. This difference is significant. Suppose you write:

    let x = 1 + unsafePerformIO (shift f >>= g) in ...

If you force x, the stack will unwind to the nearest reset. When you unwind past x’s UPDATE_FRAME, you can’t replace the blackhole with an AP_STACK, since the captured slice of the stack represents the expression

    \m -> 1 + unsafePerformIO (m >>= g)

which is a function, not a thunk. The only logical interpretation of this situation is that x is a thunk quite like

    let y = 1 + error "bang"

except that it doesn’t just abort to the enclosing reset frame when forced, it actually composes the captured continuation with the upper frames of the current stack and passes the composed continuation to f to resume computation. That’s an awful lot of trouble given the resulting semantics is going to be unpredictable, anyway!

I should be clear that I do not intend shift/reset to have any safe interface in IO directly. Even if you could make it type safe, it would break all kinds of code that manages resources using bracket. Rather, I have built a library that defines a totally separate Eff monad, and that monad uses the primops directly, wrapped in a safe interface. It isn’t possible for a user to screw things up using unsafePerformIO because there is no way to call shift from IO (unless you wrap shift# in IO yourself, but then I think you can be expected to know what you’re getting yourself into). So without mucking about with GHC.Exts, you still can’t get segfaults.

Alexis

From compl.yue at icloud.com  Thu Feb 13 07:58:37 2020
From: compl.yue at icloud.com (Compl Yue)
Date: Thu, 13 Feb 2020 15:58:37 +0800
Subject: GHC 8.6.5 now works on SmartOS - Was: GHC 8.8.2 + cabal-install 3.0.0
 finally working on SmartOS x64 20200117 (yet GHC 8.6.5 left less
 usable)
In-Reply-To: <f1d4b7bf-5398-ed9e-aeee-90b14a99ea1e@icloud.com>
References: <f1d4b7bf-5398-ed9e-aeee-90b14a99ea1e@icloud.com>
Message-ID: <2f5513d5-4631-ee0c-d70e-877d9f428f28@icloud.com>

I verified that the problem in GHC 8.6.5 can be solved by backporting 
merge request 87, a patch as well as the build steps have been updated 
at https://gitlab.haskell.org/complyue/smart-ghc8, enjoy!

On 2020/2/10 下午4:25, Compl Yue via ghc-devs wrote:
>
> Folks,
>
> I'm glad to report that I finally get GHC 8.8.2 with cabal-install 
> 3.0.0.0 working on SmartOS x64, you can checkout detailed steps and 
> all the patches applied at https://gitlab.haskell.org/complyue/smart-ghc8
>
> The only pity so far, is that the built GHC 8.6.5 has its `ghci` not 
> working with a strange iconv error, and failing building one of my 
> Haskell applications. I myself am going to use 8.8.2 where possible  
> from now on. But given not all great libs have been updated being 8.8 
> compatible, I feel it still worth some further effort to get 8.6.5 
> working.
>
> And regarding stack, as it's focused on managing binary distributions 
> of GHC (build-from-src not working very well per my experience), it'll 
> need a trusted binary repository for stack to work with on SmartOS. A 
> private http server hosting the bindists should be straight forward, 
> but I'm not interested to host a publicly available binary repository 
> with those bindist tarballs digitally signed (or there're security 
> concerns). If anyone is interested, I'd like to help as far as I can.
>
> I would particular like to see Nix be ported to 
> SmartOS/Illumos/Solaris, so I could contribute to nixpkgs instead, 
> which should save more asses in tool dependency resolution, but seems 
> that's not an option in near future.
>
> I'm thankful to folks at #ghc and #smartos who helped me so much in 
> this journey. Especially bgamari, without his hints and bleeding edge 
> patches this won't be possible at all. And thanks to geekosaur for his 
> professional insights into the linker situation on SmartOS, quick and 
> precise identification of the problem from the error messages so 
> confusing to me.
>
> https://www.mail-archive.com/smartos-discuss at lists.smartos.org/msg05016.html 
> provided the information to get me started at all, and thanks to 
> Jonathan Perkin for confirming the bootstrapping path being a viable 
> route in the first place.
>
> Cheers,
>
> Compl Yue
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.haskell.org/pipermail/ghc-devs/attachments/20200213/5c27fec9/attachment.html>

From omeragacan at gmail.com  Fri Feb 14 11:49:21 2020
From: omeragacan at gmail.com (=?UTF-8?Q?=C3=96mer_Sinan_A=C4=9Facan?=)
Date: Fri, 14 Feb 2020 14:49:21 +0300
Subject: Confused about PAP object layout
Message-ID: <CAMQQO3=D=2y3BGQJzrk3VHy9=LV_EdYjyVnAALMuq9N=KgANbA@mail.gmail.com>

Hi Simon,

In this code: (slightly simplified)

    StgPtr
    scavenge_PAP (StgPAP *pap)
    {
        evacuate(&pap->fun);
        return scavenge_PAP_payload (pap->fun, pap->payload, pap->n_args);
    }

    StgPtr
    scavenge_PAP_payload (StgClosure *fun, StgClosure **payload, StgWord size)
    {
        const StgFunInfoTable *fun_info =
get_fun_itbl(UNTAG_CONST_CLOSURE(fun));
        StgPtr p = (StgPtr)payload;

        switch (fun_info->f.fun_type) {
        case ARG_GEN_BIG:
            scavenge_large_bitmap(p, GET_FUN_LARGE_BITMAP(fun_info), size);
            p += size;
            break;
        ...
        }
        return p;
    }

Here the `size` argument in `scavenge_PAP_payload` is the number of arguments
applied to the function in `pap->fun`. But when scavenging the function's bitmap
we're using this number as the size of the bitmap which doesn't make sense to
me, because I think size of the function's bitmap and size of the PAP's payload
may be different.

Or in other words I may have the same function used in many PAPs with different
n_args, but that'd be buggy if this code is correct.

I haven't checked every single place where we build a PAP but for example the
`NEW_PAP` macro uses the argument's (another PAP) function directly, without
making any bitmap-related changes, but bumps n_args by one. If the code above is
right, then this new PAP will be scavenged incorrectly.

Am I missing anything?

Thanks,

Ömer

From marlowsd at gmail.com  Fri Feb 14 15:08:02 2020
From: marlowsd at gmail.com (Simon Marlow)
Date: Fri, 14 Feb 2020 15:08:02 +0000
Subject: Confused about PAP object layout
In-Reply-To: <CAMQQO3=D=2y3BGQJzrk3VHy9=LV_EdYjyVnAALMuq9N=KgANbA@mail.gmail.com>
References: <CAMQQO3=D=2y3BGQJzrk3VHy9=LV_EdYjyVnAALMuq9N=KgANbA@mail.gmail.com>
Message-ID: <CANsrgP+1Ve0ahkndAoSOASU4cm-K92hJ=SXG+OpYKHX4bEuO-g@mail.gmail.com>

On Fri, 14 Feb 2020 at 11:49, Ömer Sinan Ağacan <omeragacan at gmail.com>
wrote:

> Hi Simon,
>
> In this code: (slightly simplified)
>
>     StgPtr
>     scavenge_PAP (StgPAP *pap)
>     {
>         evacuate(&pap->fun);
>         return scavenge_PAP_payload (pap->fun, pap->payload, pap->n_args);
>     }
>
>     StgPtr
>     scavenge_PAP_payload (StgClosure *fun, StgClosure **payload, StgWord
> size)
>     {
>         const StgFunInfoTable *fun_info =
> get_fun_itbl(UNTAG_CONST_CLOSURE(fun));
>         StgPtr p = (StgPtr)payload;
>
>         switch (fun_info->f.fun_type) {
>         case ARG_GEN_BIG:
>             scavenge_large_bitmap(p, GET_FUN_LARGE_BITMAP(fun_info), size);
>             p += size;
>             break;
>         ...
>         }
>         return p;
>     }
>
> Here the `size` argument in `scavenge_PAP_payload` is the number of
> arguments
> applied to the function in `pap->fun`. But when scavenging the function's
> bitmap
> we're using this number as the size of the bitmap which doesn't make sense
> to
> me, because I think size of the function's bitmap and size of the PAP's
> payload
> may be different.
>

"size" is an argument to scavenge_PAP_payload(), and when we call it we
pass pap->n_args as the value, not the bitmap's size.

Does that help?

Cheers
Simon



>
> Or in other words I may have the same function used in many PAPs with
> different
> n_args, but that'd be buggy if this code is correct.
>
> I haven't checked every single place where we build a PAP but for example
> the
> `NEW_PAP` macro uses the argument's (another PAP) function directly,
> without
> making any bitmap-related changes, but bumps n_args by one. If the code
> above is
> right, then this new PAP will be scavenged incorrectly.
>
> Am I missing anything?
>
> Thanks,
>
> Ömer
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.haskell.org/pipermail/ghc-devs/attachments/20200214/68f97358/attachment.html>

From omeragacan at gmail.com  Fri Feb 14 15:30:18 2020
From: omeragacan at gmail.com (=?UTF-8?Q?=C3=96mer_Sinan_A=C4=9Facan?=)
Date: Fri, 14 Feb 2020 18:30:18 +0300
Subject: Confused about PAP object layout
In-Reply-To: <CANsrgP+1Ve0ahkndAoSOASU4cm-K92hJ=SXG+OpYKHX4bEuO-g@mail.gmail.com>
References: <CAMQQO3=D=2y3BGQJzrk3VHy9=LV_EdYjyVnAALMuq9N=KgANbA@mail.gmail.com>
 <CANsrgP+1Ve0ahkndAoSOASU4cm-K92hJ=SXG+OpYKHX4bEuO-g@mail.gmail.com>
Message-ID: <CAMQQO3kq0HxLShqdiPesnGMVnkn-AEkY3SXN8DCSSA2hxrQ7vQ@mail.gmail.com>

Right, I think that's the problem. We then pass the same "size" to
scavenge_large_bitmap as the size of the bitmap. So we assume size of the bitmap
is pap->n_args.

So the call stack is

- scavenge_PAP, calls scavenge_PAP_payload with pap->n_args as "size"
- scavenge_PAP_payload, calls scavenge_large_bitmap with "size" (== pap->n_args)
  as the bitmap's size

Is this expected?

Ömer

Simon Marlow <marlowsd at gmail.com>, 14 Şub 2020 Cum, 18:08 tarihinde şunu yazdı:
>
> On Fri, 14 Feb 2020 at 11:49, Ömer Sinan Ağacan <omeragacan at gmail.com> wrote:
>>
>> Hi Simon,
>>
>> In this code: (slightly simplified)
>>
>>     StgPtr
>>     scavenge_PAP (StgPAP *pap)
>>     {
>>         evacuate(&pap->fun);
>>         return scavenge_PAP_payload (pap->fun, pap->payload, pap->n_args);
>>     }
>>
>>     StgPtr
>>     scavenge_PAP_payload (StgClosure *fun, StgClosure **payload, StgWord size)
>>     {
>>         const StgFunInfoTable *fun_info =
>> get_fun_itbl(UNTAG_CONST_CLOSURE(fun));
>>         StgPtr p = (StgPtr)payload;
>>
>>         switch (fun_info->f.fun_type) {
>>         case ARG_GEN_BIG:
>>             scavenge_large_bitmap(p, GET_FUN_LARGE_BITMAP(fun_info), size);
>>             p += size;
>>             break;
>>         ...
>>         }
>>         return p;
>>     }
>>
>> Here the `size` argument in `scavenge_PAP_payload` is the number of arguments
>> applied to the function in `pap->fun`. But when scavenging the function's bitmap
>> we're using this number as the size of the bitmap which doesn't make sense to
>> me, because I think size of the function's bitmap and size of the PAP's payload
>> may be different.
>
>
> "size" is an argument to scavenge_PAP_payload(), and when we call it we pass pap->n_args as the value, not the bitmap's size.
>
> Does that help?
>
> Cheers
> Simon
>
>
>>
>>
>> Or in other words I may have the same function used in many PAPs with different
>> n_args, but that'd be buggy if this code is correct.
>>
>> I haven't checked every single place where we build a PAP but for example the
>> `NEW_PAP` macro uses the argument's (another PAP) function directly, without
>> making any bitmap-related changes, but bumps n_args by one. If the code above is
>> right, then this new PAP will be scavenged incorrectly.
>>
>> Am I missing anything?
>>
>> Thanks,
>>
>> Ömer

From lexi.lambda at gmail.com  Fri Feb 14 15:43:40 2020
From: lexi.lambda at gmail.com (Alexis King)
Date: Fri, 14 Feb 2020 09:43:40 -0600
Subject: Confused about PAP object layout
In-Reply-To: <CAMQQO3kq0HxLShqdiPesnGMVnkn-AEkY3SXN8DCSSA2hxrQ7vQ@mail.gmail.com>
References: <CAMQQO3=D=2y3BGQJzrk3VHy9=LV_EdYjyVnAALMuq9N=KgANbA@mail.gmail.com>
 <CANsrgP+1Ve0ahkndAoSOASU4cm-K92hJ=SXG+OpYKHX4bEuO-g@mail.gmail.com>
 <CAMQQO3kq0HxLShqdiPesnGMVnkn-AEkY3SXN8DCSSA2hxrQ7vQ@mail.gmail.com>
Message-ID: <FF5887E8-5266-473B-951D-4DDD02A349E8@gmail.com>

Disclaimer: I am not an expert. But I happened to have been looking at this code just yesterday, so I’ll try to answer to check my understanding. :)

Fundamentally, a PAP is not fully-saturated, so the number of arguments in its payload may be smaller than the information contained in the function’s bitmap. scavenge_large_bitmap calls walk_large_bitmap, which uses the bitmap as a “ruler” to guide the traversal, lining up each element in the payload to information in the bitmap. But the traversal only actually walks a payload of the specified size, so if there’s less information in the payload than there is information in the bitmap, the traversal will just terminate early.

> On Feb 14, 2020, at 09:30, Ömer Sinan Ağacan <omeragacan at gmail.com> wrote:
> 
> Right, I think that's the problem. We then pass the same "size" to
> scavenge_large_bitmap as the size of the bitmap. So we assume size of the bitmap
> is pap->n_args.
> 
> So the call stack is
> 
> - scavenge_PAP, calls scavenge_PAP_payload with pap->n_args as "size"
> - scavenge_PAP_payload, calls scavenge_large_bitmap with "size" (== pap->n_args)
>  as the bitmap's size
> 
> Is this expected?
> 
> Ömer

From ben at smart-cactus.org  Fri Feb 14 15:46:36 2020
From: ben at smart-cactus.org (Ben Gamari)
Date: Fri, 14 Feb 2020 10:46:36 -0500
Subject: GHC 8.6.5 now works on SmartOS - Was: GHC 8.8.2 + cabal-install
 3.0.0 finally working on SmartOS x64 20200117 (yet GHC 8.6.5 left less
 usable)
In-Reply-To: <2f5513d5-4631-ee0c-d70e-877d9f428f28@icloud.com>
References: <f1d4b7bf-5398-ed9e-aeee-90b14a99ea1e@icloud.com>
 <2f5513d5-4631-ee0c-d70e-877d9f428f28@icloud.com>
Message-ID: <874kvtw4rs.fsf@smart-cactus.org>

Compl Yue via ghc-devs <ghc-devs at haskell.org> writes:

> I verified that the problem in GHC 8.6.5 can be solved by backporting 
> merge request 87, a patch as well as the build steps have been updated 
> at https://gitlab.haskell.org/complyue/smart-ghc8, enjoy!
>
Great work figuring out the issue! Thanks for your work on this.

Cheers,

- Ben
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 487 bytes
Desc: not available
URL: <http://mail.haskell.org/pipermail/ghc-devs/attachments/20200214/11492030/attachment.sig>

From ben at smart-cactus.org  Fri Feb 14 15:53:13 2020
From: ben at smart-cactus.org (Ben Gamari)
Date: Fri, 14 Feb 2020 10:53:13 -0500
Subject: Confused about PAP object layout
In-Reply-To: <CAMQQO3kq0HxLShqdiPesnGMVnkn-AEkY3SXN8DCSSA2hxrQ7vQ@mail.gmail.com>
References: <CAMQQO3=D=2y3BGQJzrk3VHy9=LV_EdYjyVnAALMuq9N=KgANbA@mail.gmail.com>
 <CANsrgP+1Ve0ahkndAoSOASU4cm-K92hJ=SXG+OpYKHX4bEuO-g@mail.gmail.com>
 <CAMQQO3kq0HxLShqdiPesnGMVnkn-AEkY3SXN8DCSSA2hxrQ7vQ@mail.gmail.com>
Message-ID: <87zhdlupw9.fsf@smart-cactus.org>

Ömer Sinan Ağacan <omeragacan at gmail.com> writes:

> Right, I think that's the problem. We then pass the same "size" to
> scavenge_large_bitmap as the size of the bitmap. So we assume size of the bitmap
> is pap->n_args.
>
> So the call stack is
>
> - scavenge_PAP, calls scavenge_PAP_payload with pap->n_args as "size"
> - scavenge_PAP_payload, calls scavenge_large_bitmap with "size" (== pap->n_args)
>   as the bitmap's size
>
> Is this expected?
>
Omer and I discussed this via IRC.

I believe that the intent here is that scavenge_PAP_payload scavenges
precisely the number of arguments that the PAP's stack fragment
includes, using the function's bitmap to do so (since we must know which
of this arguments are pointers).

Moreover, we know that pap->n_args is less or equal to than the size of
the bitmap (since otherwise this wouldn't be a partial application).
Consequently this is safe and correct.

Cheers,

- Ben
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 487 bytes
Desc: not available
URL: <http://mail.haskell.org/pipermail/ghc-devs/attachments/20200214/56a05f42/attachment.sig>

From omeragacan at gmail.com  Fri Feb 14 15:55:37 2020
From: omeragacan at gmail.com (=?UTF-8?Q?=C3=96mer_Sinan_A=C4=9Facan?=)
Date: Fri, 14 Feb 2020 18:55:37 +0300
Subject: Confused about PAP object layout
In-Reply-To: <FF5887E8-5266-473B-951D-4DDD02A349E8@gmail.com>
References: <CAMQQO3=D=2y3BGQJzrk3VHy9=LV_EdYjyVnAALMuq9N=KgANbA@mail.gmail.com>
 <CANsrgP+1Ve0ahkndAoSOASU4cm-K92hJ=SXG+OpYKHX4bEuO-g@mail.gmail.com>
 <CAMQQO3kq0HxLShqdiPesnGMVnkn-AEkY3SXN8DCSSA2hxrQ7vQ@mail.gmail.com>
 <FF5887E8-5266-473B-951D-4DDD02A349E8@gmail.com>
Message-ID: <CAMQQO3=bgoRY_OQ3eMJ3m7fSN4S9Wf8QieKSm6VgYBDs6=mcBw@mail.gmail.com>

I think that makes sense, with the invariant that n_args <= bitmap_size. We
evacuate the arguments used by the function but not others. Thanks.

It's somewhat weird to see an object with useful stuff, then garbage, then
useful stuff again in the heap, but that's not an issue by itself. For example
if I have something like

    [pap_info, x, y, z]

and according to the function `y` is dead, then after evacuating I get

    [pap_info, x, <garbage>, z]

This "garbage" is evacuated again and again every time we evacuate this PAP.

Ömer

Alexis King <lexi.lambda at gmail.com>, 14 Şub 2020 Cum, 18:43 tarihinde
şunu yazdı:

>
> Disclaimer: I am not an expert. But I happened to have been looking at this code just yesterday, so I’ll try to answer to check my understanding. :)
>
> Fundamentally, a PAP is not fully-saturated, so the number of arguments in its payload may be smaller than the information contained in the function’s bitmap. scavenge_large_bitmap calls walk_large_bitmap, which uses the bitmap as a “ruler” to guide the traversal, lining up each element in the payload to information in the bitmap. But the traversal only actually walks a payload of the specified size, so if there’s less information in the payload than there is information in the bitmap, the traversal will just terminate early.
>
> > On Feb 14, 2020, at 09:30, Ömer Sinan Ağacan <omeragacan at gmail.com> wrote:
> >
> > Right, I think that's the problem. We then pass the same "size" to
> > scavenge_large_bitmap as the size of the bitmap. So we assume size of the bitmap
> > is pap->n_args.
> >
> > So the call stack is
> >
> > - scavenge_PAP, calls scavenge_PAP_payload with pap->n_args as "size"
> > - scavenge_PAP_payload, calls scavenge_large_bitmap with "size" (== pap->n_args)
> >  as the bitmap's size
> >
> > Is this expected?
> >
> > Ömer

From ben at smart-cactus.org  Fri Feb 14 17:25:22 2020
From: ben at smart-cactus.org (Ben Gamari)
Date: Fri, 14 Feb 2020 12:25:22 -0500
Subject: Confused about PAP object layout
In-Reply-To: <CAMQQO3=bgoRY_OQ3eMJ3m7fSN4S9Wf8QieKSm6VgYBDs6=mcBw@mail.gmail.com>
References: <CAMQQO3=D=2y3BGQJzrk3VHy9=LV_EdYjyVnAALMuq9N=KgANbA@mail.gmail.com>
 <CANsrgP+1Ve0ahkndAoSOASU4cm-K92hJ=SXG+OpYKHX4bEuO-g@mail.gmail.com>
 <CAMQQO3kq0HxLShqdiPesnGMVnkn-AEkY3SXN8DCSSA2hxrQ7vQ@mail.gmail.com>
 <FF5887E8-5266-473B-951D-4DDD02A349E8@gmail.com>
 <CAMQQO3=bgoRY_OQ3eMJ3m7fSN4S9Wf8QieKSm6VgYBDs6=mcBw@mail.gmail.com>
Message-ID: <87sgjdulmq.fsf@smart-cactus.org>

Ömer Sinan Ağacan <omeragacan at gmail.com> writes:

> I think that makes sense, with the invariant that n_args <= bitmap_size. We
> evacuate the arguments used by the function but not others. Thanks.
>
> It's somewhat weird to see an object with useful stuff, then garbage, then
> useful stuff again in the heap, but that's not an issue by itself. For example
> if I have something like
>
>     [pap_info, x, y, z]
>
> and according to the function `y` is dead, then after evacuating I get
>
>     [pap_info, x, <garbage>, z]
>
> This "garbage" is evacuated again and again every time we evacuate this PAP.
>
I'm not sure what you mean by "garbage". The bitmap merely determines
whether a field is a pointer, not whether it is copied during
evacuation. A field's bitmap bit not being set merely means that we won't
evacuate the value of that field during scavenging.

Nevertheless, this all deserves a comment in scavenge_PAP.

Cheers,

- Ben

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 487 bytes
Desc: not available
URL: <http://mail.haskell.org/pipermail/ghc-devs/attachments/20200214/82092d1e/attachment.sig>

From lexi.lambda at gmail.com  Fri Feb 14 19:52:47 2020
From: lexi.lambda at gmail.com (Alexis King)
Date: Fri, 14 Feb 2020 13:52:47 -0600
Subject: Calling an unknown function from low-level Cmm
Message-ID: <1A55D647-2693-4B72-BFC4-F47374E4F393@gmail.com>

Hi all,

I’m trying to understand how to properly call an unknown function from low-level Cmm code. If I’m just applying a function to a state token, it’s easy; I can just do

    R1 = io;
    jump stg_ap_v_fast [R1];

since the calling convention is consistent in that case. But what if my function takes actual arguments? I can’t do

    R1 = fun;
    R2 = arg;
    jump stg_ap_p_fast [R1, R2];

because if the calling convention doesn’t pass any arguments in registers, that would be wrong. I could check if NO_ARG_REGS is defined and generate different code in that situation, but that seems extreme. One option I think would work would be to do

    R1 = fun;
    Sp_adj(-2);
    Sp(1) = arg;
    jump RET_LBL(stg_ap_p) [R1];

but that seems wasteful if I have the argument in a register already anyway. Am I missing something?

Thanks,
Alexis

From csaba.hruska at gmail.com  Sun Feb 16 13:19:01 2020
From: csaba.hruska at gmail.com (Csaba Hruska)
Date: Sun, 16 Feb 2020 14:19:01 +0100
Subject: GHC 8.8.2 miscompiles program with -fcatch-bottoms
Message-ID: <CANjChEotP+-PeLEL8nPNd-FUHE787vXhMRmW+4sDP9EE07cU8Q@mail.gmail.com>

Hello,

I've tried to compile GHC from source using -fcatch-bottoms on the stage2.
I compiled GHC with having *GhcStage2HcOpts += -fcatch-bottoms* in build.mk.
Unfortunately it breaks programs and causes runtime error. I wanted to use
this feature in an experiment.
My motivation:

*I'm working on a typed external STG IR were the type system is based
according GHC's PrimRep ADT. In my Ext-STG IR each case alt must have the
same PrimRep result type. The vanilla Core to Stg pass breaks this
invariant but the catch-bottoms option preserves it. *

Cheers,
Csaba

Error output:

make
===--- building phase 0
make --no-print-directory -f ghc.mk phase=0 phase_0_builds
make[1]: Nothing to be done for 'phase_0_builds'.
===--- building phase 1
make --no-print-directory -f ghc.mk phase=1 phase_1_builds
make[1]: Nothing to be done for 'phase_1_builds'.
===--- building final phase
make --no-print-directory -f ghc.mk phase=final all
"inplace/bin/ghc-stage2" -hisuf dyn_hi -osuf  dyn_o -hcsuf dyn_hc -fPIC
-dynamic  -H32m -O -Wall      -hide-all-packages -i -iutils/haddock/driver
-iutils/haddock/haddock-api/src -iutils/haddock/haddock-library/src
-iutils/haddock/dist/build -Iutils/haddock/dist/build
-iutils/haddock/dist/build/haddock/autogen
-Iutils/haddock/dist/build/haddock/autogen    -optP-DIN_GHC_TREE
-optP-include -optPutils/haddock/dist/build/haddock/autogen/cabal_macros.h
-package-id array-0.5.4.0 -package-id base-4.13.0.0 -package-id
bytestring-0.10.10.0 -package-id containers-0.6.2.1 -package-id
deepseq-1.4.4.0 -package-id directory-1.3.4.0 -package-id filepath-1.4.2.1
-package-id ghc-8.8.2 -package-id ghc-boot-8.8.2 -package-id
parsec-3.1.14.0 -package-id text-1.2.4.0 -package-id transformers-0.5.6.2
-package-id xhtml-3000.2.2.1 -funbox-strict-fields -Wall -fwarn-tabs -O2
-threaded -XHaskell2010  -no-user-package-db -rtsopts  -Wno-unused-imports
-Wno-deprecations     -Wnoncanonical-monad-instances  -outputdir
utils/haddock/dist/build    -c
utils/haddock/haddock-library/src/Documentation/Haddock/Types.hs -o
utils/haddock/dist/build/Documentation/Haddock/Types.dyn_o


*ghc-stage2: internal error: evacuate(static): strange closure type 0
(GHC version 8.8.2 for x86_64_unknown_linux)    Please report this as a GHC
bug:  https://www.haskell.org/ghc/reportabug
<https://www.haskell.org/ghc/reportabug>*
utils/haddock/ghc.mk:20: recipe for target
'utils/haddock/dist/build/Documentation/Haddock/Types.dyn_o' failed
make[1]: *** [utils/haddock/dist/build/Documentation/Haddock/Types.dyn_o]
Aborted (core dumped)
Makefile:123: recipe for target 'all' failed
make: *** [all] Error 2
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.haskell.org/pipermail/ghc-devs/attachments/20200216/e2195915/attachment.html>

From omer at well-typed.com  Sun Feb 16 13:46:18 2020
From: omer at well-typed.com (omer at well-typed.com)
Date: Sun, 16 Feb 2020 16:46:18 +0300
Subject: GHC 8.8.2 miscompiles program with -fcatch-bottoms
In-Reply-To: <CANjChEotP+-PeLEL8nPNd-FUHE787vXhMRmW+4sDP9EE07cU8Q@mail.gmail.com>
Message-ID: <6c539815-9b62-4848-9311-f14339726550@email.android.com>

An HTML attachment was scrubbed...
URL: <http://mail.haskell.org/pipermail/ghc-devs/attachments/20200216/1d60e1d4/attachment.html>

From csaba.hruska at gmail.com  Sun Feb 16 20:53:31 2020
From: csaba.hruska at gmail.com (Csaba Hruska)
Date: Sun, 16 Feb 2020 21:53:31 +0100
Subject: source-dist from GHC master branch
Message-ID: <CANjChEr7B-8KKWzDoVjzGi05syy8zTnXnkE69j_EsZQ+kDCwJA@mail.gmail.com>

Hello,

I'd like to create a source distribution (source-dist) from GHC git master
branch.
I tried hadrian (hadrian/build.sh source-dist) and make (make sdist-ghc)
but both failed in different ways. Should this work (at all) for any git
snapshot?

Hadrian successfully built a source dist (ghc-8.11.0.20200215-src.tar.xz)
but it did not compile.

Regards,
Csaba

The error output (make sdist-ghc):

> make sdist-ghc
> make --no-print-directory -f ghc.mk sdist-ghc NO_INCLUDE_DEPS=YES
> NO_INCLUDE_PKGDATA=YES
> ghc.mk:1221: warning: overriding recipe for target
> 'sdist_compiler_stage2_Cmm'
> ghc.mk:1220: warning: ignoring old recipe for target
> 'sdist_compiler_stage2_Cmm'
> make[1]: *** No rule to make target 'compiler/stage2/build/Cmm.hs', needed
> by 'sdist_compiler_stage2_Cmm'. Stop.
> Makefile:162: recipe for target 'sdist-ghc' failed
> make: *** [sdist-ghc] Error 2
>
> csaba at x1:~/haskell/grin-compiler/ghc-playground/ghc$ make sdist-ghc
> make --no-print-directory -f ghc.mk sdist-ghc NO_INCLUDE_DEPS=YES
> NO_INCLUDE_PKGDATA=YES
> ghc.mk:1221: warning: overriding recipe for target
> 'sdist_compiler_stage2_Cmm'
> ghc.mk:1220: warning: ignoring old recipe for target
> 'sdist_compiler_stage2_Cmm'
> make[1]: *** No rule to make target 'compiler/stage2/build/Cmm.hs', needed
> by 'sdist_compiler_stage2_Cmm'. Stop.
> Makefile:162: recipe for target 'sdist-ghc' failed
> make: *** [sdist-ghc] Error 2


Error output hadrian:

> | Run GenPrimopCode: _build/stage0/compiler/build/primops.txt =>
> _build/stage0/compiler/build/primop-vector-tys-exports.hs-incl
> | Run Ghc FindHsDependencies Stage0: compiler/main/Annotations.hs (and 475
> more) => _build/stage0/compiler/.dependencies.mk
>
> <no location info>: error:
>     module ‘ghc-8.6.2:Lexer’ is defined in multiple files:
> compiler/GHC/Cmm/Lexer.hs
>
>  compiler/parser/Lexer.hs
> Error when running Shake build system:
>   at action, called at src/Rules.hs:71:19 in main:Rules
>   at need, called at src/Rules.hs:93:5 in main:Rules
> * Depends on: _build/stage0/lib/package.conf.d/ghc-8.11.0.20200215.conf
>   at need, called at src/Rules/Register.hs:116:5 in main:Rules.Register
> * Depends on: _build/stage0/compiler/build/libHSghc-8.11.0.20200215.a
>   at need, called at src/Rules/Library.hs:146:5 in main:Rules.Library
> * Depends on: _build/stage0/compiler/build/Cpr.o
>   at &%>, called at src/Rules/Compile.hs:62:9 in main:Rules.Compile
> * Depends on: _build/stage0/compiler/build/Cpr.o
> _build/stage0/compiler/build/Cpr.hi
>   at apply1, called at
> src/Development/Shake/Internal/Rules/Oracle.hs:159:32 in
> shake-0.18.3-a2e904f154f09728357733d7a3e8955a2a87773d2d7eb963a6f88d9d883fdde3:Development.Shake.Internal.Rules.Oracle
> * Depends on: OracleQ (KeyValues
> ("_build/stage0/compiler/.dependencies","_build/stage0/compiler/build/Cpr.o"))
>   at need, called at src/Hadrian/Oracles/TextFile.hs:96:9 in
> main:Hadrian.Oracles.TextFile
> * Depends on: _build/stage0/compiler/.dependencies
>   at readFile', called at src/Rules/Dependencies.hs:34:19 in
> main:Rules.Dependencies
>   at need, called at src/Development/Shake/Internal/Derived.hs:118:15 in
> shake-0.18.3-a2e904f154f09728357733d7a3e8955a2a87773d2d7eb963a6f88d9d883fdde3:Development.Shake.Internal.Derived
> * Depends on: _build/stage0/compiler/.dependencies.mk
> * Raised the exception:
> user error (Development.Shake.cmd, system command failed
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.haskell.org/pipermail/ghc-devs/attachments/20200216/b254b940/attachment.html>

From florian.engel at active-group.de  Tue Feb 18 10:15:45 2020
From: florian.engel at active-group.de (Florian Engel)
Date: Tue, 18 Feb 2020 11:15:45 +0100
Subject: core to core plugin before inline
Message-ID: <87imk4qjzi.fsf@active-group.de>

Hello,

I'm trying to fix an error in https://github.com/conal/concat.  The
problem is the core to core passes of the plugin are way to slow for
simple functions.  For slightly more complicated one but still simple
functions, it fails after a long time of compiling (see the errGrad
tests in examples/test/Examples.hs).  We suppose that the problem is
related to inline.  Is there a way to run the core-to-core passes of
the plugin before inline.

Best regards
Florian Engel
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 487 bytes
Desc: not available
URL: <http://mail.haskell.org/pipermail/ghc-devs/attachments/20200218/e63a260f/attachment.sig>

From compl.yue at icloud.com  Tue Feb 18 14:40:48 2020
From: compl.yue at icloud.com (Compl Yue)
Date: Tue, 18 Feb 2020 22:40:48 +0800
Subject: [smartos-discuss] GHC 8.8.2 + cabal-install 3.0.0 finally working
 on SmartOS x64 20200117 (yet GHC 8.6.5 left less usable)
In-Reply-To: <20200218114152.GC22635@joyent.com>
References: <f1d4b7bf-5398-ed9e-aeee-90b14a99ea1e@icloud.com>
 <20200218114152.GC22635@joyent.com>
Message-ID: <f06cfdef-f00d-781c-3556-a83f743883fe@icloud.com>

Great!

And I suggest add 8.6.5 to the list, as I feel it may have wider 
adoption than other versions as time being, and fortunately the previous 
ghci problem with it has been fixed by backporting merge request 87, my 
repository has updated the build steps and the backporting patch. FYI.

On 2020/2/18 下午7:41, Jonathan Perkin wrote:
> * On 2020-02-10 at 08:26 GMT, Compl Yue via smartos-discuss wrote:
>
>> And regarding stack, as it's focused on managing binary distributions of GHC
>> (build-from-src not working very well per my experience), it'll need a
>> trusted binary repository for stack to work with on SmartOS. A private http
>> server hosting the bindists should be straight forward, but I'm not
>> interested to host a publicly available binary repository with those bindist
>> tarballs digitally signed (or there're security concerns). If anyone is
>> interested, I'd like to help as far as I can.
> Thanks again for your work in paving the path forward with this.  I've
> used it to produce some official bootstraps for SmartOS here:
>
>    https://us-east.manta.joyent.com/pkgsrc/public/pkg-bootstraps/ghc-7.6.3-boot-x86_64-unknown-solaris2.tar.xz
>    https://us-east.manta.joyent.com/pkgsrc/public/pkg-bootstraps/ghc-7.10.3-boot-x86_64-unknown-solaris2.tar.xz
>    https://us-east.manta.joyent.com/pkgsrc/public/pkg-bootstraps/ghc-8.0.2-boot-x86_64-unknown-solaris2.tar.xz
>    https://us-east.manta.joyent.com/pkgsrc/public/pkg-bootstraps/ghc-8.4.4-boot-x86_64-unknown-solaris2.tar.xz
>
> For some reason I'm seeing build errors with the 8.8.1 bootstrap
> related to the rts/fs.h header but the normal release build works
> fine, I should hopefully have that fixed soon.
>
> There are also release packages now available in the SmartOS trunk
> package repository ready for users to install:
>
>    $ pkgin se ^ghc-
>    ghc-8.8.1nb1         Compiler for the functional language Haskell - 8.8 Release Series
>    ghc-8.4.4            Compiler for the functional language Haskell - 8.4 Release Series
>    ghc-8.0.2            Compiler for the functional language Haskell - 8.0 Release Series
>    ghc-7.10.3nb2        Compiler for the functional language Haskell - 7.10 Release Series
>
> I'd be particularly interested in whether Haskell users can find any
> issues with them.  Each of the release packages were built using the
> previous bootstrap kit listed above.
>
> Many thanks,
>

From ben at well-typed.com  Tue Feb 18 17:10:01 2020
From: ben at well-typed.com (Ben Gamari)
Date: Tue, 18 Feb 2020 12:10:01 -0500
Subject: core to core plugin before inline
In-Reply-To: <87imk4qjzi.fsf@active-group.de>
References: <87imk4qjzi.fsf@active-group.de>
Message-ID: <87sgj7bz4p.fsf@smart-cactus.org>

Florian Engel <florian.engel at active-group.de> writes:

> Hello,
>
> I'm trying to fix an error in https://github.com/conal/concat.  The
> problem is the core to core passes of the plugin are way to slow for
> simple functions.  For slightly more complicated one but still simple
> functions, it fails after a long time of compiling (see the errGrad
> tests in examples/test/Examples.hs).  We suppose that the problem is
> related to inline.  Is there a way to run the core-to-core passes of
> the plugin before inline.
>
Sure: you specify when your plugin will run when you register it. In the
case of `concat` it appears [1] that you insert your plugin after
several other Core-to-Core passes. IIRC, if you position your plugin
first in the [CoreTodo] returned by `install` then it will run right
after desugaring.

However, note that there is still a bit of inlining done by the "simple
optimiser" (see the CoreOpt module). However, it seems unlikely that
this inlining will cause any trouble for your plugin since it doesn't
duplicate code.

Cheers,

- Ben


[1] https://github.com/conal/concat/blob/master/plugin/src/ConCat/Plugin.hs#L1390
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 487 bytes
Desc: not available
URL: <http://mail.haskell.org/pipermail/ghc-devs/attachments/20200218/7908aa83/attachment.sig>

From ben at smart-cactus.org  Tue Feb 18 17:14:25 2020
From: ben at smart-cactus.org (Ben Gamari)
Date: Tue, 18 Feb 2020 12:14:25 -0500
Subject: source-dist from GHC master branch
In-Reply-To: <CANjChEr7B-8KKWzDoVjzGi05syy8zTnXnkE69j_EsZQ+kDCwJA@mail.gmail.com>
References: <CANjChEr7B-8KKWzDoVjzGi05syy8zTnXnkE69j_EsZQ+kDCwJA@mail.gmail.com>
Message-ID: <87pnebbyxd.fsf@smart-cactus.org>

Csaba Hruska <csaba.hruska at gmail.com> writes:

> Hello,
>
> I'd like to create a source distribution (source-dist) from GHC git master
> branch.
> I tried hadrian (hadrian/build.sh source-dist) and make (make sdist-ghc)
> but both failed in different ways. Should this work (at all) for any git
> snapshot?
>
> Hadrian successfully built a source dist (ghc-8.11.0.20200215-src.tar.xz)
> but it did not compile.
>
For the record, this is being discussed on #17848 and #17849.

Cheers,

- Ben

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 487 bytes
Desc: not available
URL: <http://mail.haskell.org/pipermail/ghc-devs/attachments/20200218/a45665c3/attachment.sig>

From b at chreekat.net  Wed Feb 19 08:09:32 2020
From: b at chreekat.net (Bryan Richter)
Date: Wed, 19 Feb 2020 10:09:32 +0200
Subject: I'm concerned about the affine-types extension and its impact on
 ghc and haskell at large
In-Reply-To: <CAHYVw0whtfJAYzxq3zYySpZgmbAFqUktuFnNP7feTZxsnSFhTg@mail.gmail.com>
References: <CAHYVw0whtfJAYzxq3zYySpZgmbAFqUktuFnNP7feTZxsnSFhTg@mail.gmail.com>
Message-ID: <CAH9BU=z8ex7oF==FHV2XNHZ8UT5zXSJ3KkMeY1QnLjztRuLAuw@mail.gmail.com>

Is this true? 6-7% slowdown across the board for all users and all use
cases of GHC?

No segment of the community uses all of GHC's features. I, for one,
appreciate GHC for its usability and not just its type level wizardry. A
great way to improve GHC's usability is to improve compile times. If this
does the opposite for everybody, whether they use these new features or
not, I am nonplussed.

If Carter is trying to raise concern about this feature, he has been
successful. :)


On Sat, 8 Feb 2020, 2.37 Carter Schonwald, <carter.schonwald at gmail.com>
wrote:

>
> https://github.com/ghc-proposals/ghc-proposals/pull/111#issuecomment-583664586
>
>
> https://www.reddit.com/r/haskell/comments/f0jigv/im_concerned_about_the_longterm_impact_of_the/
>
> As current maintainer of vector, and a member of the CLC, both of which
> roles really should be AFFIRMATIONAL stakeholders in this, I only see costs
> and concerns.
>
> As someone who's spent the past few years doing a LOT of modelling and
> prototyping around marrying linear logic, formal methods, and functional
> programming in an applied industrial setting, i should be an Affirmational
> stakeholder. yet again I am not.
>
> theres very real costs in the complexity and scope of impact that impact
> EVERY single user and stakeholder of ghc and haskell. And I do not see any
> concrete population that benefits.
>
>
> cale even makes a very constructive and articulate point
>
> > I don't know how much my opinion matters at this point, but I'd really
> like to see the non-toy real-world use cases of this before I can even
> consider thinking that it would be a good idea to merge to the main
> compiler. It has such a huge potential impact on so many people in the
> Haskell community:
> >
> >     * Library maintainers who start getting PRs that make "improvements"
> to linearity while making their libraries harder to maintain because their
> interface becomes more rigid, and harder to understand because the types
> are more complicated.
> >
> >     * Beginners, or simply ordinary users of the language who have to
> pay the mental overhead of living with the linear types and polymorphism
> spreading everywhere as types are adjusted to make terms more usable in
> places where one is concerned with linearity.
> >
> >     * Commercial users of the language who pay for the additional time
> taken by all their employees waiting for the compiler to run, regardless of
> whether or not they're using the extension. If Carter's 6-7% slowdown is
> real even in cases where one doesn't care about the extension, I can
> imagine wanting to make a fork without the extension. The compiler is
> already 2 orders of magnitude slower than I'd like. If that weren't the
> case, maybe 6-7% wouldn't be a huge deal. While ghci is often helpful at
> shortening the feedback loop, it's not always a viable solution.
> >
> >
> > But really, I just want to know how anyone would put this to practical
> use in a real setting -- it needs to be _really_ compelling to make up for
> the social cost. It can't be that hard for Tweag, or someone enthusiastic,
> to live on a fork until they have a decent case study to show the world, so
> we can say "okay, that's actually really cool, maybe we actually want that
> everywhere".
>
> _______________________________________________
> ghc-devs mailing list
> ghc-devs at haskell.org
> http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.haskell.org/pipermail/ghc-devs/attachments/20200219/e448e8c4/attachment.html>

From m at tweag.io  Wed Feb 19 08:30:22 2020
From: m at tweag.io (Boespflug, Mathieu)
Date: Wed, 19 Feb 2020 09:30:22 +0100
Subject: I'm concerned about the affine-types extension and its impact on
 ghc and haskell at large
In-Reply-To: <CAH9BU=z8ex7oF==FHV2XNHZ8UT5zXSJ3KkMeY1QnLjztRuLAuw@mail.gmail.com>
References: <CAHYVw0whtfJAYzxq3zYySpZgmbAFqUktuFnNP7feTZxsnSFhTg@mail.gmail.com>
 <CAH9BU=z8ex7oF==FHV2XNHZ8UT5zXSJ3KkMeY1QnLjztRuLAuw@mail.gmail.com>
Message-ID: <CAJ0eudpv2yz8kt832Cidg5ek5k0WyTJs5=MWJXwm2zM4B5SJvw@mail.gmail.com>

Hi Bryan,

the discussion has continued on the original PR for the GHC proposal. See
https://github.com/ghc-proposals/ghc-proposals/pull/111#issuecomment-583795811
and
following comments.

The tl;dr is that the concerns raised about performance are far too
premature: for their own curiosity someone ran a few benchmarks on the
branch of a merge request marked as WIP in the title and which has not yet
been performance optimized. See also
https://www.reddit.com/r/haskell/comments/f0jigv/im_concerned_about_the_longterm_impact_of_the/fgxwk9w/.
The conditions that would need to be satisfied before a merge are clear and
include performance requirements:
https://github.com/ghc-proposals/ghc-proposals/pull/111#issuecomment-431944078
.

On Wed, 19 Feb 2020 at 09:10, Bryan Richter <b at chreekat.net> wrote:

> Is this true? 6-7% slowdown across the board for all users and all use
> cases of GHC?
>
> No segment of the community uses all of GHC's features. I, for one,
> appreciate GHC for its usability and not just its type level wizardry. A
> great way to improve GHC's usability is to improve compile times. If this
> does the opposite for everybody, whether they use these new features or
> not, I am nonplussed.
>
> If Carter is trying to raise concern about this feature, he has been
> successful. :)
>
>
> On Sat, 8 Feb 2020, 2.37 Carter Schonwald, <carter.schonwald at gmail.com>
> wrote:
>
>>
>> https://github.com/ghc-proposals/ghc-proposals/pull/111#issuecomment-583664586
>>
>>
>> https://www.reddit.com/r/haskell/comments/f0jigv/im_concerned_about_the_longterm_impact_of_the/
>>
>> As current maintainer of vector, and a member of the CLC, both of which
>> roles really should be AFFIRMATIONAL stakeholders in this, I only see costs
>> and concerns.
>>
>> As someone who's spent the past few years doing a LOT of modelling and
>> prototyping around marrying linear logic, formal methods, and functional
>> programming in an applied industrial setting, i should be an Affirmational
>> stakeholder. yet again I am not.
>>
>> theres very real costs in the complexity and scope of impact that impact
>> EVERY single user and stakeholder of ghc and haskell. And I do not see any
>> concrete population that benefits.
>>
>>
>> cale even makes a very constructive and articulate point
>>
>> > I don't know how much my opinion matters at this point, but I'd really
>> like to see the non-toy real-world use cases of this before I can even
>> consider thinking that it would be a good idea to merge to the main
>> compiler. It has such a huge potential impact on so many people in the
>> Haskell community:
>> >
>> >     * Library maintainers who start getting PRs that make
>> "improvements" to linearity while making their libraries harder to maintain
>> because their interface becomes more rigid, and harder to understand
>> because the types are more complicated.
>> >
>> >     * Beginners, or simply ordinary users of the language who have to
>> pay the mental overhead of living with the linear types and polymorphism
>> spreading everywhere as types are adjusted to make terms more usable in
>> places where one is concerned with linearity.
>> >
>> >     * Commercial users of the language who pay for the additional time
>> taken by all their employees waiting for the compiler to run, regardless of
>> whether or not they're using the extension. If Carter's 6-7% slowdown is
>> real even in cases where one doesn't care about the extension, I can
>> imagine wanting to make a fork without the extension. The compiler is
>> already 2 orders of magnitude slower than I'd like. If that weren't the
>> case, maybe 6-7% wouldn't be a huge deal. While ghci is often helpful at
>> shortening the feedback loop, it's not always a viable solution.
>> >
>> >
>> > But really, I just want to know how anyone would put this to practical
>> use in a real setting -- it needs to be _really_ compelling to make up for
>> the social cost. It can't be that hard for Tweag, or someone enthusiastic,
>> to live on a fork until they have a decent case study to show the world, so
>> we can say "okay, that's actually really cool, maybe we actually want that
>> everywhere".
>>
>> _______________________________________________
>> ghc-devs mailing list
>> ghc-devs at haskell.org
>> http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs
>>
> _______________________________________________
> ghc-devs mailing list
> ghc-devs at haskell.org
> http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.haskell.org/pipermail/ghc-devs/attachments/20200219/9cb01775/attachment.html>

From b at chreekat.net  Wed Feb 19 08:38:51 2020
From: b at chreekat.net (Bryan Richter)
Date: Wed, 19 Feb 2020 10:38:51 +0200
Subject: I'm concerned about the affine-types extension and its impact on
 ghc and haskell at large
In-Reply-To: <CAJ0eudpv2yz8kt832Cidg5ek5k0WyTJs5=MWJXwm2zM4B5SJvw@mail.gmail.com>
References: <CAHYVw0whtfJAYzxq3zYySpZgmbAFqUktuFnNP7feTZxsnSFhTg@mail.gmail.com>
 <CAH9BU=z8ex7oF==FHV2XNHZ8UT5zXSJ3KkMeY1QnLjztRuLAuw@mail.gmail.com>
 <CAJ0eudpv2yz8kt832Cidg5ek5k0WyTJs5=MWJXwm2zM4B5SJvw@mail.gmail.com>
Message-ID: <CAH9BU=w8cj54uZ8zZ4986Ev7V47TeJtVmvqL7RUOMMO7Kwd5gg@mail.gmail.com>

Great, thanks for the update!

I am not at all surprised to hear that this whole thing is being considered
intelligently and carefully. :)


On Wed, Feb 19, 2020 at 10:30 AM Boespflug, Mathieu <m at tweag.io> wrote:

> Hi Bryan,
>
> the discussion has continued on the original PR for the GHC proposal. See
> https://github.com/ghc-proposals/ghc-proposals/pull/111#issuecomment-583795811 and
> following comments.
>
> The tl;dr is that the concerns raised about performance are far too
> premature: for their own curiosity someone ran a few benchmarks on the
> branch of a merge request marked as WIP in the title and which has not yet
> been performance optimized. See also
> https://www.reddit.com/r/haskell/comments/f0jigv/im_concerned_about_the_longterm_impact_of_the/fgxwk9w/.
> The conditions that would need to be satisfied before a merge are clear and
> include performance requirements:
> https://github.com/ghc-proposals/ghc-proposals/pull/111#issuecomment-431944078
> .
>
> On Wed, 19 Feb 2020 at 09:10, Bryan Richter <b at chreekat.net> wrote:
>
>> Is this true? 6-7% slowdown across the board for all users and all use
>> cases of GHC?
>>
>> No segment of the community uses all of GHC's features. I, for one,
>> appreciate GHC for its usability and not just its type level wizardry. A
>> great way to improve GHC's usability is to improve compile times. If this
>> does the opposite for everybody, whether they use these new features or
>> not, I am nonplussed.
>>
>> If Carter is trying to raise concern about this feature, he has been
>> successful. :)
>>
>>
>> On Sat, 8 Feb 2020, 2.37 Carter Schonwald, <carter.schonwald at gmail.com>
>> wrote:
>>
>>>
>>> https://github.com/ghc-proposals/ghc-proposals/pull/111#issuecomment-583664586
>>>
>>>
>>> https://www.reddit.com/r/haskell/comments/f0jigv/im_concerned_about_the_longterm_impact_of_the/
>>>
>>> As current maintainer of vector, and a member of the CLC, both of which
>>> roles really should be AFFIRMATIONAL stakeholders in this, I only see costs
>>> and concerns.
>>>
>>> As someone who's spent the past few years doing a LOT of modelling and
>>> prototyping around marrying linear logic, formal methods, and functional
>>> programming in an applied industrial setting, i should be an Affirmational
>>> stakeholder. yet again I am not.
>>>
>>> theres very real costs in the complexity and scope of impact that impact
>>> EVERY single user and stakeholder of ghc and haskell. And I do not see any
>>> concrete population that benefits.
>>>
>>>
>>> cale even makes a very constructive and articulate point
>>>
>>> > I don't know how much my opinion matters at this point, but I'd really
>>> like to see the non-toy real-world use cases of this before I can even
>>> consider thinking that it would be a good idea to merge to the main
>>> compiler. It has such a huge potential impact on so many people in the
>>> Haskell community:
>>> >
>>> >     * Library maintainers who start getting PRs that make
>>> "improvements" to linearity while making their libraries harder to maintain
>>> because their interface becomes more rigid, and harder to understand
>>> because the types are more complicated.
>>> >
>>> >     * Beginners, or simply ordinary users of the language who have to
>>> pay the mental overhead of living with the linear types and polymorphism
>>> spreading everywhere as types are adjusted to make terms more usable in
>>> places where one is concerned with linearity.
>>> >
>>> >     * Commercial users of the language who pay for the additional time
>>> taken by all their employees waiting for the compiler to run, regardless of
>>> whether or not they're using the extension. If Carter's 6-7% slowdown is
>>> real even in cases where one doesn't care about the extension, I can
>>> imagine wanting to make a fork without the extension. The compiler is
>>> already 2 orders of magnitude slower than I'd like. If that weren't the
>>> case, maybe 6-7% wouldn't be a huge deal. While ghci is often helpful at
>>> shortening the feedback loop, it's not always a viable solution.
>>> >
>>> >
>>> > But really, I just want to know how anyone would put this to practical
>>> use in a real setting -- it needs to be _really_ compelling to make up for
>>> the social cost. It can't be that hard for Tweag, or someone enthusiastic,
>>> to live on a fork until they have a decent case study to show the world, so
>>> we can say "okay, that's actually really cool, maybe we actually want that
>>> everywhere".
>>>
>>> _______________________________________________
>>> ghc-devs mailing list
>>> ghc-devs at haskell.org
>>> http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs
>>>
>> _______________________________________________
>> ghc-devs mailing list
>> ghc-devs at haskell.org
>> http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.haskell.org/pipermail/ghc-devs/attachments/20200219/f753c51d/attachment.html>

From omeragacan at gmail.com  Thu Feb 20 09:21:00 2020
From: omeragacan at gmail.com (=?UTF-8?Q?=C3=96mer_Sinan_A=C4=9Facan?=)
Date: Thu, 20 Feb 2020 12:21:00 +0300
Subject: Confused about PAP object layout
In-Reply-To: <87sgjdulmq.fsf@smart-cactus.org>
References: <CAMQQO3=D=2y3BGQJzrk3VHy9=LV_EdYjyVnAALMuq9N=KgANbA@mail.gmail.com>
 <CANsrgP+1Ve0ahkndAoSOASU4cm-K92hJ=SXG+OpYKHX4bEuO-g@mail.gmail.com>
 <CAMQQO3kq0HxLShqdiPesnGMVnkn-AEkY3SXN8DCSSA2hxrQ7vQ@mail.gmail.com>
 <FF5887E8-5266-473B-951D-4DDD02A349E8@gmail.com>
 <CAMQQO3=bgoRY_OQ3eMJ3m7fSN4S9Wf8QieKSm6VgYBDs6=mcBw@mail.gmail.com>
 <87sgjdulmq.fsf@smart-cactus.org>
Message-ID: <CAMQQO3nSeHAfzUJjCYvUVXCf8JwvPnVLoSO_fwMZmdgqijrpXg@mail.gmail.com>

> I'm not sure what you mean by "garbage". The bitmap merely determines whether
> a field is a pointer,

I think the bitmap is for liveness, not for whether a field is pointer or not.
Relevant code for building an info table for a function:

    mk_pieces (Fun arity (ArgGen arg_bits)) srt_label
      = do { (liveness_lit, liveness_data) <- mkLivenessBits dflags arg_bits
           ; let fun_type | null liveness_data = aRG_GEN
                          | otherwise          = aRG_GEN_BIG
                 extra_bits = [ packIntsCLit dflags fun_type arity ]
                           ++ (if inlineSRT dflags then [] else [ srt_lit ])
                           ++ [ liveness_lit, slow_entry ]
           ; return (Nothing, Nothing, extra_bits, liveness_data) }

This uses the word "liveness" rather than "pointers".

However I just realized that the word "garbage" is still not the best way to
describe what I'm trying to say. In the example

    [pap_info, x, y, z]

If the function's bitmap is [1, 0, 1], then `y` may be a dead (an unused
argument, or "garbage" as I describe in my previous email) OR it may be a
non-pointer, but used (i.e. not a garbage).

So maybe "liveness" is also not the best way to describe this bitmap, as 0 does
not mean dead but rather "don't follow in GC".

On my quest to understand and document this code better I have one more
question. When generating info tables for functions with know argument patterns
(ArgSpec) we initialize the bitmap as 0. Relevant code:

    mk_pieces (Fun arity (ArgSpec fun_type)) srt_label
      = do { let extra_bits = packIntsCLit dflags fun_type arity : srt_label
           ; return (Nothing, Nothing,  extra_bits, []) }

Here the last return value is for the liveness data. I don't understand how can
this be correct, because when we use this function in a PAP this will cause NOT
scavenging the PAP payload. Relevant code (simplified):

    STATIC_INLINE GNUC_ATTR_HOT StgPtr
    scavenge_PAP_payload (StgClosure *fun, StgClosure **payload, StgWord size)
    {
        const StgFunInfoTable *fun_info =
            get_fun_itbl(UNTAG_CONST_CLOSURE(fun));

        StgPtr p = (StgPtr)payload;

        StgWord bitmap;
        switch (fun_info->f.fun_type) {
        ...
        default:
            bitmap = BITMAP_BITS(stg_arg_bitmaps[fun_info->f.fun_type]);
        small_bitmap:
            p = scavenge_small_bitmap(p, size, bitmap);
            break;
        }
        return p;
    }

Here if I have a function with three pointer args (ARG_PPP) the shown branch
that will be taken, but because the bitmap is 0 (as shown in the mk_pieces code
above) nothing in the PAPs payload will be scavenged.

Here's an example from a debugging session:

    >>> print pap
    $10 = (StgPAP *) 0x42001fe030

    >>> print *pap
    $11 = {
      header = {
        info = 0x7fbdd1f06640 <stg_PAP_info>
      },
      arity = 2,
      n_args = 1,
      fun = 0x7fbdd2d23ffb,
      payload = 0x42001fe048
    }

So this PAP is applied one argument, which is a boxed object (a FUN_2_0):

    >>> print *get_itbl(UNTAG_CLOSURE(pap->payload[0]))
    $20 = {
      layout = {
        payload = {
          ptrs = 2,
          nptrs = 0
        },
        bitmap = 2,
        large_bitmap_offset = 2,
        __pad_large_bitmap_offset = 2,
        selector_offset = 2
      },
      type = 11,
      srt = 1914488,
      code = 0x7fbdd2b509c0 "H\215E\370L9\370r[I\203\304 M;\245X\003"
    }

However if I look at the function of this PAP:

    >>> print *get_fun_itbl(UNTAG_CLOSURE(pap->fun))
    $21 = {
      f = {
        slow_apply_offset = 16,
        __pad_slow_apply_offset = 3135120895,
        b = {
          bitmap = 74900193017889,
          bitmap_offset = 258342945,
          __pad_bitmap_offset = 258342945
        },
        fun_type = 23,
        arity = 3
      },
      i = {
        layout = {
          payload = {
            ptrs = 0,
            nptrs = 0
          },
          bitmap = 0,
          large_bitmap_offset = 0,
          __pad_large_bitmap_offset = 0,
          selector_offset = 0
        },
        type = 14,
        srt = 1916288,
        code = 0x7fbdd2b50260 <base_GHCziRead_list3_info>
"I\203\304(M;\245X\003"
      }
    }

It has arity 3. Since the first argument is a boxed object and this function has
arity 3, if the argument is actually live in the function (i.e. not an unused
argument), then the bitmap should have a 1 for this. But because the argument
pattern is known (ARG_PPP) we initialized the bitmap as 0! Not sure how this
can work.

What am I missing?

Thanks,

Ömer

Ben Gamari <ben at smart-cactus.org>, 14 Şub 2020 Cum, 20:25 tarihinde şunu yazdı:
>
> Ömer Sinan Ağacan <omeragacan at gmail.com> writes:
>
> > I think that makes sense, with the invariant that n_args <= bitmap_size. We
> > evacuate the arguments used by the function but not others. Thanks.
> >
> > It's somewhat weird to see an object with useful stuff, then garbage, then
> > useful stuff again in the heap, but that's not an issue by itself. For example
> > if I have something like
> >
> >     [pap_info, x, y, z]
> >
> > and according to the function `y` is dead, then after evacuating I get
> >
> >     [pap_info, x, <garbage>, z]
> >
> > This "garbage" is evacuated again and again every time we evacuate this PAP.
> >
> I'm not sure what you mean by "garbage". The bitmap merely determines
> whether a field is a pointer, not whether it is copied during
> evacuation. A field's bitmap bit not being set merely means that we won't
> evacuate the value of that field during scavenging.
>
> Nevertheless, this all deserves a comment in scavenge_PAP.
>
> Cheers,
>
> - Ben
>

From ben at smart-cactus.org  Thu Feb 20 15:21:50 2020
From: ben at smart-cactus.org (Ben Gamari)
Date: Thu, 20 Feb 2020 10:21:50 -0500
Subject: Confused about PAP object layout
In-Reply-To: <CAMQQO3nSeHAfzUJjCYvUVXCf8JwvPnVLoSO_fwMZmdgqijrpXg@mail.gmail.com>
References: <CAMQQO3=D=2y3BGQJzrk3VHy9=LV_EdYjyVnAALMuq9N=KgANbA@mail.gmail.com>
 <CANsrgP+1Ve0ahkndAoSOASU4cm-K92hJ=SXG+OpYKHX4bEuO-g@mail.gmail.com>
 <CAMQQO3kq0HxLShqdiPesnGMVnkn-AEkY3SXN8DCSSA2hxrQ7vQ@mail.gmail.com>
 <FF5887E8-5266-473B-951D-4DDD02A349E8@gmail.com>
 <CAMQQO3=bgoRY_OQ3eMJ3m7fSN4S9Wf8QieKSm6VgYBDs6=mcBw@mail.gmail.com>
 <87sgjdulmq.fsf@smart-cactus.org>
 <CAMQQO3nSeHAfzUJjCYvUVXCf8JwvPnVLoSO_fwMZmdgqijrpXg@mail.gmail.com>
Message-ID: <87v9o19tdi.fsf@smart-cactus.org>

Ömer Sinan Ağacan <omeragacan at gmail.com> writes:

>> I'm not sure what you mean by "garbage". The bitmap merely determines whether
>> a field is a pointer,
>
> I think the bitmap is for liveness, not for whether a field is pointer or not.
> Relevant code for building an info table for a function:
>
>     mk_pieces (Fun arity (ArgGen arg_bits)) srt_label
>       = do { (liveness_lit, liveness_data) <- mkLivenessBits dflags arg_bits
>            ; let fun_type | null liveness_data = aRG_GEN
>                           | otherwise          = aRG_GEN_BIG
>                  extra_bits = [ packIntsCLit dflags fun_type arity ]
>                            ++ (if inlineSRT dflags then [] else [ srt_lit ])
>                            ++ [ liveness_lit, slow_entry ]
>            ; return (Nothing, Nothing, extra_bits, liveness_data) }
>
> This uses the word "liveness" rather than "pointers".
>
> However I just realized that the word "garbage" is still not the best way to
> describe what I'm trying to say. In the example
>
>     [pap_info, x, y, z]
>
> If the function's bitmap is [1, 0, 1], then `y` may be a dead (an unused
> argument, or "garbage" as I describe in my previous email) OR it may be a
> non-pointer, but used (i.e. not a garbage).
>
> So maybe "liveness" is also not the best way to describe this bitmap, as 0 does
> not mean dead but rather "don't follow in GC".
>
This is indeed my understanding; "not live" in this context really just
means "not a pointer traced by the GC". I agree that "liveness" is a
poor word for it. This is briefly described in a comment in
includes/rts/storage/InfoTables.h.

> On my quest to understand and document this code better I have one more
> question. When generating info tables for functions with known
> argument patterns (ArgSpec) we initialize the bitmap as 0. Relevant
> code:
>
>     mk_pieces (Fun arity (ArgSpec fun_type)) srt_label
>       = do { let extra_bits = packIntsCLit dflags fun_type arity : srt_label
>            ; return (Nothing, Nothing,  extra_bits, []) }
>
> Here the last return value is for the liveness data. I don't understand how can
> this be correct, because when we use this function in a PAP this will cause NOT
> scavenging the PAP payload. Relevant code (simplified):
>
>     STATIC_INLINE GNUC_ATTR_HOT StgPtr
>     scavenge_PAP_payload (StgClosure *fun, StgClosure **payload, StgWord size)
>     {
>         const StgFunInfoTable *fun_info =
>             get_fun_itbl(UNTAG_CONST_CLOSURE(fun));
>
>         StgPtr p = (StgPtr)payload;
>
>         StgWord bitmap;
>         switch (fun_info->f.fun_type) {
>         ...
>         default:
>             bitmap = BITMAP_BITS(stg_arg_bitmaps[fun_info->f.fun_type]);
>         small_bitmap:
>             p = scavenge_small_bitmap(p, size, bitmap);
>             break;
>         }
>         return p;
>     }
>
> Here if I have a function with three pointer args (ARG_PPP) the shown branch
> that will be taken, but because the bitmap is 0 (as shown in the mk_pieces code
> above) nothing in the PAPs payload will be scavenged.
>
> Here's an example from a debugging session:
>
>     >>> print pap
>     $10 = (StgPAP *) 0x42001fe030
>
>     >>> print *pap
>     $11 = {
>       header = {
>         info = 0x7fbdd1f06640 <stg_PAP_info>
>       },
>       arity = 2,
>       n_args = 1,
>       fun = 0x7fbdd2d23ffb,
>       payload = 0x42001fe048
>     }
>
> So this PAP is applied one argument, which is a boxed object (a FUN_2_0):
>
>     >>> print *get_itbl(UNTAG_CLOSURE(pap->payload[0]))
>     $20 = {
>       layout = {
>         payload = {
>           ptrs = 2,
>           nptrs = 0
>         },
>         bitmap = 2,
>         large_bitmap_offset = 2,
>         __pad_large_bitmap_offset = 2,
>         selector_offset = 2
>       },
>       type = 11,
>       srt = 1914488,
>       code = 0x7fbdd2b509c0 "H\215E\370L9\370r[I\203\304 M;\245X\003"
>     }
>
> However if I look at the function of this PAP:
>
>     >>> print *get_fun_itbl(UNTAG_CLOSURE(pap->fun))
>     $21 = {
>       f = {
>         slow_apply_offset = 16,
>         __pad_slow_apply_offset = 3135120895,
>         b = {
>           bitmap = 74900193017889,
>           bitmap_offset = 258342945,
>           __pad_bitmap_offset = 258342945
>         },
>         fun_type = 23,
>         arity = 3
>       },
>       i = {
>         layout = {
>           payload = {
>             ptrs = 0,
>             nptrs = 0
>           },
>           bitmap = 0,
>           large_bitmap_offset = 0,
>           __pad_large_bitmap_offset = 0,
>           selector_offset = 0
>         },
>         type = 14,
>         srt = 1916288,
>         code = 0x7fbdd2b50260 <base_GHCziRead_list3_info>
> "I\203\304(M;\245X\003"
>       }
>     }
>
> It has arity 3. Since the first argument is a boxed object and this function has
> arity 3, if the argument is actually live in the function (i.e. not an unused
> argument), then the bitmap should have a 1 for this. But because the argument
> pattern is known (ARG_PPP) we initialized the bitmap as 0! Not sure how this
> can work.
>
> What am I missing?
>
Note that the meaning of the bit values in the bitmap are slightly
surprising: 0 is pointer, 1 is non-pointer. Consequently, a bitmap of 0
means all fields are pointers.

Does this explain the confusion?

Cheers,

- Ben
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 487 bytes
Desc: not available
URL: <http://mail.haskell.org/pipermail/ghc-devs/attachments/20200220/89708f2c/attachment.sig>

From csaba.hruska at gmail.com  Sat Feb 22 15:18:40 2020
From: csaba.hruska at gmail.com (Csaba Hruska)
Date: Sat, 22 Feb 2020 16:18:40 +0100
Subject: is backport planned for 8.8.X of MR 1304?
Message-ID: <CANjChEqBpH4UwGcTjPxPw5B5YoayU+=RA3V4irCS+WdHE2d7tg@mail.gmail.com>

Hello,

Will MR 1304 <https://gitlab.haskell.org/ghc/ghc/merge_requests/1304> be
backported to GHC 8.8.X?

Thanks,
Csaba
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.haskell.org/pipermail/ghc-devs/attachments/20200222/4ba1b949/attachment.html>

From ben at smart-cactus.org  Sun Feb 23 13:33:34 2020
From: ben at smart-cactus.org (Ben Gamari)
Date: Sun, 23 Feb 2020 08:33:34 -0500
Subject: is backport planned for 8.8.X of MR 1304?
In-Reply-To: <CANjChEqBpH4UwGcTjPxPw5B5YoayU+=RA3V4irCS+WdHE2d7tg@mail.gmail.com>
References: <CANjChEqBpH4UwGcTjPxPw5B5YoayU+=RA3V4irCS+WdHE2d7tg@mail.gmail.com>
Message-ID: <1B1AC357-58B2-48BD-B6EB-82F7EF22D529@smart-cactus.org>

On February 22, 2020 10:18:40 AM EST, Csaba Hruska <csaba.hruska at gmail.com> wrote:
>Hello,
>
>Will MR 1304 <https://gitlab.haskell.org/ghc/ghc/merge_requests/1304>
>be
>backported to GHC 8.8.X?
>
>Thanks,
>Csaba

No, this will only appear in 8.12.

Cheers, 

- Ben 

From marlowsd at gmail.com  Mon Feb 24 08:21:54 2020
From: marlowsd at gmail.com (Simon Marlow)
Date: Mon, 24 Feb 2020 08:21:54 +0000
Subject: Confused about PAP object layout
In-Reply-To: <CAMQQO3nSeHAfzUJjCYvUVXCf8JwvPnVLoSO_fwMZmdgqijrpXg@mail.gmail.com>
References: <CAMQQO3=D=2y3BGQJzrk3VHy9=LV_EdYjyVnAALMuq9N=KgANbA@mail.gmail.com>
 <CANsrgP+1Ve0ahkndAoSOASU4cm-K92hJ=SXG+OpYKHX4bEuO-g@mail.gmail.com>
 <CAMQQO3kq0HxLShqdiPesnGMVnkn-AEkY3SXN8DCSSA2hxrQ7vQ@mail.gmail.com>
 <FF5887E8-5266-473B-951D-4DDD02A349E8@gmail.com>
 <CAMQQO3=bgoRY_OQ3eMJ3m7fSN4S9Wf8QieKSm6VgYBDs6=mcBw@mail.gmail.com>
 <87sgjdulmq.fsf@smart-cactus.org>
 <CAMQQO3nSeHAfzUJjCYvUVXCf8JwvPnVLoSO_fwMZmdgqijrpXg@mail.gmail.com>
Message-ID: <CANsrgPKW5D5RvLWcqDqF9GOTaraCyFmLSQq=2S0sq7b0Jfox2Q@mail.gmail.com>

On Thu, 20 Feb 2020 at 09:21, Ömer Sinan Ağacan <omeragacan at gmail.com>
wrote:

> > I'm not sure what you mean by "garbage". The bitmap merely determines
> whether
> > a field is a pointer,
>
> I think the bitmap is for liveness, not for whether a field is pointer or
> not.
> Relevant code for building an info table for a function:
>
>     mk_pieces (Fun arity (ArgGen arg_bits)) srt_label
>       = do { (liveness_lit, liveness_data) <- mkLivenessBits dflags
> arg_bits
>            ; let fun_type | null liveness_data = aRG_GEN
>                           | otherwise          = aRG_GEN_BIG
>                  extra_bits = [ packIntsCLit dflags fun_type arity ]
>                            ++ (if inlineSRT dflags then [] else [ srt_lit
> ])
>                            ++ [ liveness_lit, slow_entry ]
>            ; return (Nothing, Nothing, extra_bits, liveness_data) }
>
> This uses the word "liveness" rather than "pointers".
>
> However I just realized that the word "garbage" is still not the best way
> to
> describe what I'm trying to say. In the example
>
>     [pap_info, x, y, z]
>
> If the function's bitmap is [1, 0, 1], then `y` may be a dead (an unused
> argument, or "garbage" as I describe in my previous email) OR it may be a
> non-pointer, but used (i.e. not a garbage).
>

I don't think we ever put a zero in the bitmap for a pointer-but-not-used
argument. We don't do liveness analysis for function arguments, as far as
I'm aware. So a 0 in the bitmap always means "non-pointer".

The only reaosn the code uses the terminology "liveness" here is that it's
sharing code with the code that handles bitmaps for stack frames, which do
deal with liveness.


> So maybe "liveness" is also not the best way to describe this bitmap, as 0
> does
> not mean dead but rather "don't follow in GC".
>

> On my quest to understand and document this code better I have one more
> question. When generating info tables for functions with know argument
> patterns
> (ArgSpec) we initialize the bitmap as 0. Relevant code:
>
>     mk_pieces (Fun arity (ArgSpec fun_type)) srt_label
>       = do { let extra_bits = packIntsCLit dflags fun_type arity :
> srt_label
>            ; return (Nothing, Nothing,  extra_bits, []) }
>
> Here the last return value is for the liveness data. I don't understand
> how can
> this be correct, because when we use this function in a PAP this will
> cause NOT
> scavenging the PAP payload. Relevant code (simplified):
>
>     STATIC_INLINE GNUC_ATTR_HOT StgPtr
>     scavenge_PAP_payload (StgClosure *fun, StgClosure **payload, StgWord
> size)
>     {
>         const StgFunInfoTable *fun_info =
>             get_fun_itbl(UNTAG_CONST_CLOSURE(fun));
>
>         StgPtr p = (StgPtr)payload;
>
>         StgWord bitmap;
>         switch (fun_info->f.fun_type) {
>         ...
>
        default:
>             bitmap = BITMAP_BITS(stg_arg_bitmaps[fun_info->f.fun_type]);
>         small_bitmap:
>             p = scavenge_small_bitmap(p, size, bitmap);
>             break;
>         }
>         return p;
>     }
>

> Here if I have a function with three pointer args (ARG_PPP) the shown
> branch
> that will be taken, but because the bitmap is 0 (as shown in the mk_pieces
> code
> above) nothing in the PAPs payload will be scavenged.
>

It gets the bitmap from stg_arg_bitmaps[fun_info->f.fun_type], not from the
info table.  Hope this helps.

Cheers
Simon



>
> Here's an example from a debugging session:
>
>     >>> print pap
>     $10 = (StgPAP *) 0x42001fe030
>
>     >>> print *pap
>     $11 = {
>       header = {
>         info = 0x7fbdd1f06640 <stg_PAP_info>
>       },
>       arity = 2,
>       n_args = 1,
>       fun = 0x7fbdd2d23ffb,
>       payload = 0x42001fe048
>     }
>
> So this PAP is applied one argument, which is a boxed object (a FUN_2_0):
>
>     >>> print *get_itbl(UNTAG_CLOSURE(pap->payload[0]))
>     $20 = {
>       layout = {
>         payload = {
>           ptrs = 2,
>           nptrs = 0
>         },
>         bitmap = 2,
>         large_bitmap_offset = 2,
>         __pad_large_bitmap_offset = 2,
>         selector_offset = 2
>       },
>       type = 11,
>       srt = 1914488,
>       code = 0x7fbdd2b509c0 "H\215E\370L9\370r[I\203\304 M;\245X\003"
>     }
>
> However if I look at the function of this PAP:
>
>     >>> print *get_fun_itbl(UNTAG_CLOSURE(pap->fun))
>     $21 = {
>       f = {
>         slow_apply_offset = 16,
>         __pad_slow_apply_offset = 3135120895,
>         b = {
>           bitmap = 74900193017889,
>           bitmap_offset = 258342945,
>           __pad_bitmap_offset = 258342945
>         },
>         fun_type = 23,
>         arity = 3
>       },
>       i = {
>         layout = {
>           payload = {
>             ptrs = 0,
>             nptrs = 0
>           },
>           bitmap = 0,
>           large_bitmap_offset = 0,
>           __pad_large_bitmap_offset = 0,
>           selector_offset = 0
>         },
>         type = 14,
>         srt = 1916288,
>         code = 0x7fbdd2b50260 <base_GHCziRead_list3_info>
> "I\203\304(M;\245X\003"
>       }
>     }
>
> It has arity 3. Since the first argument is a boxed object and this
> function has
> arity 3, if the argument is actually live in the function (i.e. not an
> unused
> argument), then the bitmap should have a 1 for this. But because the
> argument
> pattern is known (ARG_PPP) we initialized the bitmap as 0! Not sure how
> this
> can work.
>
> What am I missing?
>
> Thanks,
>
> Ömer
>
> Ben Gamari <ben at smart-cactus.org>, 14 Şub 2020 Cum, 20:25 tarihinde şunu
> yazdı:
> >
> > Ömer Sinan Ağacan <omeragacan at gmail.com> writes:
> >
> > > I think that makes sense, with the invariant that n_args <=
> bitmap_size. We
> > > evacuate the arguments used by the function but not others. Thanks.
> > >
> > > It's somewhat weird to see an object with useful stuff, then garbage,
> then
> > > useful stuff again in the heap, but that's not an issue by itself. For
> example
> > > if I have something like
> > >
> > >     [pap_info, x, y, z]
> > >
> > > and according to the function `y` is dead, then after evacuating I get
> > >
> > >     [pap_info, x, <garbage>, z]
> > >
> > > This "garbage" is evacuated again and again every time we evacuate
> this PAP.
> > >
> > I'm not sure what you mean by "garbage". The bitmap merely determines
> > whether a field is a pointer, not whether it is copied during
> > evacuation. A field's bitmap bit not being set merely means that we won't
> > evacuate the value of that field during scavenging.
> >
> > Nevertheless, this all deserves a comment in scavenge_PAP.
> >
> > Cheers,
> >
> > - Ben
> >
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.haskell.org/pipermail/ghc-devs/attachments/20200224/c2edc477/attachment.html>

From simonpj at microsoft.com  Mon Feb 24 10:45:10 2020
From: simonpj at microsoft.com (Simon Peyton Jones)
Date: Mon, 24 Feb 2020 10:45:10 +0000
Subject: Confused about PAP object layout
In-Reply-To: <CANsrgPKW5D5RvLWcqDqF9GOTaraCyFmLSQq=2S0sq7b0Jfox2Q@mail.gmail.com>
References: <CAMQQO3=D=2y3BGQJzrk3VHy9=LV_EdYjyVnAALMuq9N=KgANbA@mail.gmail.com>
 <CANsrgP+1Ve0ahkndAoSOASU4cm-K92hJ=SXG+OpYKHX4bEuO-g@mail.gmail.com>
 <CAMQQO3kq0HxLShqdiPesnGMVnkn-AEkY3SXN8DCSSA2hxrQ7vQ@mail.gmail.com>
 <FF5887E8-5266-473B-951D-4DDD02A349E8@gmail.com>
 <CAMQQO3=bgoRY_OQ3eMJ3m7fSN4S9Wf8QieKSm6VgYBDs6=mcBw@mail.gmail.com>
 <87sgjdulmq.fsf@smart-cactus.org>
 <CAMQQO3nSeHAfzUJjCYvUVXCf8JwvPnVLoSO_fwMZmdgqijrpXg@mail.gmail.com>
 <CANsrgPKW5D5RvLWcqDqF9GOTaraCyFmLSQq=2S0sq7b0Jfox2Q@mail.gmail.com>
Message-ID: <VI1PR8303MB009396D8C4F99ED1B8E08256ADEC0@VI1PR8303MB0093.EURPRD83.prod.outlook.com>

I’m not following this in detail, but do please make sure that the results of this discussion end up in a suitable Note.  Obviously it’s not transparently clear as-is, and I can see clarity emerging

Thanks!

Simon

From: ghc-devs <ghc-devs-bounces at haskell.org> On Behalf Of Simon Marlow
Sent: 24 February 2020 08:22
To: Ömer Sinan Ağacan <omeragacan at gmail.com>
Cc: ghc-devs <ghc-devs at haskell.org>
Subject: Re: Confused about PAP object layout

On Thu, 20 Feb 2020 at 09:21, Ömer Sinan Ağacan <omeragacan at gmail.com<mailto:omeragacan at gmail.com>> wrote:
> I'm not sure what you mean by "garbage". The bitmap merely determines whether
> a field is a pointer,

I think the bitmap is for liveness, not for whether a field is pointer or not.
Relevant code for building an info table for a function:

    mk_pieces (Fun arity (ArgGen arg_bits)) srt_label
      = do { (liveness_lit, liveness_data) <- mkLivenessBits dflags arg_bits
           ; let fun_type | null liveness_data = aRG_GEN
                          | otherwise          = aRG_GEN_BIG
                 extra_bits = [ packIntsCLit dflags fun_type arity ]
                           ++ (if inlineSRT dflags then [] else [ srt_lit ])
                           ++ [ liveness_lit, slow_entry ]
           ; return (Nothing, Nothing, extra_bits, liveness_data) }

This uses the word "liveness" rather than "pointers".

However I just realized that the word "garbage" is still not the best way to
describe what I'm trying to say. In the example

    [pap_info, x, y, z]

If the function's bitmap is [1, 0, 1], then `y` may be a dead (an unused
argument, or "garbage" as I describe in my previous email) OR it may be a
non-pointer, but used (i.e. not a garbage).

I don't think we ever put a zero in the bitmap for a pointer-but-not-used argument. We don't do liveness analysis for function arguments, as far as I'm aware. So a 0 in the bitmap always means "non-pointer".

The only reaosn the code uses the terminology "liveness" here is that it's sharing code with the code that handles bitmaps for stack frames, which do deal with liveness.

So maybe "liveness" is also not the best way to describe this bitmap, as 0 does
not mean dead but rather "don't follow in GC".

On my quest to understand and document this code better I have one more
question. When generating info tables for functions with know argument patterns
(ArgSpec) we initialize the bitmap as 0. Relevant code:

    mk_pieces (Fun arity (ArgSpec fun_type)) srt_label
      = do { let extra_bits = packIntsCLit dflags fun_type arity : srt_label
           ; return (Nothing, Nothing,  extra_bits, []) }

Here the last return value is for the liveness data. I don't understand how can
this be correct, because when we use this function in a PAP this will cause NOT
scavenging the PAP payload. Relevant code (simplified):

    STATIC_INLINE GNUC_ATTR_HOT StgPtr
    scavenge_PAP_payload (StgClosure *fun, StgClosure **payload, StgWord size)
    {
        const StgFunInfoTable *fun_info =
            get_fun_itbl(UNTAG_CONST_CLOSURE(fun));

        StgPtr p = (StgPtr)payload;

        StgWord bitmap;
        switch (fun_info->f.fun_type) {
        ...
        default:
            bitmap = BITMAP_BITS(stg_arg_bitmaps[fun_info->f.fun_type]);
        small_bitmap:
            p = scavenge_small_bitmap(p, size, bitmap);
            break;
        }
        return p;
    }

Here if I have a function with three pointer args (ARG_PPP) the shown branch
that will be taken, but because the bitmap is 0 (as shown in the mk_pieces code
above) nothing in the PAPs payload will be scavenged.

It gets the bitmap from stg_arg_bitmaps[fun_info->f.fun_type], not from the info table.  Hope this helps.

Cheers
Simon



Here's an example from a debugging session:

    >>> print pap
    $10 = (StgPAP *) 0x42001fe030

    >>> print *pap
    $11 = {
      header = {
        info = 0x7fbdd1f06640 <stg_PAP_info>
      },
      arity = 2,
      n_args = 1,
      fun = 0x7fbdd2d23ffb,
      payload = 0x42001fe048
    }

So this PAP is applied one argument, which is a boxed object (a FUN_2_0):

    >>> print *get_itbl(UNTAG_CLOSURE(pap->payload[0]))
    $20 = {
      layout = {
        payload = {
          ptrs = 2,
          nptrs = 0
        },
        bitmap = 2,
        large_bitmap_offset = 2,
        __pad_large_bitmap_offset = 2,
        selector_offset = 2
      },
      type = 11,
      srt = 1914488,
      code = 0x7fbdd2b509c0 "H\215E\370L9\370r[I\203\304 M;\245X\003"
    }

However if I look at the function of this PAP:

    >>> print *get_fun_itbl(UNTAG_CLOSURE(pap->fun))
    $21 = {
      f = {
        slow_apply_offset = 16,
        __pad_slow_apply_offset = 3135120895,
        b = {
          bitmap = 74900193017889,
          bitmap_offset = 258342945,
          __pad_bitmap_offset = 258342945
        },
        fun_type = 23,
        arity = 3
      },
      i = {
        layout = {
          payload = {
            ptrs = 0,
            nptrs = 0
          },
          bitmap = 0,
          large_bitmap_offset = 0,
          __pad_large_bitmap_offset = 0,
          selector_offset = 0
        },
        type = 14,
        srt = 1916288,
        code = 0x7fbdd2b50260 <base_GHCziRead_list3_info>
"I\203\304(M;\245X\003"
      }
    }

It has arity 3. Since the first argument is a boxed object and this function has
arity 3, if the argument is actually live in the function (i.e. not an unused
argument), then the bitmap should have a 1 for this. But because the argument
pattern is known (ARG_PPP) we initialized the bitmap as 0! Not sure how this
can work.

What am I missing?

Thanks,

Ömer

Ben Gamari <ben at smart-cactus.org<mailto:ben at smart-cactus.org>>, 14 Şub 2020 Cum, 20:25 tarihinde şunu yazdı:
>
> Ömer Sinan Ağacan <omeragacan at gmail.com<mailto:omeragacan at gmail.com>> writes:
>
> > I think that makes sense, with the invariant that n_args <= bitmap_size. We
> > evacuate the arguments used by the function but not others. Thanks.
> >
> > It's somewhat weird to see an object with useful stuff, then garbage, then
> > useful stuff again in the heap, but that's not an issue by itself. For example
> > if I have something like
> >
> >     [pap_info, x, y, z]
> >
> > and according to the function `y` is dead, then after evacuating I get
> >
> >     [pap_info, x, <garbage>, z]
> >
> > This "garbage" is evacuated again and again every time we evacuate this PAP.
> >
> I'm not sure what you mean by "garbage". The bitmap merely determines
> whether a field is a pointer, not whether it is copied during
> evacuation. A field's bitmap bit not being set merely means that we won't
> evacuate the value of that field during scavenging.
>
> Nevertheless, this all deserves a comment in scavenge_PAP.
>
> Cheers,
>
> - Ben
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.haskell.org/pipermail/ghc-devs/attachments/20200224/03127cbe/attachment.html>

From simonpj at microsoft.com  Mon Feb 24 14:33:37 2020
From: simonpj at microsoft.com (Simon Peyton Jones)
Date: Mon, 24 Feb 2020 14:33:37 +0000
Subject: Calling an unknown function from low-level Cmm
In-Reply-To: <1A55D647-2693-4B72-BFC4-F47374E4F393@gmail.com>
References: <1A55D647-2693-4B72-BFC4-F47374E4F393@gmail.com>
Message-ID: <VI1PR8303MB0093E4F13BF9A9E881AE11D4ADEC0@VI1PR8303MB0093.EURPRD83.prod.outlook.com>

I don’t know the answer to this, but Alexis when you find out can I ask (I know I'm a broken record on this) that you write a Note to explain, with pointers from the various places you looked when you were trying to find out the answer?

Thanks!

Simon

|  -----Original Message-----
|  From: ghc-devs <ghc-devs-bounces at haskell.org> On Behalf Of Alexis King
|  Sent: 14 February 2020 19:53
|  To: ghc-devs <ghc-devs at haskell.org>
|  Subject: Calling an unknown function from low-level Cmm
|  
|  Hi all,
|  
|  I’m trying to understand how to properly call an unknown function from
|  low-level Cmm code. If I’m just applying a function to a state token, it’s
|  easy; I can just do
|  
|      R1 = io;
|      jump stg_ap_v_fast [R1];
|  
|  since the calling convention is consistent in that case. But what if my
|  function takes actual arguments? I can’t do
|  
|      R1 = fun;
|      R2 = arg;
|      jump stg_ap_p_fast [R1, R2];
|  
|  because if the calling convention doesn’t pass any arguments in registers,
|  that would be wrong. I could check if NO_ARG_REGS is defined and generate
|  different code in that situation, but that seems extreme. One option I
|  think would work would be to do
|  
|      R1 = fun;
|      Sp_adj(-2);
|      Sp(1) = arg;
|      jump RET_LBL(stg_ap_p) [R1];
|  
|  but that seems wasteful if I have the argument in a register already
|  anyway. Am I missing something?
|  
|  Thanks,
|  Alexis
|  _______________________________________________
|  ghc-devs mailing list
|  ghc-devs at haskell.org
|  https://nam06.safelinks.protection.outlook.com/?url=http%3A%2F%2Fmail.hask
|  ell.org%2Fcgi-bin%2Fmailman%2Flistinfo%2Fghc-
|  devs&amp;data=02%7C01%7Csimonpj%40microsoft.com%7Cbc07f87bae6d4bb96b8d08d7
|  b187813d%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637173067859470793&a
|  mp;sdata=uVYlDXXGfB1vN0MD%2FM%2BBsUflWvrfkYPTEFtVJk2N7Y4%3D&amp;reserved=0

From lexi.lambda at gmail.com  Mon Feb 24 14:36:47 2020
From: lexi.lambda at gmail.com (Alexis King)
Date: Mon, 24 Feb 2020 20:06:47 +0530
Subject: Calling an unknown function from low-level Cmm
In-Reply-To: <VI1PR8303MB0093E4F13BF9A9E881AE11D4ADEC0@VI1PR8303MB0093.EURPRD83.prod.outlook.com>
References: <1A55D647-2693-4B72-BFC4-F47374E4F393@gmail.com>
 <VI1PR8303MB0093E4F13BF9A9E881AE11D4ADEC0@VI1PR8303MB0093.EURPRD83.prod.outlook.com>
Message-ID: <D9BD665E-6B52-4A73-BC46-31227DC8C910@gmail.com>

> On Feb 24, 2020, at 20:03, Simon Peyton Jones <simonpj at microsoft.com> wrote:
> 
> I don’t know the answer to this, but Alexis when you find out can I ask (I know I'm a broken record on this) that you write a Note to explain, with pointers from the various places you looked when you were trying to find out the answer?

If I do find out, I will! However, I’ve interpreted the lack of response to mean there isn’t one: the only easy way to do this is to pass the arguments on the stack using one of the stg_ap closures. I’m not sure if that non-answer is worth writing someplace, and if it is, I’m not quite sure where it ought to go. (Perhaps in GHC/Cmm/Parser.y?)

From ben at well-typed.com  Mon Feb 24 18:01:21 2020
From: ben at well-typed.com (Ben Gamari)
Date: Mon, 24 Feb 2020 13:01:21 -0500
Subject: [ANNOUNCE] GHC 8.8.3 is now available
References: <87r1zz1f09.fsf@smart-cactus.org>
Message-ID: <87h7zfamqf.fsf@smart-cactus.org>


Hello everyone,

The GHC team is proud to announce the release of GHC 8.8.3. The source
distribution, binary distributions, and documentation are available at

    https://downloads.haskell.org/~ghc/8.8.3

Release notes are also available [1].

This release fixes a handful of issues affecting 8.8.2:

 - a compiler panic (#17429) due to over-zealous eta reduction

 - a linker crash on Windows (#17575)

 - the ability to bootstrap with earlier 8.8 releases has been restored
   (#17146)

 - the `directory` submodule has been updated, fixing #17598

As always, if anything looks amiss do let us know.

Happy compiling!

Cheers,

- Ben


[1] https://downloads.haskell.org/ghc/8.8.3/docs/html/users_guide/8.8.3-notes.html
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 487 bytes
Desc: not available
URL: <http://mail.haskell.org/pipermail/ghc-devs/attachments/20200224/4d2b095b/attachment.sig>

From ben at well-typed.com  Mon Feb 24 20:37:10 2020
From: ben at well-typed.com (Ben Gamari)
Date: Mon, 24 Feb 2020 15:37:10 -0500
Subject: [ANNOUNCE] GHC 8.8.3 is now available
In-Reply-To: <notmuch-sha1-f74cdd6b3f125f30ec4ea6f72dbe29b93f674488>
References: <87r1zz1f09.fsf@smart-cactus.org>
Message-ID: <87a757afin.fsf@smart-cactus.org>

Ben Gamari <ben at well-typed.com> writes:

> --text follows this line--
> <#part sign=pgpmime>
>
> Hello everyone,
>
> The GHC team is proud to announce the release of GHC 8.8.3. The source
> distribution, binary distributions, and documentation are available at
>
>     https://downloads.haskell.org/~ghc/8.8.3
>
> Release notes are also available [1].
>
Unfortunately there has been a slight mix-up in the binary distributions
associated with this release and I will need to do a re-spin. I have
pulled down the binary distributions for the time being. They should be
back in a few hours.

Many apologies for the confusion!

Cheers,

- Ben
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 487 bytes
Desc: not available
URL: <http://mail.haskell.org/pipermail/ghc-devs/attachments/20200224/b0d67f45/attachment.sig>

From ben at well-typed.com  Tue Feb 25 01:51:30 2020
From: ben at well-typed.com (Ben Gamari)
Date: Mon, 24 Feb 2020 20:51:30 -0500
Subject: [ANNOUNCE] GHC 8.8.3 is now available
In-Reply-To: <87a757afin.fsf@smart-cactus.org>
References: <87r1zz1f09.fsf@smart-cactus.org> <87a757afin.fsf@smart-cactus.org>
Message-ID: <877e0ba0yq.fsf@smart-cactus.org>

Ben Gamari <ben at well-typed.com> writes:

> Ben Gamari <ben at well-typed.com> writes:
>
>> --text follows this line--
>> <#part sign=pgpmime>
>>
>> Hello everyone,
>>
>> The GHC team is proud to announce the release of GHC 8.8.3. The source
>> distribution, binary distributions, and documentation are available at
>>
>>     https://downloads.haskell.org/~ghc/8.8.3
>>
>> Release notes are also available [1].
>>
> Unfortunately there has been a slight mix-up in the binary distributions
> associated with this release and I will need to do a re-spin. I have
> pulled down the binary distributions for the time being. They should be
> back in a few hours.
>
> Many apologies for the confusion!
>
The issues with the release artifacts have been sorted out. Note that
these are new artifacts built from
d0bab2e3419e49cdbb1201d4650572b57f33420c, which is the commit pointed to
by the ghc-8.8.3-release tag.

While we typically try hard to avoid changing artifacts after they are
announced, given how quickly this issue was caught we thought this was
the least bad option.

Again, my apologies for the confusion and thanks for the patience.

Cheers,

- Ben

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 487 bytes
Desc: not available
URL: <http://mail.haskell.org/pipermail/ghc-devs/attachments/20200224/53a53963/attachment.sig>

From juhpetersen at gmail.com  Tue Feb 25 03:10:53 2020
From: juhpetersen at gmail.com (Jens Petersen)
Date: Tue, 25 Feb 2020 11:10:53 +0800
Subject: [ANNOUNCE] GHC 8.8.3 is now available
In-Reply-To: <87h7zfamqf.fsf@smart-cactus.org>
References: <87r1zz1f09.fsf@smart-cactus.org> <87h7zfamqf.fsf@smart-cactus.org>
Message-ID: <CAEDGBqLRY+cDtoMoPwrj5Ez3bEDx8A=H9NLk=s2VYnLhz4oq5A@mail.gmail.com>

Thank you for the release.

On Tue, 25 Feb 2020 at 02:02, Ben Gamari <ben at well-typed.com> wrote:

> [1]
> https://downloads.haskell.org/ghc/8.8.3/docs/html/users_guide/8.8.3-notes.html
>

I am getting 404 from the docs.

Cheers, Jens
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.haskell.org/pipermail/ghc-devs/attachments/20200225/1a29b0ba/attachment.html>

From ben at well-typed.com  Tue Feb 25 06:16:20 2020
From: ben at well-typed.com (Ben Gamari)
Date: Tue, 25 Feb 2020 01:16:20 -0500
Subject: [Haskell-cafe] [ANNOUNCE] GHC 8.8.3 is now available
In-Reply-To: <CAEDGBqLRY+cDtoMoPwrj5Ez3bEDx8A=H9NLk=s2VYnLhz4oq5A@mail.gmail.com>
References: <87r1zz1f09.fsf@smart-cactus.org> <87h7zfamqf.fsf@smart-cactus.org>
 <CAEDGBqLRY+cDtoMoPwrj5Ez3bEDx8A=H9NLk=s2VYnLhz4oq5A@mail.gmail.com>
Message-ID: <87y2sr8a4u.fsf@smart-cactus.org>

Jens Petersen <juhpetersen at gmail.com> writes:

> Thank you for the release.
>
> On Tue, 25 Feb 2020 at 02:02, Ben Gamari <ben at well-typed.com> wrote:
>
>> [1]
>> https://downloads.haskell.org/ghc/8.8.3/docs/html/users_guide/8.8.3-notes.html
>>
>
> I am getting 404 from the docs.

Hmm, this appears to be a CDN issue given that the file exists on the
server and

    https://downloads.haskell.org/ghc/8.8.3/docs/html/users_guide//8.8.3-notes.html

works. This is quite odd given that I am quite aggressive in purging
the CDN at this point.

I'll try to investigate tomorrow.

Cheers,

- Ben
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 487 bytes
Desc: not available
URL: <http://mail.haskell.org/pipermail/ghc-devs/attachments/20200225/730004ae/attachment.sig>

From klebinger.andreas at gmx.at  Tue Feb 25 13:29:28 2020
From: klebinger.andreas at gmx.at (Andreas Klebinger)
Date: Tue, 25 Feb 2020 14:29:28 +0100
Subject: Windows build broken
Message-ID: <02e99bac-e3dd-5c7b-a264-3500ae932854@gmx.at>

Hi devs,

it seems the windows build is broken. (Can't build stage2 locally).

Quickest way to reproduce is a validate of master.

It also happens on CI: https://gitlab.haskell.org/ghc/ghc/-/jobs/270248

I remember ben mentioning something along the lines of builds with "old"
(8.6) boot compilers being broken.
Is that the culprint? How do I get back to a functioning environment?

Cheers,
Andreas

From ben at smart-cactus.org  Wed Feb 26 00:59:28 2020
From: ben at smart-cactus.org (Ben Gamari)
Date: Tue, 25 Feb 2020 19:59:28 -0500
Subject: Windows build broken
In-Reply-To: <02e99bac-e3dd-5c7b-a264-3500ae932854@gmx.at>
References: <02e99bac-e3dd-5c7b-a264-3500ae932854@gmx.at>
Message-ID: <87r1yi88pg.fsf@smart-cactus.org>

Andreas Klebinger <klebinger.andreas at gmx.at> writes:

> Hi devs,
>
> it seems the windows build is broken. (Can't build stage2 locally).
>
> Quickest way to reproduce is a validate of master.
>
> It also happens on CI: https://gitlab.haskell.org/ghc/ghc/-/jobs/270248
>
> I remember ben mentioning something along the lines of builds with "old"
> (8.6) boot compilers being broken.
> Is that the culprint? How do I get back to a functioning environment?
>
Indeed this is due to #17861. This should be fixed by !2773 which is in
Marge's current batch.

Cheers,

- Ben
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 487 bytes
Desc: not available
URL: <http://mail.haskell.org/pipermail/ghc-devs/attachments/20200225/e93013d2/attachment.sig>

From jaskiewiczs at icloud.com  Wed Feb 26 08:32:41 2020
From: jaskiewiczs at icloud.com (Sergej Jaskiewicz)
Date: Wed, 26 Feb 2020 11:32:41 +0300
Subject: Time-profiling the compiler pipeline
Message-ID: <A4F5BD27-4552-4ED9-AAD2-0ECAA0E54A1A@icloud.com>

Hello everyone,

Recently I've been exploring ways to improve the LLVM support in GHC.

While doing that, I've encountered an issue that I think is worth looking into.
Namely, the compilation times with LLVM are considerably longer than with the native
code generator. To be more concrete, on my machine I've managed to build
the whole toolchain in just half an hour with NCG, but the same build with
LLVM-powered GHC took several hours.

If we want to improve compilation times, we should be able to measure it first.
As far as I understand, the only way for profiling the compiler pipeline today
is building GHC with special flags: https://gitlab.haskell.org/ghc/ghc/wikis/debugging/profiling-ghc

This has three important downsides:
   * We need build the whole compiler ourselves. If I'm just a user interested in why
     my program is so slow to compile and how I should fix it, this is too much for me
     to ask to do.
   * There's just too much irrelevant data in the output. For example, it doesn't
     tell me which part of my Haskell code is taking so long to compile.
     It would be nice if the report contained all the metadata that I need to
     go and fix my code.
   * If we're using LLVM, we're pretty much out of luck, because we're using
     an existing LLVM binary which is likely not built for profiling.

There have been the same problem with the Clang compiler until recently. In Clang 9,
which was released in the last year, there's a new flag '-ftime-trace' that produces
a beautiful flame chart with all the metadata needed to identify bottlenecks.
This flag tells Clang to produce a JSON file that can be visualized
in Chrome's chrome://tracing page, or on https://www.speedscope.app.
This is described in more details here:
https://aras-p.info/blog/2019/01/16/time-trace-timeline-flame-chart-profiler-for-Clang/

I'm thinking about implementing similar functionality in GHC.
One option would be to utilize the plugin infrastructure.

Maybe I'm missing something and there are better options?

— Sergej.

From cheng.shao at tweag.io  Wed Feb 26 15:02:19 2020
From: cheng.shao at tweag.io (Shao, Cheng)
Date: Wed, 26 Feb 2020 16:02:19 +0100
Subject: Blocking MVar# primops not performing stack checks?
Message-ID: <CALHrtF=FsTnHNos8E83iR0iy40KT13yqmJMbLCXZq1Mzj1LBhg@mail.gmail.com>

Hi all,

When an MVar# primop blocks, it jumps to a function in
HeapStackCheck.cmm which pushes a RET_SMALL stack frame before
returning to the scheduler (e.g. the takeMVar# primop jumps to
stg_block_takemvar for stack adjustment). But these functions directly
bump Sp without checking for possible stack overflow, I wonder if it
is a bug?

Cheers,
Cheng

From ben at smart-cactus.org  Wed Feb 26 18:21:56 2020
From: ben at smart-cactus.org (Ben Gamari)
Date: Wed, 26 Feb 2020 13:21:56 -0500
Subject: Time-profiling the compiler pipeline
In-Reply-To: <A4F5BD27-4552-4ED9-AAD2-0ECAA0E54A1A@icloud.com>
References: <A4F5BD27-4552-4ED9-AAD2-0ECAA0E54A1A@icloud.com>
Message-ID: <87mu958b0g.fsf@smart-cactus.org>

Sergej Jaskiewicz via ghc-devs <ghc-devs at haskell.org> writes:

> Hello everyone,
>
> Recently I've been exploring ways to improve the LLVM support in GHC.
>
> While doing that, I've encountered an issue that I think is worth looking into.
> Namely, the compilation times with LLVM are considerably longer than with the native
> code generator. To be more concrete, on my machine I've managed to build
> the whole toolchain in just half an hour with NCG, but the same build with
> LLVM-powered GHC took several hours.
>
Indeed this is reflective of my experience as well. This is indeed a
major problem for architectures like ARM which are only supported via
the LLVM backend.

> If we want to improve compilation times, we should be able to measure it first.
> As far as I understand, the only way for profiling the compiler pipeline today
> is building GHC with special flags: https://gitlab.haskell.org/ghc/ghc/wikis/debugging/profiling-ghc
>
This page uses "profiling" in the sense of "cost-center profiler".
However, as you point out, many performance analysis tasks do not
require the cost-center profiler.

> This has three important downsides:
>    * We need build the whole compiler ourselves. If I'm just a user interested in why
>      my program is so slow to compile and how I should fix it, this is too much for me
>      to ask to do.
>    * There's just too much irrelevant data in the output. For example, it doesn't
>      tell me which part of my Haskell code is taking so long to compile.
>      It would be nice if the report contained all the metadata that I need to
>      go and fix my code.
>    * If we're using LLVM, we're pretty much out of luck, because we're using
>      an existing LLVM binary which is likely not built for profiling.
>
> There have been the same problem with the Clang compiler until recently. In Clang 9,
> which was released in the last year, there's a new flag '-ftime-trace' that produces
> a beautiful flame chart with all the metadata needed to identify bottlenecks.
> This flag tells Clang to produce a JSON file that can be visualized
> in Chrome's chrome://tracing page, or on https://www.speedscope.app.
> This is described in more details here:
> https://aras-p.info/blog/2019/01/16/time-trace-timeline-flame-chart-profiler-for-Clang/
>
> I'm thinking about implementing similar functionality in GHC.
> One option would be to utilize the plugin infrastructure.
>
> Maybe I'm missing something and there are better options?
>
We already have such a flag: -ddump-timings [1]. Moreover, the timings are
also emitted in GHC's eventlog if you link the `ghc` executable with
`-eventlog`. Alp wrote a blog post [2] describing his experiences using all
of this a few months ago.

I have also had some amount of luck using the Linux `perf` tool to
profile GHC.

Cheers,

- Ben


[1] https://ghc.gitlab.haskell.org/ghc/doc/users_guide/debugging.html#ghc-flag--ddump-timings
[2] https://www.haskell.org/ghc/blog/20190924-eventful-ghc.html
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 487 bytes
Desc: not available
URL: <http://mail.haskell.org/pipermail/ghc-devs/attachments/20200226/fd9d692f/attachment.sig>

From omeragacan at gmail.com  Wed Feb 26 18:48:01 2020
From: omeragacan at gmail.com (=?UTF-8?Q?=C3=96mer_Sinan_A=C4=9Facan?=)
Date: Wed, 26 Feb 2020 21:48:01 +0300
Subject: Confused about PAP object layout
In-Reply-To: <VI1PR8303MB009396D8C4F99ED1B8E08256ADEC0@VI1PR8303MB0093.EURPRD83.prod.outlook.com>
References: <CAMQQO3=D=2y3BGQJzrk3VHy9=LV_EdYjyVnAALMuq9N=KgANbA@mail.gmail.com>
 <CANsrgP+1Ve0ahkndAoSOASU4cm-K92hJ=SXG+OpYKHX4bEuO-g@mail.gmail.com>
 <CAMQQO3kq0HxLShqdiPesnGMVnkn-AEkY3SXN8DCSSA2hxrQ7vQ@mail.gmail.com>
 <FF5887E8-5266-473B-951D-4DDD02A349E8@gmail.com>
 <CAMQQO3=bgoRY_OQ3eMJ3m7fSN4S9Wf8QieKSm6VgYBDs6=mcBw@mail.gmail.com>
 <87sgjdulmq.fsf@smart-cactus.org>
 <CAMQQO3nSeHAfzUJjCYvUVXCf8JwvPnVLoSO_fwMZmdgqijrpXg@mail.gmail.com>
 <CANsrgPKW5D5RvLWcqDqF9GOTaraCyFmLSQq=2S0sq7b0Jfox2Q@mail.gmail.com>
 <VI1PR8303MB009396D8C4F99ED1B8E08256ADEC0@VI1PR8303MB0093.EURPRD83.prod.outlook.com>
Message-ID: <CAMQQO3=QwVdknFq17vHR=REupWnpqo6mnOEFVaVNK5zO-A=4zg@mail.gmail.com>

So the key points from this thread are:

- PAP payloads are scavenged using the function's bitmap. Because a PAPs payload
  will have less number of closures than the function's arity the bitmap will
  always have enough bits.

- A bit in a function bitmap is NOT for liveness (e.g. does not indicate whether
  an argument used or not), but for pointers vs. non-pointers. Function bitmaps
  are called "liveness bits" in the code generator which is misleading.

- In a function bitmap (small or large), 0 means pointer, 1 means non-pointer.

  This is really what confused me in my last email above. For some reason I
  intuitively expected 1 to mean pointer, not 0. Simon M also got this wrong
  ("So a 0 in the bitmap always means non-pointer.") so maybe this is confusing
  to others too.

- For functions with known argument patterns we don't use the function's bitmap.
  These function's type are greater than ARG_BCO (2), and for those we use the
  stg_arg_bitmaps array to get the bitmap.

  For example, the bitmap for ARG_PPP (function with 3 pointer arguments) is at
  index 23 in this array, which is 0b11. For ARG_PNN it's 0b110000011. The least
  significant 6 bits are for the size (3), the remaining 0b110 means the first
  argument is a pointer, rest of the two are non-pointers.

I still don't understand why this assertion

    ASSERT(BITMAP_SIZE(bitmap) >= size);

I added to scavenge_small_bitmap in !2727 is failing though.

Ömer

Simon Peyton Jones <simonpj at microsoft.com>, 24 Şub 2020 Pzt, 13:45
tarihinde şunu yazdı:
>
> I’m not following this in detail, but do please make sure that the results of this discussion end up in a suitable Note.  Obviously it’s not transparently clear as-is, and I can see clarity emerging
>
>
>
> Thanks!
>
>
> Simon
>
>
>
> From: ghc-devs <ghc-devs-bounces at haskell.org> On Behalf Of Simon Marlow
> Sent: 24 February 2020 08:22
> To: Ömer Sinan Ağacan <omeragacan at gmail.com>
> Cc: ghc-devs <ghc-devs at haskell.org>
> Subject: Re: Confused about PAP object layout
>
>
>
> On Thu, 20 Feb 2020 at 09:21, Ömer Sinan Ağacan <omeragacan at gmail.com> wrote:
>
> > I'm not sure what you mean by "garbage". The bitmap merely determines whether
> > a field is a pointer,
>
> I think the bitmap is for liveness, not for whether a field is pointer or not.
> Relevant code for building an info table for a function:
>
>     mk_pieces (Fun arity (ArgGen arg_bits)) srt_label
>       = do { (liveness_lit, liveness_data) <- mkLivenessBits dflags arg_bits
>            ; let fun_type | null liveness_data = aRG_GEN
>                           | otherwise          = aRG_GEN_BIG
>                  extra_bits = [ packIntsCLit dflags fun_type arity ]
>                            ++ (if inlineSRT dflags then [] else [ srt_lit ])
>                            ++ [ liveness_lit, slow_entry ]
>            ; return (Nothing, Nothing, extra_bits, liveness_data) }
>
> This uses the word "liveness" rather than "pointers".
>
> However I just realized that the word "garbage" is still not the best way to
> describe what I'm trying to say. In the example
>
>     [pap_info, x, y, z]
>
> If the function's bitmap is [1, 0, 1], then `y` may be a dead (an unused
> argument, or "garbage" as I describe in my previous email) OR it may be a
> non-pointer, but used (i.e. not a garbage).
>
>
>
> I don't think we ever put a zero in the bitmap for a pointer-but-not-used argument. We don't do liveness analysis for function arguments, as far as I'm aware. So a 0 in the bitmap always means "non-pointer".
>
>
>
> The only reaosn the code uses the terminology "liveness" here is that it's sharing code with the code that handles bitmaps for stack frames, which do deal with liveness.
>
>
>
> So maybe "liveness" is also not the best way to describe this bitmap, as 0 does
> not mean dead but rather "don't follow in GC".
>
>
> On my quest to understand and document this code better I have one more
> question. When generating info tables for functions with know argument patterns
> (ArgSpec) we initialize the bitmap as 0. Relevant code:
>
>     mk_pieces (Fun arity (ArgSpec fun_type)) srt_label
>       = do { let extra_bits = packIntsCLit dflags fun_type arity : srt_label
>            ; return (Nothing, Nothing,  extra_bits, []) }
>
> Here the last return value is for the liveness data. I don't understand how can
> this be correct, because when we use this function in a PAP this will cause NOT
> scavenging the PAP payload. Relevant code (simplified):
>
>     STATIC_INLINE GNUC_ATTR_HOT StgPtr
>     scavenge_PAP_payload (StgClosure *fun, StgClosure **payload, StgWord size)
>     {
>         const StgFunInfoTable *fun_info =
>             get_fun_itbl(UNTAG_CONST_CLOSURE(fun));
>
>         StgPtr p = (StgPtr)payload;
>
>         StgWord bitmap;
>         switch (fun_info->f.fun_type) {
>         ...
>
>         default:
>             bitmap = BITMAP_BITS(stg_arg_bitmaps[fun_info->f.fun_type]);
>         small_bitmap:
>             p = scavenge_small_bitmap(p, size, bitmap);
>             break;
>         }
>         return p;
>     }
>
>
> Here if I have a function with three pointer args (ARG_PPP) the shown branch
> that will be taken, but because the bitmap is 0 (as shown in the mk_pieces code
> above) nothing in the PAPs payload will be scavenged.
>
>
>
> It gets the bitmap from stg_arg_bitmaps[fun_info->f.fun_type], not from the info table.  Hope this helps.
>
>
>
> Cheers
>
> Simon
>
>
>
>
>
>
> Here's an example from a debugging session:
>
>     >>> print pap
>     $10 = (StgPAP *) 0x42001fe030
>
>     >>> print *pap
>     $11 = {
>       header = {
>         info = 0x7fbdd1f06640 <stg_PAP_info>
>       },
>       arity = 2,
>       n_args = 1,
>       fun = 0x7fbdd2d23ffb,
>       payload = 0x42001fe048
>     }
>
> So this PAP is applied one argument, which is a boxed object (a FUN_2_0):
>
>     >>> print *get_itbl(UNTAG_CLOSURE(pap->payload[0]))
>     $20 = {
>       layout = {
>         payload = {
>           ptrs = 2,
>           nptrs = 0
>         },
>         bitmap = 2,
>         large_bitmap_offset = 2,
>         __pad_large_bitmap_offset = 2,
>         selector_offset = 2
>       },
>       type = 11,
>       srt = 1914488,
>       code = 0x7fbdd2b509c0 "H\215E\370L9\370r[I\203\304 M;\245X\003"
>     }
>
> However if I look at the function of this PAP:
>
>     >>> print *get_fun_itbl(UNTAG_CLOSURE(pap->fun))
>     $21 = {
>       f = {
>         slow_apply_offset = 16,
>         __pad_slow_apply_offset = 3135120895,
>         b = {
>           bitmap = 74900193017889,
>           bitmap_offset = 258342945,
>           __pad_bitmap_offset = 258342945
>         },
>         fun_type = 23,
>         arity = 3
>       },
>       i = {
>         layout = {
>           payload = {
>             ptrs = 0,
>             nptrs = 0
>           },
>           bitmap = 0,
>           large_bitmap_offset = 0,
>           __pad_large_bitmap_offset = 0,
>           selector_offset = 0
>         },
>         type = 14,
>         srt = 1916288,
>         code = 0x7fbdd2b50260 <base_GHCziRead_list3_info>
> "I\203\304(M;\245X\003"
>       }
>     }
>
> It has arity 3. Since the first argument is a boxed object and this function has
> arity 3, if the argument is actually live in the function (i.e. not an unused
> argument), then the bitmap should have a 1 for this. But because the argument
> pattern is known (ARG_PPP) we initialized the bitmap as 0! Not sure how this
> can work.
>
> What am I missing?
>
> Thanks,
>
> Ömer
>
> Ben Gamari <ben at smart-cactus.org>, 14 Şub 2020 Cum, 20:25 tarihinde şunu yazdı:
> >
> > Ömer Sinan Ağacan <omeragacan at gmail.com> writes:
> >
> > > I think that makes sense, with the invariant that n_args <= bitmap_size. We
> > > evacuate the arguments used by the function but not others. Thanks.
> > >
> > > It's somewhat weird to see an object with useful stuff, then garbage, then
> > > useful stuff again in the heap, but that's not an issue by itself. For example
> > > if I have something like
> > >
> > >     [pap_info, x, y, z]
> > >
> > > and according to the function `y` is dead, then after evacuating I get
> > >
> > >     [pap_info, x, <garbage>, z]
> > >
> > > This "garbage" is evacuated again and again every time we evacuate this PAP.
> > >
> > I'm not sure what you mean by "garbage". The bitmap merely determines
> > whether a field is a pointer, not whether it is copied during
> > evacuation. A field's bitmap bit not being set merely means that we won't
> > evacuate the value of that field during scavenging.
> >
> > Nevertheless, this all deserves a comment in scavenge_PAP.
> >
> > Cheers,
> >
> > - Ben
> >

From omeragacan at gmail.com  Wed Feb 26 18:52:48 2020
From: omeragacan at gmail.com (=?UTF-8?Q?=C3=96mer_Sinan_A=C4=9Facan?=)
Date: Wed, 26 Feb 2020 21:52:48 +0300
Subject: Confused about PAP object layout
In-Reply-To: <CAMQQO3=QwVdknFq17vHR=REupWnpqo6mnOEFVaVNK5zO-A=4zg@mail.gmail.com>
References: <CAMQQO3=D=2y3BGQJzrk3VHy9=LV_EdYjyVnAALMuq9N=KgANbA@mail.gmail.com>
 <CANsrgP+1Ve0ahkndAoSOASU4cm-K92hJ=SXG+OpYKHX4bEuO-g@mail.gmail.com>
 <CAMQQO3kq0HxLShqdiPesnGMVnkn-AEkY3SXN8DCSSA2hxrQ7vQ@mail.gmail.com>
 <FF5887E8-5266-473B-951D-4DDD02A349E8@gmail.com>
 <CAMQQO3=bgoRY_OQ3eMJ3m7fSN4S9Wf8QieKSm6VgYBDs6=mcBw@mail.gmail.com>
 <87sgjdulmq.fsf@smart-cactus.org>
 <CAMQQO3nSeHAfzUJjCYvUVXCf8JwvPnVLoSO_fwMZmdgqijrpXg@mail.gmail.com>
 <CANsrgPKW5D5RvLWcqDqF9GOTaraCyFmLSQq=2S0sq7b0Jfox2Q@mail.gmail.com>
 <VI1PR8303MB009396D8C4F99ED1B8E08256ADEC0@VI1PR8303MB0093.EURPRD83.prod.outlook.com>
 <CAMQQO3=QwVdknFq17vHR=REupWnpqo6mnOEFVaVNK5zO-A=4zg@mail.gmail.com>
Message-ID: <CAMQQO3mqdbhOPWagGm3sgJ_n=ZFSVCTs2e9rO4X=3e1wjosJSw@mail.gmail.com>

> I still don't understand why this assertion
>
>     ASSERT(BITMAP_SIZE(bitmap) >= size);
>
> I added to scavenge_small_bitmap in !2727 is failing though.

Ahh, this is becuase in the call sites we do a bit shift only pass the contents
of the bitmap, without the size:

    bitmap = BITMAP_BITS(stg_arg_bitmaps[fun_info->f.fun_type]);
    p = scavenge_small_bitmap(p, size, bitmap);

BITMAP_BITS is the macro that does this.

Ömer

Ömer Sinan Ağacan <omeragacan at gmail.com>, 26 Şub 2020 Çar, 21:48
tarihinde şunu yazdı:
>
> So the key points from this thread are:
>
> - PAP payloads are scavenged using the function's bitmap. Because a PAPs payload
>   will have less number of closures than the function's arity the bitmap will
>   always have enough bits.
>
> - A bit in a function bitmap is NOT for liveness (e.g. does not indicate whether
>   an argument used or not), but for pointers vs. non-pointers. Function bitmaps
>   are called "liveness bits" in the code generator which is misleading.
>
> - In a function bitmap (small or large), 0 means pointer, 1 means non-pointer.
>
>   This is really what confused me in my last email above. For some reason I
>   intuitively expected 1 to mean pointer, not 0. Simon M also got this wrong
>   ("So a 0 in the bitmap always means non-pointer.") so maybe this is confusing
>   to others too.
>
> - For functions with known argument patterns we don't use the function's bitmap.
>   These function's type are greater than ARG_BCO (2), and for those we use the
>   stg_arg_bitmaps array to get the bitmap.
>
>   For example, the bitmap for ARG_PPP (function with 3 pointer arguments) is at
>   index 23 in this array, which is 0b11. For ARG_PNN it's 0b110000011. The least
>   significant 6 bits are for the size (3), the remaining 0b110 means the first
>   argument is a pointer, rest of the two are non-pointers.
>
> I still don't understand why this assertion
>
>     ASSERT(BITMAP_SIZE(bitmap) >= size);
>
> I added to scavenge_small_bitmap in !2727 is failing though.
>
> Ömer
>
> Simon Peyton Jones <simonpj at microsoft.com>, 24 Şub 2020 Pzt, 13:45
> tarihinde şunu yazdı:
> >
> > I’m not following this in detail, but do please make sure that the results of this discussion end up in a suitable Note.  Obviously it’s not transparently clear as-is, and I can see clarity emerging
> >
> >
> >
> > Thanks!
> >
> >
> > Simon
> >
> >
> >
> > From: ghc-devs <ghc-devs-bounces at haskell.org> On Behalf Of Simon Marlow
> > Sent: 24 February 2020 08:22
> > To: Ömer Sinan Ağacan <omeragacan at gmail.com>
> > Cc: ghc-devs <ghc-devs at haskell.org>
> > Subject: Re: Confused about PAP object layout
> >
> >
> >
> > On Thu, 20 Feb 2020 at 09:21, Ömer Sinan Ağacan <omeragacan at gmail.com> wrote:
> >
> > > I'm not sure what you mean by "garbage". The bitmap merely determines whether
> > > a field is a pointer,
> >
> > I think the bitmap is for liveness, not for whether a field is pointer or not.
> > Relevant code for building an info table for a function:
> >
> >     mk_pieces (Fun arity (ArgGen arg_bits)) srt_label
> >       = do { (liveness_lit, liveness_data) <- mkLivenessBits dflags arg_bits
> >            ; let fun_type | null liveness_data = aRG_GEN
> >                           | otherwise          = aRG_GEN_BIG
> >                  extra_bits = [ packIntsCLit dflags fun_type arity ]
> >                            ++ (if inlineSRT dflags then [] else [ srt_lit ])
> >                            ++ [ liveness_lit, slow_entry ]
> >            ; return (Nothing, Nothing, extra_bits, liveness_data) }
> >
> > This uses the word "liveness" rather than "pointers".
> >
> > However I just realized that the word "garbage" is still not the best way to
> > describe what I'm trying to say. In the example
> >
> >     [pap_info, x, y, z]
> >
> > If the function's bitmap is [1, 0, 1], then `y` may be a dead (an unused
> > argument, or "garbage" as I describe in my previous email) OR it may be a
> > non-pointer, but used (i.e. not a garbage).
> >
> >
> >
> > I don't think we ever put a zero in the bitmap for a pointer-but-not-used argument. We don't do liveness analysis for function arguments, as far as I'm aware. So a 0 in the bitmap always means "non-pointer".
> >
> >
> >
> > The only reaosn the code uses the terminology "liveness" here is that it's sharing code with the code that handles bitmaps for stack frames, which do deal with liveness.
> >
> >
> >
> > So maybe "liveness" is also not the best way to describe this bitmap, as 0 does
> > not mean dead but rather "don't follow in GC".
> >
> >
> > On my quest to understand and document this code better I have one more
> > question. When generating info tables for functions with know argument patterns
> > (ArgSpec) we initialize the bitmap as 0. Relevant code:
> >
> >     mk_pieces (Fun arity (ArgSpec fun_type)) srt_label
> >       = do { let extra_bits = packIntsCLit dflags fun_type arity : srt_label
> >            ; return (Nothing, Nothing,  extra_bits, []) }
> >
> > Here the last return value is for the liveness data. I don't understand how can
> > this be correct, because when we use this function in a PAP this will cause NOT
> > scavenging the PAP payload. Relevant code (simplified):
> >
> >     STATIC_INLINE GNUC_ATTR_HOT StgPtr
> >     scavenge_PAP_payload (StgClosure *fun, StgClosure **payload, StgWord size)
> >     {
> >         const StgFunInfoTable *fun_info =
> >             get_fun_itbl(UNTAG_CONST_CLOSURE(fun));
> >
> >         StgPtr p = (StgPtr)payload;
> >
> >         StgWord bitmap;
> >         switch (fun_info->f.fun_type) {
> >         ...
> >
> >         default:
> >             bitmap = BITMAP_BITS(stg_arg_bitmaps[fun_info->f.fun_type]);
> >         small_bitmap:
> >             p = scavenge_small_bitmap(p, size, bitmap);
> >             break;
> >         }
> >         return p;
> >     }
> >
> >
> > Here if I have a function with three pointer args (ARG_PPP) the shown branch
> > that will be taken, but because the bitmap is 0 (as shown in the mk_pieces code
> > above) nothing in the PAPs payload will be scavenged.
> >
> >
> >
> > It gets the bitmap from stg_arg_bitmaps[fun_info->f.fun_type], not from the info table.  Hope this helps.
> >
> >
> >
> > Cheers
> >
> > Simon
> >
> >
> >
> >
> >
> >
> > Here's an example from a debugging session:
> >
> >     >>> print pap
> >     $10 = (StgPAP *) 0x42001fe030
> >
> >     >>> print *pap
> >     $11 = {
> >       header = {
> >         info = 0x7fbdd1f06640 <stg_PAP_info>
> >       },
> >       arity = 2,
> >       n_args = 1,
> >       fun = 0x7fbdd2d23ffb,
> >       payload = 0x42001fe048
> >     }
> >
> > So this PAP is applied one argument, which is a boxed object (a FUN_2_0):
> >
> >     >>> print *get_itbl(UNTAG_CLOSURE(pap->payload[0]))
> >     $20 = {
> >       layout = {
> >         payload = {
> >           ptrs = 2,
> >           nptrs = 0
> >         },
> >         bitmap = 2,
> >         large_bitmap_offset = 2,
> >         __pad_large_bitmap_offset = 2,
> >         selector_offset = 2
> >       },
> >       type = 11,
> >       srt = 1914488,
> >       code = 0x7fbdd2b509c0 "H\215E\370L9\370r[I\203\304 M;\245X\003"
> >     }
> >
> > However if I look at the function of this PAP:
> >
> >     >>> print *get_fun_itbl(UNTAG_CLOSURE(pap->fun))
> >     $21 = {
> >       f = {
> >         slow_apply_offset = 16,
> >         __pad_slow_apply_offset = 3135120895,
> >         b = {
> >           bitmap = 74900193017889,
> >           bitmap_offset = 258342945,
> >           __pad_bitmap_offset = 258342945
> >         },
> >         fun_type = 23,
> >         arity = 3
> >       },
> >       i = {
> >         layout = {
> >           payload = {
> >             ptrs = 0,
> >             nptrs = 0
> >           },
> >           bitmap = 0,
> >           large_bitmap_offset = 0,
> >           __pad_large_bitmap_offset = 0,
> >           selector_offset = 0
> >         },
> >         type = 14,
> >         srt = 1916288,
> >         code = 0x7fbdd2b50260 <base_GHCziRead_list3_info>
> > "I\203\304(M;\245X\003"
> >       }
> >     }
> >
> > It has arity 3. Since the first argument is a boxed object and this function has
> > arity 3, if the argument is actually live in the function (i.e. not an unused
> > argument), then the bitmap should have a 1 for this. But because the argument
> > pattern is known (ARG_PPP) we initialized the bitmap as 0! Not sure how this
> > can work.
> >
> > What am I missing?
> >
> > Thanks,
> >
> > Ömer
> >
> > Ben Gamari <ben at smart-cactus.org>, 14 Şub 2020 Cum, 20:25 tarihinde şunu yazdı:
> > >
> > > Ömer Sinan Ağacan <omeragacan at gmail.com> writes:
> > >
> > > > I think that makes sense, with the invariant that n_args <= bitmap_size. We
> > > > evacuate the arguments used by the function but not others. Thanks.
> > > >
> > > > It's somewhat weird to see an object with useful stuff, then garbage, then
> > > > useful stuff again in the heap, but that's not an issue by itself. For example
> > > > if I have something like
> > > >
> > > >     [pap_info, x, y, z]
> > > >
> > > > and according to the function `y` is dead, then after evacuating I get
> > > >
> > > >     [pap_info, x, <garbage>, z]
> > > >
> > > > This "garbage" is evacuated again and again every time we evacuate this PAP.
> > > >
> > > I'm not sure what you mean by "garbage". The bitmap merely determines
> > > whether a field is a pointer, not whether it is copied during
> > > evacuation. A field's bitmap bit not being set merely means that we won't
> > > evacuate the value of that field during scavenging.
> > >
> > > Nevertheless, this all deserves a comment in scavenge_PAP.
> > >
> > > Cheers,
> > >
> > > - Ben
> > >

From ts33n.sh3 at gmail.com  Wed Feb 26 20:53:05 2020
From: ts33n.sh3 at gmail.com (Tseen She)
Date: Wed, 26 Feb 2020 20:53:05 +0000
Subject: ghc 8.8.x change in behaviour when loading in-memory modules
Message-ID: <CACKRAT2qoqx4Z24ES7e2uNM-sfv72eVtxB0VzSq1qPMzQ2To1Q@mail.gmail.com>

Dear ghc devs,

I am in the process of updating my dev tool, which uses the ghc api, to
support ghc-8.8.x.

However, I have noticed that one of my regression tests is failing, and I
have isolated the problem to a change in behaviour in typecheckModule when
using a Target with an in-memory buffer (i.e. targetContents is a Just).

Using debug tracing, I can confirm that in the following code, the
ms_hspp_buf of a ModSummary has my in-memory version:


https://gitlab.com/tseenshe/hsinspect/-/blob/dfcb50a4e3437fbb4d0a3eb786b99c3a527d5e23/library/HsInspect/Workarounds.hs#L77-84

  minf_rdr_env' :: GHC.GhcMonad m => GHC.ModuleName -> m GlobalRdrEnv
  minf_rdr_env' m = do
    modSum <- GHC.getModSummary m
    pmod <- GHC.parseModule modSum
    tmod <- GHC.typecheckModule pmod
    let (tc_gbl_env, _) = GHC.tm_internals_ tmod
    pure $ tcg_rdr_env tc_gbl_env

You can see how I setup the session in


https://gitlab.com/tseenshe/hsinspect/-/blob/dfcb50a4e3437fbb4d0a3eb786b99c3a527d5e23/library/HsInspect/Imports.hs#L21-35

And whereas in 8.4.4 and 8.6.5 the typecheckModule call is working on the
in-memory representation, in 8.8.1 and 8.8.2 it is falling back to the
version on disk and therefore failing, because I have made in-memory
changes that are necessary.

Aside: I am able to observe this because I have a function that trims
everything except the module declaration and import section, and can
therefore tolerate typos in the rest of the file. Now my code fails if the
entire file does not typecheck, which is also incredibly slow, because I
only want access to the import section.

Best regards,
Tseen She
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.haskell.org/pipermail/ghc-devs/attachments/20200226/e471da0f/attachment.html>

From ts33n.sh3 at gmail.com  Wed Feb 26 21:58:31 2020
From: ts33n.sh3 at gmail.com (Tseen She)
Date: Wed, 26 Feb 2020 21:58:31 +0000
Subject: 8.8.x change in lookupModule / findModule semantics?
Message-ID: <CACKRAT0jEg8dknB7rbHb9WgomxnG4RGKo16b4pcXk4uceJf_Wg@mail.gmail.com>

Hi ghc-devs,

When upgrading my dev tool to use the 8.8.x api I have found that calls to
lookupModule / findModule are failing on uncompilable code, whereas these
calls succeeded on 8.4 and 8.6.

Unlike another question that I have asked [1] which seems related, I don't
intend to do any typechecking here, I only want access to the ModuleInfo.

Has this been an intentional change? Is there a way to recover the original
behaviour?

If I cannot get this to work then it means that a new feature introduced in
https://gitlab.haskell.org/ghc/ghc/merge_requests/1541 that I was quite
excited about is unusable :-(

[1] https://mail.haskell.org/pipermail/ghc-devs/2020-February/018655.html
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.haskell.org/pipermail/ghc-devs/attachments/20200226/ef527df3/attachment.html>

From dxld at darkboxed.org  Wed Feb 26 23:04:14 2020
From: dxld at darkboxed.org (Daniel =?iso-8859-1?Q?Gr=F6ber?=)
Date: Thu, 27 Feb 2020 00:04:14 +0100
Subject: 8.8.x change in lookupModule / findModule semantics?
In-Reply-To: <CACKRAT0jEg8dknB7rbHb9WgomxnG4RGKo16b4pcXk4uceJf_Wg@mail.gmail.com>
References: <CACKRAT0jEg8dknB7rbHb9WgomxnG4RGKo16b4pcXk4uceJf_Wg@mail.gmail.com>
Message-ID: <20200226230414.GA27770@Eli.lan>

Hi,

looking at `importsOnly`:

    #if MIN_VERSION_GLASGOW_HASKELL(8,8,1,0)
      pp <- liftIO $ preprocess sess file Nothing Nothing
      let (dflags, tmp) = case pp of
            Left _ -> error $ "preprocessing failed " <> show file
            Right success -> success
    #else

Since 8.8.1 `preprocess` takes an addition argument of type `Maybe
StringBuffer` that you're setting to `Nothing` here, so the pipeline will
read your on-disk file instead of using the buffer.

The whole point of that change was to allow passing in-memory buffers
straight into the pipeline for tooling so I'm hoping you can use that to
simplify your workarounds, see
https://gitlab.haskell.org/ghc/ghc/merge_requests/1014/ for context.

--Daniel

On Wed, Feb 26, 2020 at 09:58:31PM +0000, Tseen She wrote:
> Hi ghc-devs,
>
> When upgrading my dev tool to use the 8.8.x api I have found that calls to
> lookupModule / findModule are failing on uncompilable code, whereas these
> calls succeeded on 8.4 and 8.6.
>
> Unlike another question that I have asked [1] which seems related, I don't
> intend to do any typechecking here, I only want access to the ModuleInfo.
>
> Has this been an intentional change? Is there a way to recover the original
> behaviour?
>
> If I cannot get this to work then it means that a new feature introduced in
> https://gitlab.haskell.org/ghc/ghc/merge_requests/1541 that I was quite
> excited about is unusable :-(
>
> [1] https://mail.haskell.org/pipermail/ghc-devs/2020-February/018655.html

> _______________________________________________
> ghc-devs mailing list
> ghc-devs at haskell.org
> http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 833 bytes
Desc: not available
URL: <http://mail.haskell.org/pipermail/ghc-devs/attachments/20200227/e3ef0b5f/attachment.sig>

From ts33n.sh3 at gmail.com  Thu Feb 27 08:02:16 2020
From: ts33n.sh3 at gmail.com (Tseen She)
Date: Thu, 27 Feb 2020 08:02:16 +0000
Subject: 8.8.x change in lookupModule / findModule semantics?
In-Reply-To: <20200226230414.GA27770@Eli.lan>
References: <CACKRAT0jEg8dknB7rbHb9WgomxnG4RGKo16b4pcXk4uceJf_Wg@mail.gmail.com>
 <20200226230414.GA27770@Eli.lan>
Message-ID: <CACKRAT03rJtxkCrTaae0SBHFZ1G0C72fgXusGGQzFStcd-UdmQ@mail.gmail.com>

Thanks Daniel,

I think we might be mixing up the two threads here, please let me know if
I've misunderstood, so I'll respond as if this was in response to the
thread about in-memory buffers: the function you've hightlighted is working
as intended (thanks for clarifying the API change, I had to work that out
myself and was just guessing) and produces the minimal buffer that is then
used by the next function.

As to the relation between importsOnly and lookupModule / findModule, they
are independent.

On Wed, 26 Feb 2020 at 23:05, Daniel Gröber <dxld at darkboxed.org> wrote:

> Hi,
>
> looking at `importsOnly`:
>
>     #if MIN_VERSION_GLASGOW_HASKELL(8,8,1,0)
>       pp <- liftIO $ preprocess sess file Nothing Nothing
>       let (dflags, tmp) = case pp of
>             Left _ -> error $ "preprocessing failed " <> show file
>             Right success -> success
>     #else
>
> Since 8.8.1 `preprocess` takes an addition argument of type `Maybe
> StringBuffer` that you're setting to `Nothing` here, so the pipeline will
> read your on-disk file instead of using the buffer.
>
> The whole point of that change was to allow passing in-memory buffers
> straight into the pipeline for tooling so I'm hoping you can use that to
> simplify your workarounds, see
> https://gitlab.haskell.org/ghc/ghc/merge_requests/1014/ for context.
>
> --Daniel
>
> On Wed, Feb 26, 2020 at 09:58:31PM +0000, Tseen She wrote:
> > Hi ghc-devs,
> >
> > When upgrading my dev tool to use the 8.8.x api I have found that calls
> to
> > lookupModule / findModule are failing on uncompilable code, whereas these
> > calls succeeded on 8.4 and 8.6.
> >
> > Unlike another question that I have asked [1] which seems related, I
> don't
> > intend to do any typechecking here, I only want access to the ModuleInfo.
> >
> > Has this been an intentional change? Is there a way to recover the
> original
> > behaviour?
> >
> > If I cannot get this to work then it means that a new feature introduced
> in
> > https://gitlab.haskell.org/ghc/ghc/merge_requests/1541 that I was quite
> > excited about is unusable :-(
> >
> > [1]
> https://mail.haskell.org/pipermail/ghc-devs/2020-February/018655.html
>
> > _______________________________________________
> > ghc-devs mailing list
> > ghc-devs at haskell.org
> > http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.haskell.org/pipermail/ghc-devs/attachments/20200227/e2478901/attachment.html>

From marlowsd at gmail.com  Thu Feb 27 08:37:31 2020
From: marlowsd at gmail.com (Simon Marlow)
Date: Thu, 27 Feb 2020 08:37:31 +0000
Subject: Confused about PAP object layout
In-Reply-To: <CAMQQO3=QwVdknFq17vHR=REupWnpqo6mnOEFVaVNK5zO-A=4zg@mail.gmail.com>
References: <CAMQQO3=D=2y3BGQJzrk3VHy9=LV_EdYjyVnAALMuq9N=KgANbA@mail.gmail.com>
 <CANsrgP+1Ve0ahkndAoSOASU4cm-K92hJ=SXG+OpYKHX4bEuO-g@mail.gmail.com>
 <CAMQQO3kq0HxLShqdiPesnGMVnkn-AEkY3SXN8DCSSA2hxrQ7vQ@mail.gmail.com>
 <FF5887E8-5266-473B-951D-4DDD02A349E8@gmail.com>
 <CAMQQO3=bgoRY_OQ3eMJ3m7fSN4S9Wf8QieKSm6VgYBDs6=mcBw@mail.gmail.com>
 <87sgjdulmq.fsf@smart-cactus.org>
 <CAMQQO3nSeHAfzUJjCYvUVXCf8JwvPnVLoSO_fwMZmdgqijrpXg@mail.gmail.com>
 <CANsrgPKW5D5RvLWcqDqF9GOTaraCyFmLSQq=2S0sq7b0Jfox2Q@mail.gmail.com>
 <VI1PR8303MB009396D8C4F99ED1B8E08256ADEC0@VI1PR8303MB0093.EURPRD83.prod.outlook.com>
 <CAMQQO3=QwVdknFq17vHR=REupWnpqo6mnOEFVaVNK5zO-A=4zg@mail.gmail.com>
Message-ID: <CANsrgPLm0OHj5OeaXCOrLKjiQimUdp6ZfL5DX3W8DuVwzxYkbQ@mail.gmail.com>

On Wed, 26 Feb 2020 at 18:48, Ömer Sinan Ağacan <omeragacan at gmail.com>
wrote:

> So the key points from this thread are:
>
> - PAP payloads are scavenged using the function's bitmap. Because a PAPs
> payload
>   will have less number of closures than the function's arity the bitmap
> will
>   always have enough bits.
>
> - A bit in a function bitmap is NOT for liveness (e.g. does not indicate
> whether
>   an argument used or not), but for pointers vs. non-pointers. Function
> bitmaps
>   are called "liveness bits" in the code generator which is misleading.
>

I think of all bitmaps as representing "liveness" (or equivalently
"pointerhood") for the purposes of GC. There's no difference from the GC's
perspective between a non-pointer and a pointer that it doesn't need to
follow.

In fact there's nothing to prevent us using the function bitmap to indicate
dead arguments too - it would require zero changes in the RTS, the compiler
would only need to mark unused pointer arguments as non-pointers in the
bitmap. Probably wouldn't be worth very much overall, but I do recall one
space leak that would have been cured by this.

- In a function bitmap (small or large), 0 means pointer, 1 means
> non-pointer.
>

This is true of bitmaps generally I think, not just function bitmaps.

  This is really what confused me in my last email above. For some reason I
>   intuitively expected 1 to mean pointer, not 0. Simon M also got this
> wrong
>

Oops :)

I think there may originally have been a good reason to have it this way
around: before eval/apply, we used bitmaps to describe stack frames, but we
didn't need to encode a size in the bitmap because the default was for the
stack contents to be pointers unless there was something to tell us
otherwise. So a zero suffix of a bitmap just meant "the rest is just normal
stack". This changed with eval/apply, but we kept the convention that zero
meant pointer in a bitmap.


>   ("So a 0 in the bitmap always means non-pointer.") so maybe this is
> confusing
>   to others too.
>
> - For functions with known argument patterns we don't use the function's
> bitmap.
>   These function's type are greater than ARG_BCO (2), and for those we use
> the
>   stg_arg_bitmaps array to get the bitmap.
>
>   For example, the bitmap for ARG_PPP (function with 3 pointer arguments)
> is at
>   index 23 in this array, which is 0b11. For ARG_PNN it's 0b110000011. The
> least
>   significant 6 bits are for the size (3), the remaining 0b110 means the
> first
>   argument is a pointer, rest of the two are non-pointers.
>

Actually I think documentation on this is missing in the wiki, I guess I
never got around to updating it when we implemented eval/apply. This page
should really describe function info tables:
https://gitlab.haskell.org/ghc/ghc/wikis/commentary/rts/storage/heap-objects#info-tables

If you want to add documentation that would be a good place.

Cheers
Simon

I still don't understand why this assertion
>
>     ASSERT(BITMAP_SIZE(bitmap) >= size);
>
> I added to scavenge_small_bitmap in !2727 is failing though.
>
> Ömer
>
> Simon Peyton Jones <simonpj at microsoft.com>, 24 Şub 2020 Pzt, 13:45
> tarihinde şunu yazdı:
> >
> > I’m not following this in detail, but do please make sure that the
> results of this discussion end up in a suitable Note.  Obviously it’s not
> transparently clear as-is, and I can see clarity emerging
> >
> >
> >
> > Thanks!
> >
> >
> > Simon
> >
> >
> >
> > From: ghc-devs <ghc-devs-bounces at haskell.org> On Behalf Of Simon Marlow
> > Sent: 24 February 2020 08:22
> > To: Ömer Sinan Ağacan <omeragacan at gmail.com>
> > Cc: ghc-devs <ghc-devs at haskell.org>
> > Subject: Re: Confused about PAP object layout
> >
> >
> >
> > On Thu, 20 Feb 2020 at 09:21, Ömer Sinan Ağacan <omeragacan at gmail.com>
> wrote:
> >
> > > I'm not sure what you mean by "garbage". The bitmap merely determines
> whether
> > > a field is a pointer,
> >
> > I think the bitmap is for liveness, not for whether a field is pointer
> or not.
> > Relevant code for building an info table for a function:
> >
> >     mk_pieces (Fun arity (ArgGen arg_bits)) srt_label
> >       = do { (liveness_lit, liveness_data) <- mkLivenessBits dflags
> arg_bits
> >            ; let fun_type | null liveness_data = aRG_GEN
> >                           | otherwise          = aRG_GEN_BIG
> >                  extra_bits = [ packIntsCLit dflags fun_type arity ]
> >                            ++ (if inlineSRT dflags then [] else [
> srt_lit ])
> >                            ++ [ liveness_lit, slow_entry ]
> >            ; return (Nothing, Nothing, extra_bits, liveness_data) }
> >
> > This uses the word "liveness" rather than "pointers".
> >
> > However I just realized that the word "garbage" is still not the best
> way to
> > describe what I'm trying to say. In the example
> >
> >     [pap_info, x, y, z]
> >
> > If the function's bitmap is [1, 0, 1], then `y` may be a dead (an unused
> > argument, or "garbage" as I describe in my previous email) OR it may be a
> > non-pointer, but used (i.e. not a garbage).
> >
> >
> >
> > I don't think we ever put a zero in the bitmap for a
> pointer-but-not-used argument. We don't do liveness analysis for function
> arguments, as far as I'm aware. So a 0 in the bitmap always means
> "non-pointer".
> >
> >
> >
> > The only reaosn the code uses the terminology "liveness" here is that
> it's sharing code with the code that handles bitmaps for stack frames,
> which do deal with liveness.
> >
> >
> >
> > So maybe "liveness" is also not the best way to describe this bitmap, as
> 0 does
> > not mean dead but rather "don't follow in GC".
> >
> >
> > On my quest to understand and document this code better I have one more
> > question. When generating info tables for functions with know argument
> patterns
> > (ArgSpec) we initialize the bitmap as 0. Relevant code:
> >
> >     mk_pieces (Fun arity (ArgSpec fun_type)) srt_label
> >       = do { let extra_bits = packIntsCLit dflags fun_type arity :
> srt_label
> >            ; return (Nothing, Nothing,  extra_bits, []) }
> >
> > Here the last return value is for the liveness data. I don't understand
> how can
> > this be correct, because when we use this function in a PAP this will
> cause NOT
> > scavenging the PAP payload. Relevant code (simplified):
> >
> >     STATIC_INLINE GNUC_ATTR_HOT StgPtr
> >     scavenge_PAP_payload (StgClosure *fun, StgClosure **payload, StgWord
> size)
> >     {
> >         const StgFunInfoTable *fun_info =
> >             get_fun_itbl(UNTAG_CONST_CLOSURE(fun));
> >
> >         StgPtr p = (StgPtr)payload;
> >
> >         StgWord bitmap;
> >         switch (fun_info->f.fun_type) {
> >         ...
> >
> >         default:
> >             bitmap = BITMAP_BITS(stg_arg_bitmaps[fun_info->f.fun_type]);
> >         small_bitmap:
> >             p = scavenge_small_bitmap(p, size, bitmap);
> >             break;
> >         }
> >         return p;
> >     }
> >
> >
> > Here if I have a function with three pointer args (ARG_PPP) the shown
> branch
> > that will be taken, but because the bitmap is 0 (as shown in the
> mk_pieces code
> > above) nothing in the PAPs payload will be scavenged.
> >
> >
> >
> > It gets the bitmap from stg_arg_bitmaps[fun_info->f.fun_type], not from
> the info table.  Hope this helps.
> >
> >
> >
> > Cheers
> >
> > Simon
> >
> >
> >
> >
> >
> >
> > Here's an example from a debugging session:
> >
> >     >>> print pap
> >     $10 = (StgPAP *) 0x42001fe030
> >
> >     >>> print *pap
> >     $11 = {
> >       header = {
> >         info = 0x7fbdd1f06640 <stg_PAP_info>
> >       },
> >       arity = 2,
> >       n_args = 1,
> >       fun = 0x7fbdd2d23ffb,
> >       payload = 0x42001fe048
> >     }
> >
> > So this PAP is applied one argument, which is a boxed object (a FUN_2_0):
> >
> >     >>> print *get_itbl(UNTAG_CLOSURE(pap->payload[0]))
> >     $20 = {
> >       layout = {
> >         payload = {
> >           ptrs = 2,
> >           nptrs = 0
> >         },
> >         bitmap = 2,
> >         large_bitmap_offset = 2,
> >         __pad_large_bitmap_offset = 2,
> >         selector_offset = 2
> >       },
> >       type = 11,
> >       srt = 1914488,
> >       code = 0x7fbdd2b509c0 "H\215E\370L9\370r[I\203\304 M;\245X\003"
> >     }
> >
> > However if I look at the function of this PAP:
> >
> >     >>> print *get_fun_itbl(UNTAG_CLOSURE(pap->fun))
> >     $21 = {
> >       f = {
> >         slow_apply_offset = 16,
> >         __pad_slow_apply_offset = 3135120895,
> >         b = {
> >           bitmap = 74900193017889,
> >           bitmap_offset = 258342945,
> >           __pad_bitmap_offset = 258342945
> >         },
> >         fun_type = 23,
> >         arity = 3
> >       },
> >       i = {
> >         layout = {
> >           payload = {
> >             ptrs = 0,
> >             nptrs = 0
> >           },
> >           bitmap = 0,
> >           large_bitmap_offset = 0,
> >           __pad_large_bitmap_offset = 0,
> >           selector_offset = 0
> >         },
> >         type = 14,
> >         srt = 1916288,
> >         code = 0x7fbdd2b50260 <base_GHCziRead_list3_info>
> > "I\203\304(M;\245X\003"
> >       }
> >     }
> >
> > It has arity 3. Since the first argument is a boxed object and this
> function has
> > arity 3, if the argument is actually live in the function (i.e. not an
> unused
> > argument), then the bitmap should have a 1 for this. But because the
> argument
> > pattern is known (ARG_PPP) we initialized the bitmap as 0! Not sure how
> this
> > can work.
> >
> > What am I missing?
> >
> > Thanks,
> >
> > Ömer
> >
> > Ben Gamari <ben at smart-cactus.org>, 14 Şub 2020 Cum, 20:25 tarihinde
> şunu yazdı:
> > >
> > > Ömer Sinan Ağacan <omeragacan at gmail.com> writes:
> > >
> > > > I think that makes sense, with the invariant that n_args <=
> bitmap_size. We
> > > > evacuate the arguments used by the function but not others. Thanks.
> > > >
> > > > It's somewhat weird to see an object with useful stuff, then
> garbage, then
> > > > useful stuff again in the heap, but that's not an issue by itself.
> For example
> > > > if I have something like
> > > >
> > > >     [pap_info, x, y, z]
> > > >
> > > > and according to the function `y` is dead, then after evacuating I
> get
> > > >
> > > >     [pap_info, x, <garbage>, z]
> > > >
> > > > This "garbage" is evacuated again and again every time we evacuate
> this PAP.
> > > >
> > > I'm not sure what you mean by "garbage". The bitmap merely determines
> > > whether a field is a pointer, not whether it is copied during
> > > evacuation. A field's bitmap bit not being set merely means that we
> won't
> > > evacuate the value of that field during scavenging.
> > >
> > > Nevertheless, this all deserves a comment in scavenge_PAP.
> > >
> > > Cheers,
> > >
> > > - Ben
> > >
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.haskell.org/pipermail/ghc-devs/attachments/20200227/e2d2bb73/attachment.html>

From dxld at darkboxed.org  Thu Feb 27 11:39:10 2020
From: dxld at darkboxed.org (Daniel =?iso-8859-1?Q?Gr=F6ber?=)
Date: Thu, 27 Feb 2020 12:39:10 +0100
Subject: 8.8.x change in lookupModule / findModule semantics?
In-Reply-To: <CACKRAT03rJtxkCrTaae0SBHFZ1G0C72fgXusGGQzFStcd-UdmQ@mail.gmail.com>
References: <CACKRAT0jEg8dknB7rbHb9WgomxnG4RGKo16b4pcXk4uceJf_Wg@mail.gmail.com>
 <20200226230414.GA27770@Eli.lan>
 <CACKRAT03rJtxkCrTaae0SBHFZ1G0C72fgXusGGQzFStcd-UdmQ@mail.gmail.com>
Message-ID: <20200227113910.GB27770@Eli.lan>

Hi,

On Thu, Feb 27, 2020 at 08:02:16AM +0000, Tseen She wrote:
> I think we might be mixing up the two threads here, please let me know if
> I've misunderstood

Ah yes, I was reading both threads and ended up replying to the wrong one
:)

> so I'll respond as if this was in response to the
> thread about in-memory buffers: the function you've hightlighted is working
> as intended (thanks for clarifying the API change, I had to work that out
> myself and was just guessing) and produces the minimal buffer that is then
> used by the next function.

You know, it does look like you're implementing in importsOnly is very
simmilar to ghc's internal getPreprocessedImports. While that function
isn't exported it's just a simple wrapper around HeaderInfo.getImports
which is. You might want to use that API instead or is there a reason you
had to reimplement all of it?

Anyways, looking at the code for parseModule as long as the ModSummary you
pass in has ms_hspp_buf set it should use the buffer instead of the
file. The code will eventually call `hscParse'` which has:

    let src_filename  = ms_hspp_file mod_summary
        maybe_src_buf = ms_hspp_buf  mod_summary

    --------------------------  Parser  ----------------
    -- sometimes we already have the buffer in memory, perhaps
    -- because we needed to parse the imports out of it, or get the
    -- module name.
    buf <- case maybe_src_buf of
               Just b  -> return b
               Nothing -> liftIO $ hGetStringBuffer src_filename

typecheckModule will just use the ParsedSource parseModule produced so it
shouldn't do any more file reading.

> As to the relation between importsOnly and lookupModule / findModule, they
> are independent.

I'm not actually seeing any calls to {lookup,find}Module in your code so
it's very hard to say if the behaviour changed. Do you have a reproducer
for that?

--Daniel
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 833 bytes
Desc: not available
URL: <http://mail.haskell.org/pipermail/ghc-devs/attachments/20200227/35cd79d9/attachment.sig>

From ts33n.sh3 at gmail.com  Thu Feb 27 14:50:39 2020
From: ts33n.sh3 at gmail.com (Tseen She)
Date: Thu, 27 Feb 2020 14:50:39 +0000
Subject: 8.8.x change in lookupModule / findModule semantics?
In-Reply-To: <20200227113910.GB27770@Eli.lan>
References: <CACKRAT0jEg8dknB7rbHb9WgomxnG4RGKo16b4pcXk4uceJf_Wg@mail.gmail.com>
 <20200226230414.GA27770@Eli.lan>
 <CACKRAT03rJtxkCrTaae0SBHFZ1G0C72fgXusGGQzFStcd-UdmQ@mail.gmail.com>
 <20200227113910.GB27770@Eli.lan>
Message-ID: <CACKRAT1vjBi4nFkXJyHEqEsnqvCZCuYvN+H=i_Gi_ta878ruJg@mail.gmail.com>

On Thu, 27 Feb 2020 at 11:40, Daniel Gröber <dxld at darkboxed.org> wrote:
> You know, it does look like you're implementing in importsOnly is very
> simmilar to ghc's internal getPreprocessedImports

Very likely :-) but if it's not exported I am stuck with the workaround.

. While that function
> isn't exported it's just a simple wrapper around HeaderInfo.getImports
> which is. You might want to use that API instead or is there a reason you
> had to reimplement all of it?

getImports only returns located module names, but I need to recover the
source code with module definition, pragmas, and imports, so that I can
typecheck it (actually I just need to run namer, but that isn't exposed as
a separate phase and typechecking basically isn't anything more than namer
when there is nothing in the file except imports).

Just to be clear though, there is absolutely no problem with the
importsOnly code so much as I'm aware, although I'd be happy to throw it
away if this was made available inside ghc :-)

> Anyways, looking at the code for parseModule as long as the ModSummary you
> pass in has ms_hspp_buf set it should use the buffer instead of the
> file.

Yes, I agree. That function is fine. Using debug tracing, I can confirm
that the ms_hspp_buf of a ModSummary has my in-memory version: I have
isolated the problem to a change in behaviour in the next line:
typecheckModule.


> typecheckModule will just use the ParsedSource parseModule produced so it
> shouldn't do any more file reading.

This is where I am not so sure. I am fairly certain that something has
broken in the typechecking line. I will try again with debug tracing to
confirm what the ParsedSource looks like.


> > As to the relation between importsOnly and lookupModule / findModule,
they
> > are independent.
>
> I'm not actually seeing any calls to {lookup,find}Module in your code so
> it's very hard to say if the behaviour changed. Do you have a reproducer
> for that?


That's because it is in a different branch than the in-memory stuff. I am
more or less getting a problem / regression with this code:

  let target = GHC.Target (GHC.TargetModule mn) False Nothing
  GHC.removeTarget $ TargetModule (traceShow (showGhc mn) mn)
  GHC.addTarget target
  _ <- GHC.load $ GHC.LoadUpTo mn
  m <- GHC.lookupModule mn Nothing

And in my WIP branch
https://gitlab.com/tseenshe/hsinspect/-/blob/wip/ghc882/library/HsInspect/Imports.hs#L38

BTW, if you can point me to something in ghc that does inferModuleName then
that'd be useful in several places.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.haskell.org/pipermail/ghc-devs/attachments/20200227/d36ca4a1/attachment.html>

From ts33n.sh3 at gmail.com  Thu Feb 27 15:02:35 2020
From: ts33n.sh3 at gmail.com (Tseen She)
Date: Thu, 27 Feb 2020 15:02:35 +0000
Subject: 8.8.x change in lookupModule / findModule semantics?
In-Reply-To: <CACKRAT1vjBi4nFkXJyHEqEsnqvCZCuYvN+H=i_Gi_ta878ruJg@mail.gmail.com>
References: <CACKRAT0jEg8dknB7rbHb9WgomxnG4RGKo16b4pcXk4uceJf_Wg@mail.gmail.com>
 <20200226230414.GA27770@Eli.lan>
 <CACKRAT03rJtxkCrTaae0SBHFZ1G0C72fgXusGGQzFStcd-UdmQ@mail.gmail.com>
 <20200227113910.GB27770@Eli.lan>
 <CACKRAT1vjBi4nFkXJyHEqEsnqvCZCuYvN+H=i_Gi_ta878ruJg@mail.gmail.com>
Message-ID: <CACKRAT2HiP4wjeCSNouXp_W_o9Fy1kmxvZ7cBBNYyt97XrAteA@mail.gmail.com>

Quick followup with trace debugging:

On Thu, 27 Feb 2020 at 14:50, Tseen She <ts33n.sh3 at gmail.com> wrote:

> > typecheckModule will just use the ParsedSource parseModule produced so it
> > shouldn't do any more file reading.
>
> This is where I am not so sure. I am fairly certain that something has
> broken in the typechecking line. I will try again with debug tracing to
> confirm what the ParsedSource looks like.
>

I can confirm that a showPpr on `pm_parsed_source` has the in-memory
version.

But, perhaps, the problem might be that pragmas are not carried over? In my
test the "importsOnly" version of the file is preserving two LANGUAGE
pragmas that are necessary in order to parse the imports section (one of my
tests uses PackageImports). Just a guess, but does 8.8.x now require more
manual passing of the dynflags? This seems like a regression / bug to me
but I'd be willing to workaround it if it is an intentional behaviour
change.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.haskell.org/pipermail/ghc-devs/attachments/20200227/04274d93/attachment.html>

From ts33n.sh3 at gmail.com  Thu Feb 27 15:18:49 2020
From: ts33n.sh3 at gmail.com (Tseen She)
Date: Thu, 27 Feb 2020 15:18:49 +0000
Subject: 8.8.x change in lookupModule / findModule semantics?
In-Reply-To: <CACKRAT2HiP4wjeCSNouXp_W_o9Fy1kmxvZ7cBBNYyt97XrAteA@mail.gmail.com>
References: <CACKRAT0jEg8dknB7rbHb9WgomxnG4RGKo16b4pcXk4uceJf_Wg@mail.gmail.com>
 <20200226230414.GA27770@Eli.lan>
 <CACKRAT03rJtxkCrTaae0SBHFZ1G0C72fgXusGGQzFStcd-UdmQ@mail.gmail.com>
 <20200227113910.GB27770@Eli.lan>
 <CACKRAT1vjBi4nFkXJyHEqEsnqvCZCuYvN+H=i_Gi_ta878ruJg@mail.gmail.com>
 <CACKRAT2HiP4wjeCSNouXp_W_o9Fy1kmxvZ7cBBNYyt97XrAteA@mail.gmail.com>
Message-ID: <CACKRAT2Ss-B1D2K_+oZyxjNo6qp+Y_kmUTn1Mx0472qdWD7n9A@mail.gmail.com>

and it looks like the language pragmas slurped from the in-memory buffer is
empty

  tmod <- GHC.typecheckModule (trace (showGhc . GHC.extensions $
GHC.ms_hspp_opts modSum) pmod)

gives me [] on 8.8 but is [Off ImplicitPrelude, On PackageImports] on
8.6.5... that looks like an 8.8 regression to me.


On Thu, 27 Feb 2020 at 15:02, Tseen She <ts33n.sh3 at gmail.com> wrote:

> Quick followup with trace debugging:
>
> On Thu, 27 Feb 2020 at 14:50, Tseen She <ts33n.sh3 at gmail.com> wrote:
>
>> > typecheckModule will just use the ParsedSource parseModule produced so
>> it
>> > shouldn't do any more file reading.
>>
>> This is where I am not so sure. I am fairly certain that something has
>> broken in the typechecking line. I will try again with debug tracing to
>> confirm what the ParsedSource looks like.
>>
>
> I can confirm that a showPpr on `pm_parsed_source` has the in-memory
> version.
>
> But, perhaps, the problem might be that pragmas are not carried over? In
> my test the "importsOnly" version of the file is preserving two LANGUAGE
> pragmas that are necessary in order to parse the imports section (one of my
> tests uses PackageImports). Just a guess, but does 8.8.x now require more
> manual passing of the dynflags? This seems like a regression / bug to me
> but I'd be willing to workaround it if it is an intentional behaviour
> change.
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.haskell.org/pipermail/ghc-devs/attachments/20200227/1951b13d/attachment.html>

From ts33n.sh3 at gmail.com  Thu Feb 27 15:43:35 2020
From: ts33n.sh3 at gmail.com (Tseen She)
Date: Thu, 27 Feb 2020 15:43:35 +0000
Subject: 8.8.x change in lookupModule / findModule semantics?
In-Reply-To: <CACKRAT2Ss-B1D2K_+oZyxjNo6qp+Y_kmUTn1Mx0472qdWD7n9A@mail.gmail.com>
References: <CACKRAT0jEg8dknB7rbHb9WgomxnG4RGKo16b4pcXk4uceJf_Wg@mail.gmail.com>
 <20200226230414.GA27770@Eli.lan>
 <CACKRAT03rJtxkCrTaae0SBHFZ1G0C72fgXusGGQzFStcd-UdmQ@mail.gmail.com>
 <20200227113910.GB27770@Eli.lan>
 <CACKRAT1vjBi4nFkXJyHEqEsnqvCZCuYvN+H=i_Gi_ta878ruJg@mail.gmail.com>
 <CACKRAT2HiP4wjeCSNouXp_W_o9Fy1kmxvZ7cBBNYyt97XrAteA@mail.gmail.com>
 <CACKRAT2Ss-B1D2K_+oZyxjNo6qp+Y_kmUTn1Mx0472qdWD7n9A@mail.gmail.com>
Message-ID: <CACKRAT09r1moiKu5-6sdODGO6zzgTyd+cJURvKbo=a5BJZG77Q@mail.gmail.com>

Sorry for the spam, but I think this is definitely a bug/regression in
ghc-8.8.1 (still present in 8.8.3).

I don't know which commit introduced the problem, but it seems that
getModSummary is no longer reporting the correct ms_hspp_opts, at least for
an in-memory file but it could also be for a file on disk as well (I
haven't excluded that as a possibility).

Here is my workaround

#if MIN_VERSION_GLASGOW_HASKELL(8,8,1,0)
  dflags <- GHC.getSessionDynFlags
  let file = GHC.ms_hspp_file modSum
  buf <- case GHC.ms_hspp_buf modSum of
        Nothing -> liftIO $ hGetStringBuffer file
        Just b -> pure b
  let pragmas = getOptions dflags buf file
  (dflags', _, _) <- parseDynamicFilePragma dflags pragmas
  let modSum' = modSum { GHC.ms_hspp_opts = dflags' }
#else
  let modSum' = modSum
#endif



On Thu, 27 Feb 2020 at 15:18, Tseen She <ts33n.sh3 at gmail.com> wrote:

> and it looks like the language pragmas slurped from the in-memory buffer
> is empty
>
>   tmod <- GHC.typecheckModule (trace (showGhc . GHC.extensions $
> GHC.ms_hspp_opts modSum) pmod)
>
> gives me [] on 8.8 but is [Off ImplicitPrelude, On PackageImports] on
> 8.6.5... that looks like an 8.8 regression to me.
>
>
> On Thu, 27 Feb 2020 at 15:02, Tseen She <ts33n.sh3 at gmail.com> wrote:
>
>> Quick followup with trace debugging:
>>
>> On Thu, 27 Feb 2020 at 14:50, Tseen She <ts33n.sh3 at gmail.com> wrote:
>>
>>> > typecheckModule will just use the ParsedSource parseModule produced so
>>> it
>>> > shouldn't do any more file reading.
>>>
>>> This is where I am not so sure. I am fairly certain that something has
>>> broken in the typechecking line. I will try again with debug tracing to
>>> confirm what the ParsedSource looks like.
>>>
>>
>> I can confirm that a showPpr on `pm_parsed_source` has the in-memory
>> version.
>>
>> But, perhaps, the problem might be that pragmas are not carried over? In
>> my test the "importsOnly" version of the file is preserving two LANGUAGE
>> pragmas that are necessary in order to parse the imports section (one of my
>> tests uses PackageImports). Just a guess, but does 8.8.x now require more
>> manual passing of the dynflags? This seems like a regression / bug to me
>> but I'd be willing to workaround it if it is an intentional behaviour
>> change.
>>
>>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.haskell.org/pipermail/ghc-devs/attachments/20200227/0e05dec1/attachment.html>

From dxld at darkboxed.org  Thu Feb 27 18:12:11 2020
From: dxld at darkboxed.org (Daniel =?iso-8859-1?Q?Gr=F6ber?=)
Date: Thu, 27 Feb 2020 19:12:11 +0100
Subject: 8.8.x change in lookupModule / findModule semantics?
In-Reply-To: <CACKRAT09r1moiKu5-6sdODGO6zzgTyd+cJURvKbo=a5BJZG77Q@mail.gmail.com>
References: <CACKRAT0jEg8dknB7rbHb9WgomxnG4RGKo16b4pcXk4uceJf_Wg@mail.gmail.com>
 <20200226230414.GA27770@Eli.lan>
 <CACKRAT03rJtxkCrTaae0SBHFZ1G0C72fgXusGGQzFStcd-UdmQ@mail.gmail.com>
 <20200227113910.GB27770@Eli.lan>
 <CACKRAT1vjBi4nFkXJyHEqEsnqvCZCuYvN+H=i_Gi_ta878ruJg@mail.gmail.com>
 <CACKRAT2HiP4wjeCSNouXp_W_o9Fy1kmxvZ7cBBNYyt97XrAteA@mail.gmail.com>
 <CACKRAT2Ss-B1D2K_+oZyxjNo6qp+Y_kmUTn1Mx0472qdWD7n9A@mail.gmail.com>
 <CACKRAT09r1moiKu5-6sdODGO6zzgTyd+cJURvKbo=a5BJZG77Q@mail.gmail.com>
Message-ID: <20200227181211.GC27770@Eli.lan>

Hi,

On Thu, Feb 27, 2020 at 03:43:35PM +0000, Tseen She wrote:
> Sorry for the spam, but I think this is definitely a bug/regression in
> ghc-8.8.1 (still present in 8.8.3).

No worries

> I don't know which commit introduced the problem, but it seems that
> getModSummary is no longer reporting the correct ms_hspp_opts, at least for
> an in-memory file but it could also be for a file on disk as well (I
> haven't excluded that as a possibility).

On a quick testcase I cannot reproduce this behaviour:

    -- $ ghc -package ghc -package ghc-paths TargetContents.hs
    module Main where

    import GHC
    import GHC.Paths (libdir)
    import MonadUtils
    import DynFlags
    import StringBuffer
    import Data.Time.Clock

    main :: IO ()
    main = do
      defaultErrorHandler defaultFatalMessager defaultFlushOut $ do
        runGhc (Just libdir) $ do
          dflags0 <- getSessionDynFlags
          (dflags1, _, _)
              <- parseDynamicFlags dflags0 $ map noLoc ["-package", "base"]
          _ <- setSessionDynFlags dflags1

          t <- liftIO getCurrentTime
          setTargets [Target (TargetFile "Main.hs" Nothing) False (Just (stringToStringBuffer buffer, t)) ]

          _ <- depanal [] False

          ms <- getModSummary (mkModuleName "Main")
          pm <- parseModule ms

          liftIO $ print $ extensions $ ms_hspp_opts ms

          _ <- typecheckModule pm

          return ()

    buffer = "{-# LANGUAGE PackageImports #-}\nimport \"base\" Data.List\nmain = return ()"

Running it it prints the PackageImports ext from the in-memory buffer just
fine:

    $ ghc-8.8.1 -package ghc -package ghc-paths TargetContents.hs
    $ ./TargetContents
    [On PackageImports]

One change in behaviour to note is that when doing this with <8.8 we first
need to create the Main.hs file so GHC doesn't complain about it missing.

Maybe you can fiddle with the test case until it reflects what you're
doing?

--Daniel
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 833 bytes
Desc: not available
URL: <http://mail.haskell.org/pipermail/ghc-devs/attachments/20200227/9a73ee0e/attachment.sig>

From ts33n.sh3 at gmail.com  Thu Feb 27 20:37:12 2020
From: ts33n.sh3 at gmail.com (Tseen She)
Date: Thu, 27 Feb 2020 20:37:12 +0000
Subject: 8.8.x change in lookupModule / findModule semantics?
In-Reply-To: <20200227181211.GC27770@Eli.lan>
References: <CACKRAT0jEg8dknB7rbHb9WgomxnG4RGKo16b4pcXk4uceJf_Wg@mail.gmail.com>
 <20200226230414.GA27770@Eli.lan>
 <CACKRAT03rJtxkCrTaae0SBHFZ1G0C72fgXusGGQzFStcd-UdmQ@mail.gmail.com>
 <20200227113910.GB27770@Eli.lan>
 <CACKRAT1vjBi4nFkXJyHEqEsnqvCZCuYvN+H=i_Gi_ta878ruJg@mail.gmail.com>
 <CACKRAT2HiP4wjeCSNouXp_W_o9Fy1kmxvZ7cBBNYyt97XrAteA@mail.gmail.com>
 <CACKRAT2Ss-B1D2K_+oZyxjNo6qp+Y_kmUTn1Mx0472qdWD7n9A@mail.gmail.com>
 <CACKRAT09r1moiKu5-6sdODGO6zzgTyd+cJURvKbo=a5BJZG77Q@mail.gmail.com>
 <20200227181211.GC27770@Eli.lan>
Message-ID: <CACKRAT0D3WsEbBLUmN2kd_LKOfTo5rLv3jZ0bwPxttJdgqYX+Q@mail.gmail.com>

Bingo!

      setTargets [Target (TargetFile "Main.hs" (Just $ Hsc HsSrcFile))
False (Just (stringToStringBuffer buffer, t))]

i.e. an explicit phase.

I will just use Nothing in my TargetFile. Was this an intended change?

On Thu, 27 Feb 2020 at 18:13, Daniel Gröber <dxld at darkboxed.org> wrote:

> Hi,
>
> On Thu, Feb 27, 2020 at 03:43:35PM +0000, Tseen She wrote:
> > Sorry for the spam, but I think this is definitely a bug/regression in
> > ghc-8.8.1 (still present in 8.8.3).
>
> No worries
>
> > I don't know which commit introduced the problem, but it seems that
> > getModSummary is no longer reporting the correct ms_hspp_opts, at least
> for
> > an in-memory file but it could also be for a file on disk as well (I
> > haven't excluded that as a possibility).
>
> On a quick testcase I cannot reproduce this behaviour:
>
>     -- $ ghc -package ghc -package ghc-paths TargetContents.hs
>     module Main where
>
>     import GHC
>     import GHC.Paths (libdir)
>     import MonadUtils
>     import DynFlags
>     import StringBuffer
>     import Data.Time.Clock
>
>     main :: IO ()
>     main = do
>       defaultErrorHandler defaultFatalMessager defaultFlushOut $ do
>         runGhc (Just libdir) $ do
>           dflags0 <- getSessionDynFlags
>           (dflags1, _, _)
>               <- parseDynamicFlags dflags0 $ map noLoc ["-package", "base"]
>           _ <- setSessionDynFlags dflags1
>
>           t <- liftIO getCurrentTime
>           setTargets [Target (TargetFile "Main.hs" Nothing) False (Just
> (stringToStringBuffer buffer, t)) ]
>
>           _ <- depanal [] False
>
>           ms <- getModSummary (mkModuleName "Main")
>           pm <- parseModule ms
>
>           liftIO $ print $ extensions $ ms_hspp_opts ms
>
>           _ <- typecheckModule pm
>
>           return ()
>
>     buffer = "{-# LANGUAGE PackageImports #-}\nimport \"base\"
> Data.List\nmain = return ()"
>
> Running it it prints the PackageImports ext from the in-memory buffer just
> fine:
>
>     $ ghc-8.8.1 -package ghc -package ghc-paths TargetContents.hs
>     $ ./TargetContents
>     [On PackageImports]
>
> One change in behaviour to note is that when doing this with <8.8 we first
> need to create the Main.hs file so GHC doesn't complain about it missing.
>
> Maybe you can fiddle with the test case until it reflects what you're
> doing?
>
> --Daniel
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.haskell.org/pipermail/ghc-devs/attachments/20200227/c8b0f0f1/attachment.html>

From ts33n.sh3 at gmail.com  Thu Feb 27 20:49:43 2020
From: ts33n.sh3 at gmail.com (Tseen She)
Date: Thu, 27 Feb 2020 20:49:43 +0000
Subject: 8.8.x change in lookupModule / findModule semantics?
In-Reply-To: <CACKRAT0D3WsEbBLUmN2kd_LKOfTo5rLv3jZ0bwPxttJdgqYX+Q@mail.gmail.com>
References: <CACKRAT0jEg8dknB7rbHb9WgomxnG4RGKo16b4pcXk4uceJf_Wg@mail.gmail.com>
 <20200226230414.GA27770@Eli.lan>
 <CACKRAT03rJtxkCrTaae0SBHFZ1G0C72fgXusGGQzFStcd-UdmQ@mail.gmail.com>
 <20200227113910.GB27770@Eli.lan>
 <CACKRAT1vjBi4nFkXJyHEqEsnqvCZCuYvN+H=i_Gi_ta878ruJg@mail.gmail.com>
 <CACKRAT2HiP4wjeCSNouXp_W_o9Fy1kmxvZ7cBBNYyt97XrAteA@mail.gmail.com>
 <CACKRAT2Ss-B1D2K_+oZyxjNo6qp+Y_kmUTn1Mx0472qdWD7n9A@mail.gmail.com>
 <CACKRAT09r1moiKu5-6sdODGO6zzgTyd+cJURvKbo=a5BJZG77Q@mail.gmail.com>
 <20200227181211.GC27770@Eli.lan>
 <CACKRAT0D3WsEbBLUmN2kd_LKOfTo5rLv3jZ0bwPxttJdgqYX+Q@mail.gmail.com>
Message-ID: <CACKRAT2s8ohFkFfv+hjcf3HvBR+ML9Rj_L1KJNEydav1V0fh-w@mail.gmail.com>

Additionally, now that importsOnly is working for me on 8.8.x I can use
lookupModule on *that* instead of the original module file, and then I can
make use of the newly exposed modInfoRdrEnv thus answering my second
thread's question too.

It is, however, a shame that lookupModule seems to require the entire file
to parse / typecheck. That also smells like a regression, but not one that
impacts me anymore.


On Thu, 27 Feb 2020 at 20:37, Tseen She <ts33n.sh3 at gmail.com> wrote:

> Bingo!
>
>       setTargets [Target (TargetFile "Main.hs" (Just $ Hsc HsSrcFile))
> False (Just (stringToStringBuffer buffer, t))]
>
> i.e. an explicit phase.
>
> I will just use Nothing in my TargetFile. Was this an intended change?
>
> On Thu, 27 Feb 2020 at 18:13, Daniel Gröber <dxld at darkboxed.org> wrote:
>
>> Hi,
>>
>> On Thu, Feb 27, 2020 at 03:43:35PM +0000, Tseen She wrote:
>> > Sorry for the spam, but I think this is definitely a bug/regression in
>> > ghc-8.8.1 (still present in 8.8.3).
>>
>> No worries
>>
>> > I don't know which commit introduced the problem, but it seems that
>> > getModSummary is no longer reporting the correct ms_hspp_opts, at least
>> for
>> > an in-memory file but it could also be for a file on disk as well (I
>> > haven't excluded that as a possibility).
>>
>> On a quick testcase I cannot reproduce this behaviour:
>>
>>     -- $ ghc -package ghc -package ghc-paths TargetContents.hs
>>     module Main where
>>
>>     import GHC
>>     import GHC.Paths (libdir)
>>     import MonadUtils
>>     import DynFlags
>>     import StringBuffer
>>     import Data.Time.Clock
>>
>>     main :: IO ()
>>     main = do
>>       defaultErrorHandler defaultFatalMessager defaultFlushOut $ do
>>         runGhc (Just libdir) $ do
>>           dflags0 <- getSessionDynFlags
>>           (dflags1, _, _)
>>               <- parseDynamicFlags dflags0 $ map noLoc ["-package",
>> "base"]
>>           _ <- setSessionDynFlags dflags1
>>
>>           t <- liftIO getCurrentTime
>>           setTargets [Target (TargetFile "Main.hs" Nothing) False (Just
>> (stringToStringBuffer buffer, t)) ]
>>
>>           _ <- depanal [] False
>>
>>           ms <- getModSummary (mkModuleName "Main")
>>           pm <- parseModule ms
>>
>>           liftIO $ print $ extensions $ ms_hspp_opts ms
>>
>>           _ <- typecheckModule pm
>>
>>           return ()
>>
>>     buffer = "{-# LANGUAGE PackageImports #-}\nimport \"base\"
>> Data.List\nmain = return ()"
>>
>> Running it it prints the PackageImports ext from the in-memory buffer just
>> fine:
>>
>>     $ ghc-8.8.1 -package ghc -package ghc-paths TargetContents.hs
>>     $ ./TargetContents
>>     [On PackageImports]
>>
>> One change in behaviour to note is that when doing this with <8.8 we first
>> need to create the Main.hs file so GHC doesn't complain about it missing.
>>
>> Maybe you can fiddle with the test case until it reflects what you're
>> doing?
>>
>> --Daniel
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.haskell.org/pipermail/ghc-devs/attachments/20200227/8ee6dc14/attachment.html>

From dxld at darkboxed.org  Thu Feb 27 23:20:47 2020
From: dxld at darkboxed.org (Daniel =?iso-8859-1?Q?Gr=F6ber?=)
Date: Fri, 28 Feb 2020 00:20:47 +0100
Subject: 8.8.x change in lookupModule / findModule semantics?
In-Reply-To: <CACKRAT2s8ohFkFfv+hjcf3HvBR+ML9Rj_L1KJNEydav1V0fh-w@mail.gmail.com>
 <CACKRAT0D3WsEbBLUmN2kd_LKOfTo5rLv3jZ0bwPxttJdgqYX+Q@mail.gmail.com>
Message-ID: <20200227232047.GD27770@Eli.lan>

Hi,

On Thu, Feb 27, 2020 at 08:37:12PM +0000, Tseen She wrote:
>       setTargets [Target (TargetFile "Main.hs" (Just $ Hsc HsSrcFile))
> False (Just (stringToStringBuffer buffer, t))]
>
> i.e. an explicit phase.
>
> I will just use Nothing in my TargetFile. Was this an intended change?

I found the reason for the change, my commit 0f9ec9d1ff ("Allow using
tagetContents for modules needing preprocessing") removes the special
casing in `preprocessFile` for in-memory buffers:

    -preprocessFile hsc_env src_fn mb_phase (Just (buf, _time))
    -  = do
    -        let dflags = hsc_dflags hsc_env
    -        let local_opts = getOptions dflags buf src_fn
    -
    -        (dflags', leftovers, warns)
    -            <- parseDynamicFilePragma dflags local_opts

It used to do a `parseDynamicFilePragma` which parses the LANGUAGE and
OPTIONS pragmas.

I think this change is actually for the better though, as this dflags
modification does not occur with a regular file, so we're actually being
more consistent.

I just tested this out and when setting the phase but not passing a buffer
8.6 will also fail to get the pragma in the ModSummary. When not giving an
explicit phase it works for 8.6, 8.8, with and without in-memory buffers
though. So I think that is the proper solution here, unless you can think
of a reason running the rest of the pipeline is a problem in this case?

On Thu, Feb 27, 2020 at 08:49:43PM +0000, Tseen She wrote:
> It is, however, a shame that lookupModule seems to require the entire file
> to parse / typecheck. That also smells like a regression, but not one that
> impacts me anymore.

I would still like to reproduce your problem but I'm again not sure how
you're triggering it, this is what I have so far:

    -- $ ghc -package ghc -package ghc-paths FindModule.hs
    module Main where

    import GHC
    import GHC.Paths (libdir)
    import MonadUtils
    import DynFlags
    import Module

    main :: IO ()
    main = do
      defaultErrorHandler defaultFatalMessager defaultFlushOut $ do
        runGhc (Just libdir) $ do
          dflags0 <- getSessionDynFlags
          (dflags1, _, _)
              <- parseDynamicFlags dflags0 $ map noLoc ["-package", "base"]
          _ <- setSessionDynFlags dflags1

          setTargets [Target (TargetFile "Main.hs" Nothing) False Nothing]

          _ <- load LoadAllTargets

          m <- findModule (mkModuleName "Main") Nothing
          liftIO $ print $ (unitIdFS (moduleUnitId m), moduleNameFS (moduleName m))

Running it with 8.8 and 8.4 i get the same results:

    $ ghc-$ver -package ghc -package ghc-paths FindModule.hs
    $ echo 'main = return()' > Main.hs # a working file
    $ ./FindModule
    ("main","Main")

    $ echo '=' > Main.hs  # a syntax error
    $ ./FindModule
    Main.hs:1:1: error:
        parse error on input ‘=’
        Perhaps you need a 'let' in a 'do' block?
        e.g. 'let x = 5' instead of 'x = 5'
      |
    1 | =
      | ^
    <command line>: module is not loaded: ‘Main’ (Main.hs)

so this seems to be consistent?

--Daniel
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 833 bytes
Desc: not available
URL: <http://mail.haskell.org/pipermail/ghc-devs/attachments/20200228/9d0fd5d9/attachment.sig>

From ts33n.sh3 at gmail.com  Fri Feb 28 08:35:14 2020
From: ts33n.sh3 at gmail.com (Tseen She)
Date: Fri, 28 Feb 2020 08:35:14 +0000
Subject: 8.8.x change in lookupModule / findModule semantics?
In-Reply-To: <20200227232047.GD27770@Eli.lan>
References: <CACKRAT2s8ohFkFfv+hjcf3HvBR+ML9Rj_L1KJNEydav1V0fh-w@mail.gmail.com>
 <CACKRAT0D3WsEbBLUmN2kd_LKOfTo5rLv3jZ0bwPxttJdgqYX+Q@mail.gmail.com>
 <20200227232047.GD27770@Eli.lan>
Message-ID: <CACKRAT1EH_E_v5sLQ+vQ3q60zBooAk-H4Q0OzOMqHNY3U4ai+A@mail.gmail.com>

Thanks you Daniel, that explains the mystery of the in-memory buffer.

With regards to findModule / lookupModule, it seems that the semantics have
not changed. This is perhaps an artefact of the previous issue: i.e. the
lookupModule was likely loading the disk version (not the in-memory)
version of a file.

If I discover anything else, I will create a new thread. I will also let
the group know when I cut a release, because this has been very helpful and
I would like to get a wider audience looking at the code to give me
recommendations for improvements (and especially to avoid reinventing the
wheel of what is already inside ghc).



On Thu, 27 Feb 2020 at 23:21, Daniel Gröber <dxld at darkboxed.org> wrote:

> Hi,
>
> On Thu, Feb 27, 2020 at 08:37:12PM +0000, Tseen She wrote:
> >       setTargets [Target (TargetFile "Main.hs" (Just $ Hsc HsSrcFile))
> > False (Just (stringToStringBuffer buffer, t))]
> >
> > i.e. an explicit phase.
> >
> > I will just use Nothing in my TargetFile. Was this an intended change?
>
> I found the reason for the change, my commit 0f9ec9d1ff ("Allow using
> tagetContents for modules needing preprocessing") removes the special
> casing in `preprocessFile` for in-memory buffers:
>
>     -preprocessFile hsc_env src_fn mb_phase (Just (buf, _time))
>     -  = do
>     -        let dflags = hsc_dflags hsc_env
>     -        let local_opts = getOptions dflags buf src_fn
>     -
>     -        (dflags', leftovers, warns)
>     -            <- parseDynamicFilePragma dflags local_opts
>
> It used to do a `parseDynamicFilePragma` which parses the LANGUAGE and
> OPTIONS pragmas.
>
> I think this change is actually for the better though, as this dflags
> modification does not occur with a regular file, so we're actually being
> more consistent.
>
> I just tested this out and when setting the phase but not passing a buffer
> 8.6 will also fail to get the pragma in the ModSummary. When not giving an
> explicit phase it works for 8.6, 8.8, with and without in-memory buffers
> though. So I think that is the proper solution here, unless you can think
> of a reason running the rest of the pipeline is a problem in this case?
>
> On Thu, Feb 27, 2020 at 08:49:43PM +0000, Tseen She wrote:
> > It is, however, a shame that lookupModule seems to require the entire
> file
> > to parse / typecheck. That also smells like a regression, but not one
> that
> > impacts me anymore.
>
> I would still like to reproduce your problem but I'm again not sure how
> you're triggering it, this is what I have so far:
>
>     -- $ ghc -package ghc -package ghc-paths FindModule.hs
>     module Main where
>
>     import GHC
>     import GHC.Paths (libdir)
>     import MonadUtils
>     import DynFlags
>     import Module
>
>     main :: IO ()
>     main = do
>       defaultErrorHandler defaultFatalMessager defaultFlushOut $ do
>         runGhc (Just libdir) $ do
>           dflags0 <- getSessionDynFlags
>           (dflags1, _, _)
>               <- parseDynamicFlags dflags0 $ map noLoc ["-package", "base"]
>           _ <- setSessionDynFlags dflags1
>
>           setTargets [Target (TargetFile "Main.hs" Nothing) False Nothing]
>
>           _ <- load LoadAllTargets
>
>           m <- findModule (mkModuleName "Main") Nothing
>           liftIO $ print $ (unitIdFS (moduleUnitId m), moduleNameFS
> (moduleName m))
>
> Running it with 8.8 and 8.4 i get the same results:
>
>     $ ghc-$ver -package ghc -package ghc-paths FindModule.hs
>     $ echo 'main = return()' > Main.hs # a working file
>     $ ./FindModule
>     ("main","Main")
>
>     $ echo '=' > Main.hs  # a syntax error
>     $ ./FindModule
>     Main.hs:1:1: error:
>         parse error on input ‘=’
>         Perhaps you need a 'let' in a 'do' block?
>         e.g. 'let x = 5' instead of 'x = 5'
>       |
>     1 | =
>       | ^
>     <command line>: module is not loaded: ‘Main’ (Main.hs)
>
> so this seems to be consistent?
>
> --Daniel
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.haskell.org/pipermail/ghc-devs/attachments/20200228/6ac8a18e/attachment.html>

From ts33n.sh3 at gmail.com  Fri Feb 28 11:28:11 2020
From: ts33n.sh3 at gmail.com (Tseen She)
Date: Fri, 28 Feb 2020 11:28:11 +0000
Subject: hsinspect-0.0.11 released
Message-ID: <CACKRAT0_Lu-UXe4qn5QMhZPX8kTArtiMD=zy_NZwPmJGRMohBQ@mail.gmail.com>

Hello ghc developers,

With many thanks to Daniel Grober, I have released 0.0.11 of hsinspect to
Hackage, which now supports ghc versions 8.4.x, 8.6.x and 8.8.x.

I would like to bring this tool to the attention of this community of ghc
developers because it is using the ghc api exclusively to provide semantic
information about .hs files, so it may be of interest to you to see a tool
using your api in the wild.

And selfishly, I learnt so much from Daniel's reviews of two short
functions recently (which were subject to a change in ghc behavour in 8.8)
that I would very much welcome further reviews by this group.

The source code is split across three parts:

The plugin (an alternative to hie-bios, which didn't work well for me):

https://gitlab.com/tseenshe/hsinspect/-/blob/master/plugin/GhcFlags/Plugin.hs

The library, one file per feature:

https://gitlab.com/tseenshe/hsinspect/-/tree/master/library/HsInspect

and the Executable (setup, then just call the relevant library):

https://gitlab.com/tseenshe/hsinspect/-/blob/master/exe/Main.hs

My usecase is to provide semantic information for my Emacs mode, which is a
completely from-scratch haskell-mode that I've been working on in my free
time for the past year:

  https://gitlab.com/tseenshe/haskell-tng.el

For anybody curious about the wider project, there are some screenshots of
the features in the README, which also fully justifies why I undertook this
endeavour: i.e. why I don't believe I could achieve what I wanted in
haskell-mode or LSP.

I am content for this tool to be used just by myself, my work colleagues,
and anybody who is interested in contributing.

Best regards,
Tseen She
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.haskell.org/pipermail/ghc-devs/attachments/20200228/e0a765d6/attachment.html>

From juhpetersen at gmail.com  Sat Feb 29 09:51:53 2020
From: juhpetersen at gmail.com (Jens Petersen)
Date: Sat, 29 Feb 2020 17:51:53 +0800
Subject: [ANNOUNCE] GHC 8.8.3 is now available
In-Reply-To: <877e0ba0yq.fsf@smart-cactus.org>
References: <87r1zz1f09.fsf@smart-cactus.org> <87a757afin.fsf@smart-cactus.org>
 <877e0ba0yq.fsf@smart-cactus.org>
Message-ID: <CAEDGBq+VuNTuqv6XU43wadPpt-qJt_sMg57NqJMz=HBLDuGKDw@mail.gmail.com>

On Tue, 25 Feb 2020 at 09:52, Ben Gamari <ben at well-typed.com> wrote:

> > Ben Gamari <ben at well-typed.com> writes:
> >> The GHC team is proud to announce the release of GHC 8.8.3.
>

Thank you, if you are in Fedora you can now install this with:

`sudo dnf --enablerepo=updates-testing-modular` module install ghc:8.8`

Jens
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.haskell.org/pipermail/ghc-devs/attachments/20200229/1acfc999/attachment.html>