Status of recent DWARF work

[Ben Gamari]

Hello everyone!

Over the last month or so I've been working on better incorporating
Peter Wortmann's DWARF work into the GHC. The immedate goals are
two-fold,

  * Allow access to reified stacks in the RTS and user programs
  * Introduce some basic statistical profiling functionality into the
    RTS

While I have been slowly writing up a more general description of the
work, this work has already gone too long unannounced so I'm just going
to plop the list of relevant Differentials here.

Note that this is also available on the Wiki
(https://ghc.haskell.org/trac/ghc/wiki/DWARF/80Status), complete with
hyperlinked Diffs.

The hope here is that this summary will make it easier for others to
review this work.

Cheers,

- Ben


Notation
--------

These are in various states of completion which I'll encode with the
following designations,

 * MERGED: Already merged to `master`, listed here for completeness

 * REVIEW: Believed to be finished with no expectation of major rework
           being necessary. Should be in merge-worthy condition, pending
           code review.

 * RFC:    Code done, builds, and tested to some extent; needs design
           review.

 * EXPLOR: Exploratory work, may not even build but included to document
           the expected future direction of the work.

 * IDEA:   Just an idea, no implementation yet.

I'll list the commits in the order of their logical progression,


The Patches
-----------

These preparatory commits address a few bugs and deficiencies in the
current DWARF production implementation,

 * [MERGED]  Phab:D1172 Dwarf: Fix DW_AT_use_UTF8 attribute
 * [MERGED]  Phab:D1173 Dwarf: Produce {low,high}_pc attributes for compilation units
 * [MERGED]  Phab:D1174 Dwarf: Produce .dwarf_aranges section
 * [MERGED]  Phab:D1220 Dwarf: Ensure block length is encoded correctly

These introduce DWARF parsing and stack unwinding to the RTS by
introducing an optional dependency on `libdw`. This is the same library
used by `perf`.

 * [REVIEW]  Phab:D1196: Libdw: Add libdw-based stack unwinding
 * [REVIEW]  Phab:D1197: Signals: Print backtrace on SIGUSR2

With the RTS groundwork in place we can plumb things in for use by user
programs,

 * [REVIEW]  Phab:D1198: Provide DWARF-based backtraces to Haskell-land

Unfortunately up until this point we have no ability to unwind out of
Haskell code back to the C stack. These commits introduce the ability to
unwind all the way back to `_start`,

 * [REVIEW] Phab:D1224: Dwarf: Preserve stack pointer register
 * [REVIEW] Phab:D1225: cmm: Expose machine's stack pointer and return address registers
 * [REVIEW] Phab:D1223: StgStartup: Add DWARF unwinding annotations for stg_stop_thread

This should be enough to get reasonable backtraces for error-handling
and reporting.

However, we also want profiling. For this, however, DWARF annotations
alone aren't sufficient. The plan here is to incorporate a more details
From the source notes produced by GHC into the DWARF vendor extension
DIEs.

The RTS then takes these DIEs during program initialiation and
emits a representation of them to the eventlog for later use by analysis
code,

 * [RFC]    Phab:D1279: Output source notes in extended DWARF DIEs
 * [RFC]    Phab:D1280: rts: Emit debug information about program to event log
 * [REVIEW] Phab:D1281: Support multiple debug output levels

Now we have everything necessary to add some basic statistical
profiling. Here we collect samples from heap checks and black hole block
events and emit them to the event log,

 * [RFC]    Phab:D1215: A simple statistical profiler
 * [RFC]    Phab:D1216: StatProfile: Heap and black-hole sampling

This all appears to work and I have some rather crude analysis tools
which I should really clean up a bit. Ideally someone would dust off
Peter's Threadscope integration as this would make for an extremely
compelling performance analysis story.

I also have yet to examine the impact of profiling on performance when
not enabled. In principle it should be cheap enough to compile in
unconditionally (although event log support is needed) but this needs to
be measured.

In his prototype implementation Peter also had the lovely ability to
preserve simplified Core for later examination. This would be a nice
thing to have but probably won't happen for 8.0. The challenge here is
recording the Core fragments without introducing enormous amounts of
redundancy.

 * [EXPLOR] Phab:D1213: Core notes
 * [IDEA]   Record tree of Core fragments into DWARF DIEs

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 472 bytes
Desc: not available
URL: <http://mail.haskell.org/pipermail/ghc-devs/attachments/20151002/93a84515/attachment.sig>