[GHC] #10951: HPC program has poor error reporting / strange CLI in general

[GHC]

#10951: HPC program has poor error reporting / strange CLI in general
-------------------------------------+-------------------------------------
        Reporter:  mgsloan           |                Owner:
            Type:  bug               |               Status:  new
        Priority:  low               |            Milestone:
       Component:  Code Coverage     |              Version:  7.10.2
      Resolution:                    |             Keywords:
Operating System:  Unknown/Multiple  |         Architecture:
                                     |  Unknown/Multiple
 Type of failure:  None/Unknown      |            Test Case:
      Blocked By:                    |             Blocking:
 Related Tickets:                    |  Differential Rev(s):
-------------------------------------+-------------------------------------
Description changed by mgsloan:

Old description:

> 1. Many erroneous usages of the hpc program result in reporting 100%
> (0/0) coverage. Proposed resolution:
>
>   * Report an error and exit with failure.  I can't think of a legitimate
> case where you would want a report that doesn't consider any expressions
> / declarations.
>
> * If a --include flag is specified, I think it would be valuable to
> output all the module names, so that it's clearer why the --include flag
> didn't work.
>
> * At the very least, (0/0) coverage shouldn't be reported as "100%".
>
> 2. Positional arguments after the initial tix file get interpreted as
> --include filters, filtering which modules / packages are considered.  By
> default, HPC includes all modules in the report, unless include filters
> are supplied.  If you accidentally provide a positional argument which
> isn't a valid thing for "--include", then your reports will always be
> (0/0) coverage.  Proposed resolution:
>
> * Deprecate using positional arguments for include filters.  This way it
> will complain about this instead of yielding trivial (0/0) coverage
> reports
>
> 3. Ideally, you'd be able to pass in multiple tix files and combine them
> in memory instead of using "hpc combine" and generating intermediate tix
> files.  This would be a better usage of positional arguments than include
> filters.  For now, the additional tix files would need to be flag
> arguments.  This also isn't all that useful until
> https://ghc.haskell.org/trac/ghc/ticket/1853 is fixed.
>
> 4. `--srcdir`, `--hpcdir`, and `--reset-hpcdirs` interface has strange
> semantics for searching for mix files.  It literally does something
> roughly like `[sd </> hd | sd <- srcDirs, hd <- hpcDirs]`, and uses this
> when looking for mix files.  This would be OK, except that the `srcdir`
> must be the work dir used for compilation, in order for source markup to
> be generated.  This means that you are tied to having your hpcdir be a
> sub-directory of the compilation work dir.  Proposed resolution:
>
> * Add `--mixdir`, specifying a path to a mix file directory.  Can be
> supplied multiple times.  If `--mixdir` is provided, then it's an error
> to also use `--hpcdir` / `--reset-hpcdirs`.
>
> * If GHC is provided an absolute `-hpcdir`, then use it rather than
> appending to the CWD.
>
> Instead of fixing the hpc program, I have considered forking it and
> giving it a new executable name, so that the CLI and behavior can be
> entirely changed.  Thoughts on this alternative?  It is appealing because
> it seems tricky to fix the hpc program's CLI while ensuring that
> backwards compatibility is maintained.
>
> I am keen on working on fixing this situation, but I want consensus from
> y'all about the approach that should be taken.
>
> Maybe it makes sense for this hpc-ng program to be developed outside of
> the GHC repo?  This way it can use more recent libraries for generating
> HTML and parsing command line arguments.

New description:

 1. Many erroneous usages of the hpc program result in reporting 100% (0/0)
 coverage. Proposed resolution:

   * Report an error and exit with failure.  I can't think of a legitimate
 case where you would want a report that doesn't consider any expressions /
 declarations.

 * If a --include flag is specified, I think it would be valuable to output
 all the module names, so that it's clearer why the --include flag didn't
 work.

 * At the very least, (0/0) coverage shouldn't be reported as "100%".

 2. Positional arguments after the initial tix file get interpreted as
 --include filters, filtering which modules / packages are considered.  By
 default, HPC includes all modules in the report, unless include filters
 are supplied.  If you accidentally provide a positional argument which
 isn't a valid thing for "--include", then your reports will always be
 (0/0) coverage.  Proposed resolution:

 * Deprecate using positional arguments for include filters.  This way it
 will complain about this instead of yielding trivial (0/0) coverage
 reports

 3. Ideally, you'd be able to pass in multiple tix files and combine them
 in memory instead of using "hpc combine" and generating intermediate tix
 files.  This would be a better usage of positional arguments than include
 filters.  For now, the additional tix files would need to be flag
 arguments.  This also isn't all that useful until
 https://ghc.haskell.org/trac/ghc/ticket/1853 is fixed.

 4. `--srcdir`, `--hpcdir`, and `--reset-hpcdirs` interface has strange
 semantics for searching for mix files.  It literally does something
 roughly like `[sd </> hd | sd <- srcDirs, hd <- hpcDirs]`, and uses this
 when looking for mix files.  This would be OK, except that the `srcdir`
 must be the work dir used for compilation, in order for source markup to
 be generated.  This means that you are tied to having your hpcdir be a
 sub-directory of the compilation work dir.  Proposed resolution:

 * Add `--mixdir`, specifying a path to a mix file directory.  Can be
 supplied multiple times.  If `--mixdir` is provided, then it's an error to
 also use `--hpcdir` / `--reset-hpcdirs`.

 * If GHC is provided an absolute `-hpcdir`, then use it rather than
 appending to the CWD.

 5. If anything goes wrong when matching up a module in a tix file with mix
 metadata, the error is "can not find MODNAME in DIRS". (see the code here:
 https://github.com/ghc/packages-
 hpc/blob/fb14d3428ba24d36e779736989dae3092a50a957/Trace/Hpc/Mix.hs#L87)
 This is quite confusing because this same message is used for all of the
 following cases:

 * There's a parse error of the mix file

 * Any IO error happens

 * The module hash mismatches

 Instead of fixing the hpc program, I have considered forking it and giving
 it a new executable name, so that the CLI and behavior can be entirely
 changed.  Thoughts on this alternative?  It is appealing because it seems
 tricky to fix the hpc program's CLI while ensuring that backwards
 compatibility is maintained.

 I am keen on working on fixing this situation, but I want consensus from
 y'all about the approach that should be taken.

 Maybe it makes sense for this hpc-ng program to be developed outside of
 the GHC repo?  This way it can use more recent libraries for generating
 HTML and parsing command line arguments.

--

--
Ticket URL: <http://ghc.haskell.org/trac/ghc/ticket/10951#comment:2>
GHC <http://www.haskell.org/ghc/>
The Glasgow Haskell Compiler