[GHC] #14473: Implement Underscores in Numeric Literals Proposal (NumericUnderscores extension)

[GHC]

#14473: Implement Underscores in Numeric Literals Proposal (NumericUnderscores
extension)
-------------------------------------+-------------------------------------
        Reporter:  takenobu          |                Owner:  takenobu
            Type:  task              |               Status:  patch
        Priority:  normal            |            Milestone:  8.6.1
       Component:  Compiler          |              Version:  8.2.1
      Resolution:                    |             Keywords:
Operating System:  Unknown/Multiple  |         Architecture:
                                     |  Unknown/Multiple
 Type of failure:  None/Unknown      |            Test Case:
      Blocked By:                    |             Blocking:
 Related Tickets:  #13126 #9224      |  Differential Rev(s):  Phab:D4235
       Wiki Page:                    |
-------------------------------------+-------------------------------------

Comment (by Ben Gamari <ben@…>):

 In [changeset:"4a13c5b1f4beb53cbf1f3529acdf3ba37528e694/ghc"
 4a13c5b1/ghc]:
 {{{
 #!CommitTicketReference repository="ghc"
 revision="4a13c5b1f4beb53cbf1f3529acdf3ba37528e694"
 Implement underscores in numeric literals (NumericUnderscores extension)

 Implement the proposal of underscores in numeric literals.
 Underscores in numeric literals are simply ignored.

 The specification of the feature is available here:
 https://github.com/ghc-proposals/ghc-proposals/blob/master/proposals/000
 9-numeric-underscores.rst

 For a discussion of the various choices:
 https://github.com/ghc-proposals/ghc-proposals/pull/76

 Implementation detail:

 * Added dynamic flag
   * `NumericUnderscores` extension flag is added for this feature.

 * Alex "Regular expression macros" in Lexer.x
   * Add `@numspc` (numeric spacer) macro to represent multiple
     underscores.
   * Modify `@decimal`, `@decimal`, `@binary`, `@octal`, `@hexadecimal`,
     `@exponent`, and `@bin_exponent` macros to include `@numspc`.

 * Alex "Rules" in Lexer.x
   * To be simpler, we have only the definitions with underscores.
     And then we have a separate function (`tok_integral` and `tok_frac`)
     that validates the literals.

 * Validation functions in Lexer.x
   * `tok_integral` and `tok_frac` functions validate
     whether contain underscores or not.
     If `NumericUnderscores` extensions are not enabled,
     check that there are no underscores.
   * `tok_frac` function is created by merging `strtoken` and
     `init_strtoken`.
   * `init_strtoken` is deleted. Because it is no longer used.

 * Remove underscores from target literal string
   * `parseUnsignedInteger`, `readRational__`, and `readHexRational} use
     the customized `span'` function to remove underscores.

 * Added Testcase
   * testcase for NumericUnderscores enabled.
       NumericUnderscores0.hs and NumericUnderscores1.hs
   * testcase for NumericUnderscores disabled.
       NoNumericUnderscores0.hs and NoNumericUnderscores1.hs
   * testcase to invalid pattern for NumericUnderscores enabled.
       NumericUnderscoresFail0.hs and NumericUnderscoresFail1.hs

 Test Plan: `validate` including the above testcase

 Reviewers: goldfire, bgamari

 Reviewed By: bgamari

 Subscribers: carter, rwbarton, thomie

 GHC Trac Issues: #14473

 Differential Revision: https://phabricator.haskell.org/D4235
 }}}

-- 
Ticket URL: <http://ghc.haskell.org/trac/ghc/ticket/14473#comment:8>
GHC <http://www.haskell.org/ghc/>
The Glasgow Haskell Compiler