Unicode lexer issue?
Ralf Lammel
Ralf.Lammel at microsoft.com
Mon Jan 9 20:58:19 EST 2006
Simon,
I am just trying to build GHC under Windows. (It worked previously.)
(I am using Alex version 2.0.1, if this is relevant.)
Make dies this way:
parser/Lexer.x:561:10:
lexical error in string/character literal at character '\8704'
<<ghc: 163455240 bytes, 14 GCs, 481756/944180 avg/max bytes residency (2
samples
), 16M in use, 0.01 INIT (0.00 elapsed), 0.55 MUT (0.55 elapsed), 0.05
GC (0.06
elapsed) :ghc>>
make[2]: *** [stage2/parser/Lexer.o] Error 1
make[2]: Leaving directory
`/cygdrive/c/HOME/projects/ghc-fptools/ghc/compiler'
make[1]: *** [stage2] Error 2
make[1]: Leaving directory `/cygdrive/c/HOME/projects/ghc-fptools'
make: *** [bootstrap2] Error 2
Does this ring any bell?
Is this related to the recent check-in (see below).
The only alerting issue that I saw:
$ ./configure --host=i386-unknown-mingw32 --with-gcc=c:/mingw/bin/gcc
configure: WARNING: If you wanted to set the --build type, don't use
--host.
If a cross compiler is detected then cross compile mode will be
used.
Thanks for a quick idea.
Ralf
> -----Original Message-----
> From: cvs-all-bounces at haskell.org [mailto:cvs-all-bounces at haskell.org]
On
> Behalf Of Simon Marlow
> Sent: Friday, January 06, 2006 8:30 AM
> To: cvs-ghc at haskell.org
> Subject: cvs commit: fptools/ghc/compiler HsVersions.h Makefile
> fptools/ghc/compiler/basicTypes Id.lhs Literal.lhs MkId.lhs Module.lhs
> Name.lhs OccName.lhs RdrName.lhs fptools/ghc/compiler/cmm CLabel.hs
Cmm.hs
> CmmLex.x CmmParse.y ...
>
> simonmar 2006/01/06 08:30:19 PST
>
> Modified files:
> ghc/compiler HsVersions.h Makefile
> ghc/compiler/basicTypes Id.lhs Literal.lhs MkId.lhs Module.lhs
> Name.lhs OccName.lhs RdrName.lhs
> ghc/compiler/cmm CLabel.hs Cmm.hs CmmLex.x CmmParse.y
> PprC.hs PprCmm.hs
> ghc/compiler/codeGen CgProf.hs CgUtils.hs ClosureInfo.lhs
> ghc/compiler/deSugar Check.lhs DsForeign.lhs DsMeta.hs
> DsUtils.lhs
> ghc/compiler/ghci ByteCodeGen.lhs ByteCodeLink.lhs
> InteractiveUI.hs
> ghc/compiler/hsSyn Convert.lhs HsDecls.lhs HsUtils.lhs
> ghc/compiler/iface LoadIface.lhs MkIface.lhs
> ghc/compiler/main DriverMkDepend.hs DriverPipeline.hs
> Finder.lhs GHC.hs HscTypes.lhs
> ghc/compiler/nativeGen PprMach.hs
> ghc/compiler/ndpFlatten FlattenMonad.hs
> ghc/compiler/parser Ctype.lhs Lexer.x Parser.y.pp
> ParserCore.y RdrHsSyn.lhs
> ghc/compiler/prelude PrelNames.lhs PrelRules.lhs PrimOp.lhs
> TysPrim.lhs TysWiredIn.lhs
> ghc/compiler/profiling CostCentre.lhs
> ghc/compiler/rename RnEnv.lhs RnExpr.lhs RnNames.lhs
> ghc/compiler/simplCore SetLevels.lhs SimplMonad.lhs
> Simplify.lhs
> ghc/compiler/stgSyn CoreToStg.lhs
> ghc/compiler/typecheck Inst.lhs TcClassDcl.lhs TcGenDeriv.lhs
> TcInstDcls.lhs TcRnDriver.lhs
> TcSplice.lhs
> ghc/compiler/types TypeRep.lhs
> ghc/compiler/utils Binary.hs BufWrite.hs FastString.lhs
> FastTypes.lhs Pretty.lhs StringBuffer.lhs
> Added files:
> ghc/compiler/utils Encoding.hs
> Removed files:
> ghc/compiler/utils PrimPacked.lhs UnicodeUtil.lhs
> Log:
> Add support for UTF-8 source files
>
> GHC finally has support for full Unicode in source files. Source
> files are now assumed to be UTF-8 encoded, and the full range of
> Unicode characters can be used, with classifications recognised
using
> the implementation from Data.Char. This incedentally means that
only
> the stage2 compiler will recognise Unicode in source files, because
I
> was too lazy to port the unicode classifier code into libcompat.
>
> Additionally, the following synonyms for keywords are now
recognised:
>
> forall symbol (U+2200) forall
> right arrow (U+2192) ->
> left arrow (U+2190) <-
> horizontal ellipsis (U+22EF) ..
>
> there are probably more things we could add here.
>
> This will break some source files if Latin-1 characters are being
used.
> In most cases this should result in a UTF-8 decoding error. Later
on
> if we want to support more encodings (perhaps with a pragma to
specify
> the encoding), I plan to do it by recoding into UTF-8 before
parsing.
>
> Internally, there were some pretty big changes:
>
> - FastStrings are now stored in UTF-8
>
> - Z-encoding has been moved right to the back end. Previously we
> used to Z-encode every identifier on the way in for simplicity,
> and only decode when we needed to show something to the user.
> Instead, we now keep every string in its UTF-8 encoding, and
> Z-encode right before printing it out. To avoid Z-encoding the
> same string multiple times, the Z-encoding is cached inside the
> FastString the first time it is requested.
>
> This speeds up the compiler - I've measured some definite
> improvement in parsing at least, and I expect compilations
overall
> to be faster too. It also cleans up a lot of cruft from the
> OccName interface. Z-encoding is nicely hidden inside the
> Outputable instance for Names & OccNames now.
>
> - StringBuffers are UTF-8 too, and are now represented as
> ForeignPtrs.
>
> - I've put together some test cases, not by any means exhaustive,
> but there are some interesting UTF-8 decoding error cases that
> aren't obvious. Also, take a look at unicode001.hs for a demo.
>
> Revision Changes Path
> 1.37 +11 -3 fptools/ghc/compiler/HsVersions.h
> 1.310 +1 -1 fptools/ghc/compiler/Makefile
> 1.133 +7 -12 fptools/ghc/compiler/basicTypes/Id.lhs
> 1.49 +6 -3 fptools/ghc/compiler/basicTypes/Literal.lhs
> 1.126 +2 -2 fptools/ghc/compiler/basicTypes/MkId.lhs
> 1.61 +18 -24 fptools/ghc/compiler/basicTypes/Module.lhs
> 1.120 +20 -18 fptools/ghc/compiler/basicTypes/Name.lhs
> 1.68 +111 -384 fptools/ghc/compiler/basicTypes/OccName.lhs
> 1.44 +9 -17 fptools/ghc/compiler/basicTypes/RdrName.lhs
> 1.19 +2 -2 fptools/ghc/compiler/cmm/CLabel.hs
> 1.9 +2 -2 fptools/ghc/compiler/cmm/Cmm.hs
> 1.5 +2 -2 fptools/ghc/compiler/cmm/CmmLex.x
> 1.10 +5 -1 fptools/ghc/compiler/cmm/CmmParse.y
> 1.24 +10 -13 fptools/ghc/compiler/cmm/PprC.hs
> 1.10 +3 -1 fptools/ghc/compiler/cmm/PprCmm.hs
> 1.11 +2 -2 fptools/ghc/compiler/codeGen/CgProf.hs
> 1.19 +9 -4 fptools/ghc/compiler/codeGen/CgUtils.hs
> 1.72 +3 -3 fptools/ghc/compiler/codeGen/ClosureInfo.lhs
> 1.43 +3 -2 fptools/ghc/compiler/deSugar/Check.lhs
> 1.96 +4 -5 fptools/ghc/compiler/deSugar/DsForeign.lhs
> 1.78 +6 -6 fptools/ghc/compiler/deSugar/DsMeta.hs
> 1.83 +7 -6 fptools/ghc/compiler/deSugar/DsUtils.lhs
> 1.112 +13 -12 fptools/ghc/compiler/ghci/ByteCodeGen.lhs
> 1.46 +6 -6 fptools/ghc/compiler/ghci/ByteCodeLink.lhs
> 1.220 +5 -5 fptools/ghc/compiler/ghci/InteractiveUI.hs
> 1.70 +4 -3 fptools/ghc/compiler/hsSyn/Convert.lhs
> 1.115 +2 -2 fptools/ghc/compiler/hsSyn/HsDecls.lhs
> 1.12 +2 -2 fptools/ghc/compiler/hsSyn/HsUtils.lhs
> 1.37 +2 -2 fptools/ghc/compiler/iface/LoadIface.lhs
> 1.37 +3 -3 fptools/ghc/compiler/iface/MkIface.lhs
> 1.55 +1 -1 fptools/ghc/compiler/main/DriverMkDepend.hs
> 1.217 +2 -2 fptools/ghc/compiler/main/DriverPipeline.hs
> 1.87 +3 -3 fptools/ghc/compiler/main/Finder.lhs
> 1.53 +1 -1 fptools/ghc/compiler/main/GHC.hs
> 1.143 +1 -1 fptools/ghc/compiler/main/HscTypes.lhs
> 1.18 +3 -9 fptools/ghc/compiler/nativeGen/PprMach.hs
> 1.12 +3 -3 fptools/ghc/compiler/ndpFlatten/FlattenMonad.hs
> 1.12 +97 -95 fptools/ghc/compiler/parser/Ctype.lhs
> 1.32 +164 -78 fptools/ghc/compiler/parser/Lexer.x
> 1.39 +3 -3 fptools/ghc/compiler/parser/Parser.y.pp
> 1.25 +6 -6 fptools/ghc/compiler/parser/ParserCore.y
> 1.93 +4 -5 fptools/ghc/compiler/parser/RdrHsSyn.lhs
> 1.101 +10 -11 fptools/ghc/compiler/prelude/PrelNames.lhs
> 1.46 +2 -2 fptools/ghc/compiler/prelude/PrelRules.lhs
> 1.109 +5 -5 fptools/ghc/compiler/prelude/PrimOp.lhs
> 1.52 +2 -2 fptools/ghc/compiler/prelude/TysPrim.lhs
> 1.98 +10 -8 fptools/ghc/compiler/prelude/TysWiredIn.lhs
> 1.43 +5 -7 fptools/ghc/compiler/profiling/CostCentre.lhs
> 1.200 +6 -3 fptools/ghc/compiler/rename/RnEnv.lhs
> 1.147 +1 -2 fptools/ghc/compiler/rename/RnExpr.lhs
> 1.204 +7 -5 fptools/ghc/compiler/rename/RnNames.lhs
> 1.67 +6 -6 fptools/ghc/compiler/simplCore/SetLevels.lhs
> 1.61 +1 -2 fptools/ghc/compiler/simplCore/SimplMonad.lhs
> 1.167 +1 -2 fptools/ghc/compiler/simplCore/Simplify.lhs
> 1.120 +2 -2 fptools/ghc/compiler/stgSyn/CoreToStg.lhs
> 1.159 +2 -4 fptools/ghc/compiler/typecheck/Inst.lhs
> 1.154 +1 -2 fptools/ghc/compiler/typecheck/TcClassDcl.lhs
> 1.120 +6 -6 fptools/ghc/compiler/typecheck/TcGenDeriv.lhs
> 1.182 +1 -2 fptools/ghc/compiler/typecheck/TcInstDcls.lhs
> 1.134 +3 -3 fptools/ghc/compiler/typecheck/TcRnDriver.lhs
> 1.67 +4 -4 fptools/ghc/compiler/typecheck/TcSplice.lhs
> 1.45 +2 -2 fptools/ghc/compiler/types/TypeRep.lhs
> 1.25 +24 -63 fptools/ghc/compiler/utils/Binary.hs
> 1.2 +6 -19 fptools/ghc/compiler/utils/BufWrite.hs
> 1.55 +349 -385 fptools/ghc/compiler/utils/FastString.lhs
> 1.5 +1 -1 fptools/ghc/compiler/utils/FastTypes.lhs
> 1.30 +1 -11 fptools/ghc/compiler/utils/Pretty.lhs
> 1.68 +118 -107 fptools/ghc/compiler/utils/StringBuffer.lhs
> _______________________________________________
> Cvs-ghc mailing list
> Cvs-ghc at haskell.org
> http://www.haskell.org/mailman/listinfo/cvs-ghc
More information about the Glasgow-haskell-bugs
mailing list