[Haskell-cafe] GHC lexer questions

[Phyx]

Hi all,

I have a few questions about the GHC lexer (currently using GHC Glasgow
Haskell Compiler, Version 6.13.20100320, for Haskell 98, stage 2 booted by
GHC version 6.12.1)

 

And I notice that when invoking the lexer I get a few unexpected results
that I was hoping someone could clarify.

 

The first part has to do with braces. Opening braces are lexed fine, but the
closing ones throw an error. I assume the lexer is keeping some kind of
state and trying to do brace matching? (but isn't this supposed to be done
by the parser?)

 

?parseLine("{")->tag

cStatelessParseResultSOk

?parseLine("}")->tag

cStatelessParseResultSFailed

 

and

 

?parseLine("{-")->tag

cStatelessParseResultSOk

?parseLine("-}")->tag

cStatelessParseResultSFailed

 

and in a statement "let foo = .." "foo" isn't lexed as an identifier, "f" is
lexed as a ITvocurly and the "oo" as an identifier. I find this  rather odd,
and maybe it's me doing something wrong but doubt it.

 

The original Haskell code I call from FFI is 

 

lexSourceString :: String -> IO (StatelessParseResult [Located Token])

lexSourceString source =  

 do

   buffer <- stringToStringBuffer source

   let srcLoc  = mkSrcLoc (mkFastString "internal:string") 1 1

   let dynFlag = defaultDynFlags 

   let result  = lexTokenStream buffer srcLoc dynFlag

   return $ convert result

 

-- | convert the build in ParseResult to out custom StateLessParseResult

--   mainly because the State is difficult to marshal and we don't really
seem to need it

convert :: ParseResult a -> StatelessParseResult a

convert (POk _ ty)        = SOk ty

convert (PFailed src msg) = SFailed src (showSDocDump msg)

 

-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://www.haskell.org/pipermail/haskell-cafe/attachments/20100422/d3187749/attachment-0001.html