[Haskell-cafe] Alex Lexer Performance Issues

[Malcolm Wallace]

On 22 Jun 2011, at 15:53, Tristan Ravitch wrote:

> On Wed, Jun 22, 2011 at 07:48:40AM +0100, Stephen Tetley wrote:
>> How fast is good old String rather than ByteString?
>> 
>> For lexing, String is a good fit (cheap deconstruction at the head /
>> front). For your particular case, maybe it loses due to the large file
>> size, maybe it doesn't...
> 
> I gave it a shot and the percentages in the profile are approximately
> the same (and peak memory usage was about double).  I might end up
> having to parse the original binary format instead of the text format.

There is an old folklore that lexing is usually the most expensive phase of any compiler-like traversal.  50% of time and space expended on lexing was pretty common twenty years ago.

Regards,
    Malcolm