Tokenizing Strings

[David Roundy]

On Wed, Apr 02, 2003 at 11:26:46AM +1000, jamesd@mena.org.au wrote:
> 
> in this case, I have a string containing multiples fields seperated by *two*
> blank lines (\n\n). I can't just break on the newline character, as single
> newline characters can be found inside each field.
> 
> any idea how I can do this without too much hassle?

Heres a somewhat stupid, and somewhat compact way of doing this, assuming
that three blank lines counts as an identical delimiter.  It doesn't even
pretend to attempt to solve the general case of multiple character
delimiters, but instead breaks on empty lines.

tok s = map (concatMap (++ "\n")) $ break null $ lines s

(warning: untested code!)

David Roundy