[Haskell] ByteString tokens

[Duncan Coutts]

On Mon, 2006-10-02 at 14:20 -0700, Chad Scherrer wrote:
> The Haddock documentation days there is a function
> 
> tokens :: (Char -> Bool) -> ByteString -> [ByteString]
> in Data.ByteString.Lazy.Char8

> Any idea where it went?

We removed it for the moment. As ByteString has been moved into the base
package, the api we provide now will have to be stable for some time to
come. We wanted to start with a minimal api since that is easier to keep
stable.

In particular, the rationale for removing 'tokens' was that it is not
clear that the current semantics is the best for such an important name.
We'd rather wait for more of a consensus on what a function called
'tokens' should do. There's quite a range of options.


If you need the previous implementation you can use this:

-- | Like 'splitWith', except that sequences of adjacent separators are
-- treated as a single separator. eg.
--
-- > tokens (=='a') "aabbaca" == ["bb","c"]
--
tokens :: (Word8 -> Bool) -> ByteString -> [ByteString]
tokens f = List.filter (not.null) . splitWith f


Duncan