[Haskell-cafe] Haskell audio libraries & audio formats

[Henning Thielemann]

On Sun, 24 Aug 2008, Eric Kidd wrote:

> Greetings, Haskell folks!
>
> I'm working on a toy program for drum synthesis.  This has lead me to
> explore the various sound-related libraries in Hackage.  Along the way,
> I've learned several things:
>
>  1. There's a lot of Haskell sound libraries, but no agreement on how
>     to represent buffers of audio data.

In my sound synthesis library
    http://darcs.haskell.org/synthesizer/
  I use mainly two representations:

1. Chunky sequences based on StorableVector. This is for efficient storage 
and functions that need to process data in a non-causal way. They are not 
good for fusion.
    http://darcs.haskell.org/synthesizer/src/Synthesizer/Storable/Signal.hs

2. A Stream like sequences without wrappers. In fact, they are no data 
storages but generator functions equipped with a seed. I use them for 
causal signal processes. I think that most functions needed for drum 
synthesis are causal. By inlining them, many subsequent causal signal 
processes are fused to one.
    http://darcs.haskell.org/synthesizer/src/Synthesizer/State/Signal.hs


>  2. Some of the most useful sound libraries aren't listed in Hackage's
>     "Sound" section, including HCodecs, SDL-mixer and hogg.

my library is also not at Hackage, it's too much experimental currently

>  3. The Haskell OpenAL bindings are great, but ALUT has been removed
>     from MacOS 10.5.  Of course, a pure-Haskell version of ALUT would
>     be pretty easy to write, and it could be used as a fallback.
>  4. '[[a]]' is _not_ a good representation for multi-channel audio
>     data.  Manipulating even 10 seconds of audio is noticeably slow.

Using the signal representations I mentioned above, I can synthesize some 
songs in realtime. For stereo signals I use Signal (a,a). However, a 
StereoFrame data type with Storable instance might be even better.

>  5. HCodecs is probably the best library for reading WAVE files.  It
>     uses 'DiffUArray Int a' to represent audio data, and not '[[a]]'.
>     It also supports MIDI and SoundFont, which is quite handy.
>  6. YampaSynth looks really cool, and it might be possible to turn it
>     into a reusable MIDI synthesizer library for use with HCodecs.
>     This would be extremely useful: it would provide a basis for all
>     kinds of crazy MIDI-based programming in Haskell.

If you are after realtime synthesis you might also want to look into 
SuperCollider. I posted some code to control SuperCollider via MIDI to 
haskell-art. I can send it to you, if you like.

> What would the ideal Haskell sound API look like?  Personally, I would
> love to see:
>
>  a. Something like HCodecs's Data.Audio as the basic sound format.
>     Data.Audio is an array-based API, and it supports converting
>     between a wide range of common sample formats.  I don't know how
>     fast this is, but it already exists, and it's used by YampaSynth.
>  b. OpenAL for sound playback, with a portable version of ALUT.  This
>     may require writing a pure-Haskell version of ALUT.

So far I used Sox' play command fed by a pipe with runInteractiveCommand.

> Other nice-to-have features might include:
>
>  e. A standard MIDI format, based on either the HCodecs package or the
>     midi package.  (HCodecs is used by YampaSynth, and the midi package
>     is used by alsa-midi.)

midi package is mainly used by Haskore

>  f. A modular version of YampaSynth which can convert MIDI data
>     structures into Data.Audio values.
>
> It looks like Haskell could be a really sweet audio programming
> environment with just a bit of integration work.  What do folks think?
> Are there other libraries I should look at more closely?  Other features
> that should be included in an ideal audio API?

What do you mean by 'API'? I think there should not be one library 
handling all kind of music tasks. For interoperability common data 
structures are nice. However, what signal representation concerns, 
I think we need more than one representation, because different 
applications have different needs.


> Haskell audio libraries

I have advertised my libraries for years in Haskell-Wiki and HCAR, but it 
seems that people do not look there. :-(



> Reading and writing sound files:
>  HCodecs: (Audible a) => DiffUArray Int a
>  hsndfile: MArray with Float and Double
>  HSoundFile: [[Double]]
>  ALUT: OpenAL.AL.Buffer
>  WAVE: [[Int32]]
>
> Playing sounds:
>  OpenAL: Ptr UInt8, Ptr Int16, 1-N channels
>  SDL-mixer: ForeignPtr ChunkStruct, ForeignPtr MusicStruct

Sox via pipe, using any data structure which can be written to a file


> Sound processing libraries:
>  dsp: Array a, [a]
>  jack: Operates via mapping functions
>    mainMono :: (CFloat -> IO CFloat) -> IO ()
>    mainStereo :: ((CFloat, CFloat) -> IO (CFloat, CFloat)) -> IO ()

synthesizer: StorableVector, State, Generic (older: list, fusable list)


> MIDI-based:
>  HCodecs: Reads and writes MIDI files
>  midi: Reads and writes MIDI files
>  alsa-midi: Uses midi library
>  YampaSynth: Stand-alone program

Haskore uses MIDI and other back-ends like Haskell Synthesizer, CSound, 
SuperCollider


> Special-purpose APIs only (FFIs, etc.):
>  hCsound: CsoundPtr
>  hsc3: UGen

HasSound


> No public sound-buffer API:
>  hbeat: The relevant source files are missing!
>  hogg: Very low-level API for Ogg internals only
>  libmpd: No sound buffer API
>  sonic-visualizer: No sound buffer API
>  truelevel: Stand-alone program (uses WAVE)
>  wavconvert: Stand-alone program