View source for Sound programming, sound coding, sound codecs

{{avnotes}}

__TOC__

==conversion==

* sox (command line tool)

* libsndfile [http://www.mega-nerd.com/libsndfile/]

* libaudiofile [http://www.68k.org/~michael/audiofile/]


Sample rate conersion:
* libsamplerate [http://www.mega-nerd.com/SRC/] (a.k.a. Secret Rabbit Code)

<!--
* libinterp 
-->


==General/wider purpose audio programming==
* Csound
** http://en.wikipedia.org/wiki/Csound
** http://www.csounds.com/
<!--
** C, some C++
** modular
** Bindings for Python, Java, LISP, Tcl, C++, and others
-->

* Nyquist
** http://en.wikipedia.org/wiki/Nyquist_(programming_language)
** http://www.cs.cmu.edu/afs/cs.cmu.edu/project/music/web/music.software.html

* Pure Data
** http://en.wikipedia.org/wiki/Pure_Data
** http://puredata.info/
<!--
** C, some C++
** can be used from Python, Ruby, Scheme, and others
-->

* Supercollider
** http://www.audiosynth.com/ 
** http://en.wikipedia.org/wiki/SuperCollider

* C++ Library for Audio and Music (CLAM)
** http://www.clam.iua.upf.edu/
** http://iua-share.upf.es/wikis/clam/index.php/Main_Page

* ChucK
** http://en.wikipedia.org/wiki/ChucK

* CREATE Signal Library
** http://fastlabinc.com/CSL/

* Reaktor
** http://en.wikipedia.org/wiki/Reaktor

* Max, MSP(, Jitter)
** http://en.wikipedia.org/wiki/Max/MSP
** http://dactilde.com/index.php?title=Main_Page
** See also [http://en.wikipedia.org/wiki/PeRColate PeRColate]
* Synz
** http://www.uweb.ucsb.edu/~ljputnam/synz.html 

* Open Sound World
** http://osw.sourceforge.net/ 

* SndObj
** http://music.nuim.ie//musictec/SndObj/main.html
** http://en.wikipedia.org/wiki/Sound_Object_(SndObj)_Library 

* Create Signal Library (CSL)
** http://fastlabinc.com/CSL/

* Synthesis Toolkit (STK)
** http://en.wikipedia.org/wiki/Synthesis_Toolkit
** http://ccrma.stanford.edu/software/stk/

<!--
* cmix
** http://en.wikipedia.org/wiki/CMix
** http://music.columbia.edu/cmc/RTcmix/

*Bidule 
** http://en.wikipedia.org/wiki/Bidule


See also:
* http://en.wikipedia.org/wiki/Category:Audio_programming_languages
* http://en.wikipedia.org/wiki/Comparison_of_audio_synthesis_environments

* http://wiki.python.org/moin/PythonInMusic

-->

==Helpers, codecs simple and complex==
{{stub}}

See e.g.:
* http://en.wikipedia.org/wiki/List_of_codecs#Audio_codecs
* http://en.wikipedia.org/wiki/Comparison_of_audio_codecs


The following table is meant as a smaller table to give an overview of the appoaches, algoritms, and codecs that are out there, from all fields of application, what they basically do <!--(and why you might want to (not) choose it.)-->

Note:
* 'Speech' usually  
:: focuses on space-efficient coding 
:: focuses on low delay in encoding and decoding
:: assumes relatively isolated speech

* 'Music' usually means focus on 
:: close (perceptual) reproduction of the original
<!--
Space taken, and difficulty of decoding, is secondary to most.

MP3
Vorbis and Opus
AAC
FLAC,

Some music stores may make other formats likely to see, e.g. 
WMA
ALAC
-->



{| border="1" cellpadding="5" cellspacing="0" 
|- 
! '''Name'''
! style="width:1%" |'''Used for'''
! '''Techniques, bitrates'''<br/><br/>'''Notes'''
! style="width:1%" | '''Links'''

|- 
| AAC usually refers to AAC-LC (defined in MPEG-2, MPEG-4)
| Music
| Also known as MPEG-2 NBC{{verify}}.<br/>Can be seen as improvement over MP3<br/>See also MPEG-4 HE-AAC.
| [http://en.wikipedia.org/wiki/Advanced_Audio_Coding]

|- 
| AC-3 / Dolby Digital
| 
| Known as AC-3 (often seen without the dash), Dolby Digital (sometimes DD), ATSC A/52<br/>Its early introduction (1986, before MP3) and 5.1 support made it a common choice for that to store 5.1. It being supported (alongside [[DTS]]) in digital interconnects like [[TOSLINK]] helped too<br/>
Quality/limitations is comparable to MP2, MP3 {{verify}}, including that it has some trouble with impulses, high frequency stereo{{verify}}<br/>Best above 192kbps{{verify}}<br/>Cf. its successor, [[Dolby Digital Plus]]<br/>See also [[Electronics_project_notes/Audio_notes_-_multichannel_and_surround#AC-3]]
| [http://en.wikipedia.org/wiki/Dolby_Digital]

|- 
| AC-4
| 
| e.g. one of the various codecs allowed in [https://en.wikipedia.org/wiki/ATSC_3.0 ATSC 3.0 broadcast] 
| [https://en.wikipedia.org/wiki/Dolby_AC-4]

|- 
| ADX
| 
| Has an ADPCM variant and MPEG-2 variant
| [http://en.wikipedia.org/wiki/ADX_%28file_format%29]

|- 
| AIFF-C, AIFC 
| 
| Compressed variant[https://en.wikipedia.org/wiki/Audio_Interchange_File_Format#Common_compression_types] of AIFF (or, on macOS, possibly uncompressed but [[little-endian]] instead)
| 

|- 
| AIFF
|
| [[IFF]]-based format used for audio
| 

|-
| AMBE
| Speech
| Proprietary
| [http://en.wikipedia.org/wiki/Advanced_Multi-Band_Excitation]

|- 
| AMR, AMR-NB
| 
| Patented.
| 

|- 
| AMR-WB 
| Speech
| Patented. See also G.722.2
| [http://en.wikipedia.org/wiki/AMR-WB]

|- 
| AMR-WB+
| 
| Patented.
| 

|- 
| Apple Lossless (ALAC)
| Lossless
| Proprietary. Made for iOS and iTunes, which also chose not to support FLAC. tl;dr: Apple people prefer it, purely for ease.
| [http://en.wikipedia.org/wiki/Apple_Lossless]

|- 
| apt-X
| 
| Lossless variant exists.
| [http://en.wikipedia.org/wiki/Apt-X]

|- 
| ATRAC
| 
| Family of codecs
| [http://en.wikipedia.org/wiki/Adaptive_Transform_Acoustic_Coding]

|- 
| CELP and variants
| speech
| Can be a codec in itself, but now usually understood as a group of variants (ACELP, RCELP, LD-CELP, VSELP, others), or used as a part of some codec (QCELP, many), though the lines between these can be blurry.

|-
| CELT
| general, low latency
| royalty-free, open standard. Basically merged into Opus now
| [http://en.wikipedia.org/wiki/CELT]

|-
| CVSD, CVSDM
| Speech
| 
| [http://en.wikipedia.org/wiki/Continuously_variable_slope_delta_modulation]


|-
| Dolby Digital Plus 
| 
| a.k.a. DD+, DDP. E-AC-3 (Enhanced AC-3), EC-3
| [https://en.wikipedia.org/wiki/Dolby_Digital_Plus]


|-
| DSS (Digital Speech Standard)
| 
| 
| [http://en.wikipedia.org/wiki/Digital_Speech_Standard]

|-
| DTS
| 
| Patented.<br/>Seen on DVDs, comparable to AC3{{verify}}, also in that it needs a comparatively high bitrate to sound decent.{{verify}}
| [http://en.wikipedia.org/wiki/DTS_%28sound_system%29]

|-
| DTS-HD (DTS++)
| 
| Extension of DTS
| [http://en.wikipedia.org/wiki/DTS-HD_Master_Audio]

|- 
| DRA
| 
| 
| [http://en.wikipedia.org/wiki/Dynamic_Resolution_Adaptation]

|- 
| Dolby TrueHD
| Lossless
| Uses (and expands on) Meridian Lossless Packing
| [http://en.wikipedia.org/wiki/Dolby_TrueHD]


|- 
| EVRC
| Speech
| Used in CMDA2000
| [http://en.wikipedia.org/wiki/EVRC]

|- 
| EVRC-B
| Speech
| Used in CMDA2000
| [http://en.wikipedia.org/wiki/EVRC-B]

|- 
| FLAC
| Lossless
|
| [http://en.wikipedia.org/wiki/FLAC]

|-
| GSM-EFR, GSM 06.60
| Speech
| 
| [http://en.wikipedia.org/wiki/Enhanced_full_rate]

|-
| GSM-FR, GSM 06.10
| Speech
| 
| [http://en.wikipedia.org/wiki/GSM-FR]

|-
| GSM-HR, GSM 06.20
| 
| 
| [http://en.wikipedia.org/wiki/Half_Rate]

|-
| HILN
| Speech
| 
| [http://en.wikipedia.org/wiki/Harmonic_and_Individual_Lines_and_Noise]

|-
| iLBC (Internet Low Bit Rate Codec)
| 
| 
| [http://en.wikipedia.org/wiki/ILBC]

|-
| Impala
| 
| Used in FORscene
| [http://en.wikipedia.org/wiki/FORscene#Impala]

|-
| iSAC (Internet Speech Audio Codec)
| Speech
| Proprietary
| [http://en.wikipedia.org/wiki/Internet_Speech_Audio_Codec]

|- 
| ITU-T G.721 (Superceded by G.726) 
| Speech
| ADPCM at 32 kbit/s.
| 

|- 
| ITU-T G.723 (Superceded by G.726)
| Speech
| ADPCM at 24 and 40 kbit/s
| [http://en.wikipedia.org/wiki/G.723]

|- 
| ITU-T G.723.1
| Speech
| 
| [http://en.wikipedia.org/wiki/G.723.1]

|- 
| ITU-T G.726
| Speech
| ADPCM at 16, 24, 32, and 40 kbit/s (meant to supersede G.721, and G.723)
| [http://en.wikipedia.org/wiki/G.726]

|- 
| ITU-T G.711
| Speech
| A-Law/mu-Law PCM at 64kbit/s
| [http://en.wikipedia.org/wiki/G.711]

|- 
| ITU-T G.718
| Speech
| 32 to 128 kbit/s for speech and decent-quality music
| [http://en.wikipedia.org/wiki/G.718]

|- 
| ITU-T G.719
| Speech
| 
| [http://en.wikipedia.org/wiki/G.719]

|- 
| ITU-T G.722 
| Speech
| SB-ADPCM at 48, 56 and 64 kbit/s
| [http://en.wikipedia.org/wiki/G.722]

|- 
| ITU-T G.722.1
| Speech
| (24 and 32 kbit/s)
| [http://en.wikipedia.org/wiki/G.722.1]

|- 
| ITU-T G.722.2
| Speech
| Often refers to AMR-WB (Adaptive Multi-Rate Wideband)
| [http://en.wikipedia.org/wiki/G.722.2]

|- 
| ITU-T G.728
| Speech
| 16kbit/s, LD-CELP
| [http://en.wikipedia.org/wiki/G.728]


|- 
| ITU-T G.729
| Speech
| CS-ACELP
| [http://en.wikipedia.org/wiki/G.729]

|- 
| ITU-T G.729.1
| Speech
| 
| [http://en.wikipedia.org/wiki/G.729.1]

|- 
| Monkey's Audio (APE)
| Lossless
| 
| [http://en.wikipedia.org/wiki/Monkey%27s_Audio]

|- 
| MPEG-1(/MPEG-2) Layer I, Layer II, Layer III (MP3)
| Music
| You usually want Layer III<br/>(MPEG-2 extends options somewhat)<br/>Sometimes has trouble with impulses, stereo high frequency content (regardless of bitrate){{verify}}
| [http://en.wikipedia.org/wiki/MPEG-1_Audio_Layer_I] [http://en.wikipedia.org/wiki/MPEG-1_Audio_Layer_II] [http://en.wikipedia.org/wiki/MPEG-1_Audio_Layer_III]


|- 
| MPEG-1(/MPEG-2) Layer II
| Music
| (MPEG-2 extends options somewhat)
| 

|- 
| MPEG-1(/MPEG-2) Layer III (MP3)
| Music
| (MPEG-2 extends options somewhat)
| 

|- 
| MPEG-4 ALS
| Lossless
| 
| [http://en.wikipedia.org/wiki/Audio_Lossless_Coding]

|- 
| MPEG-4 DST
| 
| 
| [http://en.wikipedia.org/wiki/Direct_Stream_Transfer#DST]

|- 
| MPEG-4 HVXC
| Speech
| 
| [http://en.wikipedia.org/wiki/HVXC]

|- 
| MPEG-4 HE-AAC
| 
| 
| [http://en.wikipedia.org/wiki/High-Efficiency_Advanced_Audio_Coding]

|- 
| MPEG-4 SLS
| 
| 
| [http://en.wikipedia.org/wiki/MPEG-4_SLS]

|- 
| MPEG-4 Structured Audio
| 
| 
| 

|- 
| NIST
| 
| (niche?)
| [https://www.fon.hum.uva.nl/praat/manual/Sound_files_2_5__NIST_files.html]

|- 
| OSQ
| Lossless
| (proprietary, so generally Steinberg-only)
| [http://en.wikipedia.org/wiki/Original_Sound_Quality]

|- 
| OptimFROG
| Lossless
| (proprietary lossless format)
| [http://en.wikipedia.org/wiki/OptimFROG]

|- 
| MusePack (MPC)
| Music
|
| [http://en.wikipedia.org/wiki/Musepack]

|- 
| NeXT/Sun (.au)
| 
| Originally headerless (8000 Hz 8-bit μ-law PCM data), [https://en.wikipedia.org/wiki/Au_file_format#New_format later] with header, varied bith depth, and various lossy compression options
| [https://en.wikipedia.org/wiki/Au_file_format]

|- 
| SVOPC
| Speech
| Designed to deal with packet loss. Previously used by Skype (where it has been replaced with SILK)
| [http://en.wikipedia.org/wiki/SVOPC]

|- 
| SILK
| Speech
| Used e.g. by skype. Patented, royalty free uses possible. 
| [http://en.wikipedia.org/wiki/SILK]

|- 
| VMR-WB 
| Speech
| 
| [http://en.wikipedia.org/wiki/VMR-WB]


|- 
| TTA (True Audio)
| Lossless
| 
| [http://en.wikipedia.org/wiki/TTA_%28codec%29]

|- 
| Truespeech
| Speech
| Proprietary
| [http://en.wikipedia.org/wiki/Truespeech]


|- 
| QDesign/Ravesound
| 
| Previously known as LBPack. Used by older Quicktime variants
| [http://en.wikipedia.org/wiki/QDesign]


|- 
| WAVPack
| Lossless
| 
| [http://en.wikipedia.org/wiki/WavPack]

|- 
| LPAC, LTAC
| Lossless
| Lossless Predictive/Transform Audio Compression. Has mostly become MPEG-4 Audio Lossless Coding instead.
| [http://en.wikipedia.org/wiki/LPAC] [http://en.wikipedia.org/wiki/LTAC]

|- 
| Meridian Lossless Packing (MLP)
| Lossless
| Used in DVD-Audio, and in HD DVD, Blu-Ray through Dolby TrueHD
| [http://en.wikipedia.org/wiki/Meridian_Lossless_Packing]

|- 
| Opus
| Music/general
| patent-free open standard. Successor to Vorbis, more applicable to lower latency{{verify}}
| [https://en.wikipedia.org/wiki/Opus_(audio_format)] [https://opus-codec.org/]


|- 
| RTAudio
| Speech
| Proprietary
| [http://en.wikipedia.org/wiki/RTAudio]


|- 
| RealAudio
| Speech, music
| 
| [http://en.wikipedia.org/wiki/RealAudio]

|-
| SHN (Shorten)
| Lossless
| 
| [http://en.wikipedia.org/wiki/Shorten]


|- 
| Siren 7, Siren 14, Siren 22
| Speech
| Patented (royalty free use possible). See also G.722.1 
| [http://en.wikipedia.org/wiki/Siren_Codec]

|- 
| Speex
| speech
| Beats various older speech codecs at low-bitrate speech
| [http://en.wikipedia.org/wiki/Speex] [http://www.speex.org/] 

|- 
| TwinVQ
| 
| Proprietary (Yamaha, NTT) 
| [http://en.wikipedia.org/wiki/TwinVQ]


|-
| SMV (Selectable Mode Vocoder)
| Speech
| Used in CMDA2000
| [http://en.wikipedia.org/wiki/SMV]

|- 
| Vorbis 
| Music/general
| patent-free open standard
| [http://en.wikipedia.org/wiki/Vorbis] [http://www.vorbis.com/]

|-
| VOX (Dialogic ADPCM)
| 
| 4-bit ADPCM, often 8000Hz sampling, less commonly 6000Hz
| [http://en.wikipedia.org/wiki/Dialogic_ADPCM]


|- 
| WMA
| Music
| Proprietary format.<br/>There are variants on basic WMA targeting higher quality audio (WMA Professional), voice coding (WMA Voice), lossless coding (WMA Lossless). Hardware players tend to not support these.<br>Optional DRM.<br/>Music quality comparable to Ogg, AAC{{verify}}
| [http://en.wikipedia.org/wiki/Windows_Media_Audio]



|}


Lossless codecs tend to compress to perhaps half of what the 44kHz Stereo PCM samples would take.
There is variation, but not very much.

<!--
MP3 is technically an MPEG-2 Transport Stream (why it works out as just a bunch of frames without even a header), containing ADTS.


-->




<!--
Flash uses 
* MP3
* Nellymoser (a.k.a. Asao) [http://en.wikipedia.org/wiki/Nellymoser_Asao_Codec_in_Flash], since Flash 6 
* Speex, since Flash 10
* G.711 since Flash 11{{verify}}
-->


libraries:
* libsamplerate (and wrappers, like pysamplerate)


file format reading:
* libaudiofile (AIFF, AIFC, WAVE, and NeXT/Sun) [http://www.68k.org/~michael/audiofile/]
* various mpeg libraries primarily for MPEG1 (which includes MP3), sometimes MPEG2
* 

And see also:
* http://www.cnpbagwell.com/audio.html


Perceptive quality measures:
* MOS - Mean Opinion Score [http://en.wikipedia.org/wiki/Mean_Opinion_Score]
* PEAQ [http://en.wikipedia.org/wiki/PEAQ]
* PSQM (ITU-T P.861) [http://en.wikipedia.org/wiki/PSQM]
* PESQ (ITU-T P.862) [http://en.wikipedia.org/wiki/PESQ]





<!--
==On audiophile choices==

Lossless formats like (raw like WAV or AIFF, or compressed lossless like FLAC, ALAC, WavPack, Monkey's Audio, etc.) can avoid some imperfections, but the improvement is limited due to the way CDs are mastered. 


The game of representing what you have as best as possible only matters when you can criticize the mix.

Most of the reasons to go for lossless, such as more accurate representation of dynamic range and phase information throughout all frequencies, are also the most likely to have been mangled during mixing - primarily because modern pop, rock, rap, and hip-hop are made to sound loud on radio (see [[loudness war]]), which means the dynamic range you are trying to save was already gone.


This is actually one of the valid reasons to choose LPs.

Not because they are a better medium - digital representation is measurably better in theory and practice, but only as a medium, and medium is structurally less relevant for music that is remotely popular. 

The reasons I'm getting to can be divided into
* a CD will store whatever you give it, an LP will not
: an LP is more likely to skip on certain types of loudness

* it's a different market
: assuming the loudness war is still a thing, it's only useful to do this on CD and other digital masters, because DJs will use these, not LPs.
:: meaning there is incentive to do loudness-on-the-radio mastering on all of these (except, it seems, for classical music) simply because digital is the most common medium.
: if they're lazy, they'll use the same mix for CD and LP. With a little luck, the LP mix is different (in part because of the first point)
: In every case it all depends on how it was recorded, and how the sound engineer was influenced (few of them ''want'' the compression, but have managers who give them no choice).


-->

==File formats==

===MP3===
<!--
====File structure====

MP3 files mostly consists of MP3 frames, and in practice often have ID3v1 metadata (at the end) or ID3v2 metadata (at the start).


MP3 decoders do not really need to know about ID3, because they synchronize to MP3 frames. ID3v2 was specifically designed to never cause synchronization, while still being able to represent binary data.


MP3 frames are byte-aligned, though contain bitstreams.

MP3 frames start with a 4-byte header. 
The structure and size of further frame contents (optional CRC, size of audio data) are implied by header values.


There are plenty of libraries that deal well with decoding MP3, and interacting with ID3 tags.
Audio players and such are probably best off using them, because they'll often deal with more cases 
common deviations (e.g. Xing), common data errors, than you will anytime soon.


====Quality====

MP3 data basically represents frequency content, rather than a waveform.
It's a [https://en.wikipedia.org/wiki/Transform_coding transform coder], working in the frequency domain.

Encoders have some choice in the psychoacoustic model they use, which varies how usefully you can use a frame. 
It was long known that the first encoders weren't so good, while others were fundamentally better. 
LAME has been at or near the top for a long while.

There is a lot to be said about how to evaluate psychoacoustic models, and how to do so from the perspective of the human ear. Throwing a bit of error-minimization math at it isn't enough. 
Still, in theory this discussion is finite, in that we could find near-optimal choices (both for the model and the concrete bits of MP3 that estimate such a model) we can all agree with.



Since it cares more about frequency amplitude than phase, 
and the way the spec splits frequencies means it loses time information.
Which leads to pre/post echos - but how bad those are depends on the actual content.
It will typically deal well enough with natural transients - but that does mean it's worse for some electronic music.

Throwing 320kbps at it won't fix this.

Coders that consider it can alleviate this, but but there are limits.



The largest question is whether MP3 is good enough for all the things you want it for,
or whether even the cleverest encoder will be limited by the standard, and there is another that can do better.

There are a few short answers to that:
MP3 is good enough for most things,
and that other things don't have the same limitations.


See also:
* http://www.mp3-tech.org/content/?mp3%20limitations
-->





===WAV and friends===

The WAVE format (often WAV due to the file extension) is a specific use of [[RIFF]]

WAV is usually raw, uncompressed [[linear PCM]] - but it can also store ADPCM, uLaw {{verify}}{{verify}}.


As a fairly simple file format carrying raw data, it is useful as an interchange format.

<!--
WAV is a RIFF file with a {{inlinecode|WAVE}} chunk that contains a header, before the {{inlinecode|data}} chunk.
: ...though note that other extensions have been introduced since, so it's best to actually read the header (than, say, assume it's 44100 16-bit stereo starting at byte 44)


That header is 
* 32 bits: {{inlinecode|WAVE}}
* 32 bits: {{inlinecode|fmt }}
* 32 bits: size of this header chunk's data (usually 16)
* 16 bits: type of data, usually 0x01 for PCM
* 16 bits: channels (usually 1 or 2)
:: {{comment|(but the field is an 16-bit int so technically allows up to 65536{{verify}}, and you do e.g. see 4-track recording into WAV - but it's avoided where possible because not everything can read that)}}
* 32 bits: sampling frequency
:: technically allows up to 4.2 GHz - but you typically see one of the dozen or so typical sampling rates. WAV has been seen used to store higher rate data, but not a lot because it also has an implied 4GByte file size limit
* 32 bits: data rate in bytes per second (this seems redundant?)
* 16 bits: block alignment
* 16 bits: sample depth (bits, usually 16, sometimes 24{{verify}}{)

file readers will often also consume the data chunk's header, to position the file at the start of the PCM data.
 

https://blog.argcv.com/articles/6850.c

-->

====W64====
<!--
W64 (Wave 64) is basically a slight tweak of the WAV header to break the 4GB limit.

WAV is very easy to support, but W64 is probably only supported by audio tools
just because a lot of don't need wave files that are many hours long.

-->

====BWF====

BWF is an extension of WAV that is used e.g. by (non-linear) digital recorders.

BWF files still use the .WAV extension, and is backwards compatible because it mostly just adding metadata in extension chunks.

It adds things like timecodes



ITU recommendation ITU-R BS.1352-3, Annex 1.

https://en.wikipedia.org/wiki/Broadcast_Wave_Format


====RF64====

Like BWF, but breaks WAV/BWF's 4GB limitation, and allows more channels{{verify}}.

Used in broadcasting, and audio archiving.

https://en.wikipedia.org/wiki/RF64







====Older formats====

=====VOC file=====
<!--

VOC was Creative Labs format, used for a while, in games and outside it,
but was displaces by the already somewhat established WAV

Could store a few different formats, e.g. 8-bit and 16-bit PCM, some ADPCM, a-law, u-law, though it seems the first two were the most used.

https://moddingwiki.shikadi.net/wiki/VOC_Format
-->

[[Category:Audio, video, images]]