Electronic music - notes on audio APIs: Difference between revisions

From Helpful
Jump to navigation Jump to search
mNo edit summary
 
(13 intermediate revisions by the same user not shown)
Line 1: Line 1:
{{#addbodyclass:tag_tech}}
{{#addbodyclass:tag_media}}
{{avnotes}}
{{avnotes}}


Line 483: Line 486:


===Windows APIs===
===Windows APIs===
====APIs in terms of latency====
<!--
<!--
tl;dr:
 
* 1 to 3ms of overall latency comes from hardware necessity (coded into the driver)
Latency tl;dr:
: and after you bought something, you have little to no control over that
* what you can get
:: not thinking about this gives you 30-100ms of overall latency
:: with some specific tweaking you can expect 3-10ms out of a lot of modern hardware
:: 1-3ms, without issues, is relatively rare
 
 
* 1 to 3ms of overall latency comes from hardware necessity (a smaller part of which may ''also'' be hardcoded choices in the driver)
: and after you bought hardware, you have ''no'' control over what it ''needs'' due to its design, and ''little'' over the driver
:: hardware is what you bought
:: hardware is what you bought
:: driver comes with your hardware
:: driver comes with your hardware
Line 495: Line 507:
:: some parts of that are configurable
:: some parts of that are configurable
:: though unless driver and app specifically expose these choices as settings, they are still effectively fixed
:: though unless driver and app specifically expose these choices as settings, they are still effectively fixed


* newer sound subsystems have put more thought into design regarding lower latency, so allow lower latencies with less trouble
* newer sound subsystems have put more thought into design regarding lower latency, so allow lower latencies with less trouble
Line 508: Line 519:
:: originally it was by far the easiest way to get low latency
:: originally it was by far the easiest way to get low latency
:: though these days the difference may be smaller, sometimes almost nonexistent - but ASIO is still more of a known quantity
:: though these days the difference may be smaller, sometimes almost nonexistent - but ASIO is still more of a known quantity


* if the sound API has a mixer, you can often get lower latency  
* if the sound API has a mixer, you can often get lower latency  
Line 514: Line 526:
: how much better varies (e.g. the newer WASAPI can be pretty decent even in shared mode)
: how much better varies (e.g. the newer WASAPI can be pretty decent even in shared mode)


* the next bit of gain is in lowering (the program's own) buffers
* the next bit of latency improvement is in lowering any buffers, particularly the client application's
: at some point that'll start breaking up due to programs not always having samples soon enough.
: at some point that'll start breaking up due to programs not always having samples soon enough.
:: this usually happens somewhere in the 2-5ms range.
:: this usually happens somewhere in the 2-5ms range - you should assume apps increase to ~6ms even if the hardware can do 3ms
 
* what you can get
:: not thinking about this gives you 30-100ms of overall latency
:: with some specific tweaking you can expect 3-10ms out of a lot of modern hardware
:: 1-3ms, without issues, is relatively rare
 
 


-->
====APIs that exist====
<!--
* '''Drivers and APIs are a source of a much of confusion, and more misinformation.'''
: Which is quite understandable, because
:: in Windows there is no clear-cut distinction between driver API and sound APIs,
:: implementation details to most relevant APIs have changed over time and the fact is not always well documented
:: some APIs will be there for compatibility, but now actually be wrapping others, so even information that was once very precise may now be wrong
:: DAWs sometimes call things by friendlier names - of which the meaning is not always clear
:: I'm still not all that clear sure where precisely which buffers are in which case


'''Drivers and APIs are a source of a much of confusion, and more misinformation.'''
* roughly: there is ASIO, Core Audio/WASAPI, what we call WDM/KS and WDM, DirectSound, MME
:: ...which is actually a blend of lower-level and higher-level APIs, that are not 100% separable
:: but you generally want an option earlier in that list




Which is quite understandable, because
: in Windows there is no clear-cut distinction between driver API and sound APIs,
: implementation details to most relevant APIs have changed over time and the fact is not always well documented
: some APIs will be there for compatibility, but now actually be wrapping others, so even information that was once very precise may now be wrong
: DAWs sometimes call things by friendlier names - of which the meaning is not always clear
: I'm still not all that clear sure where precisely which buffers are in which case




tl;dr
* ASIO
* ASIO
:: is a do-one-thing-and-do-it-well API
:: is a do-one-thing-and-do-it-well API
Line 545: Line 555:


* WDM  
* WDM  
: ''technically'' the acryonym means "the name for the package that all windows drivers for ''all'' hardware for the last two decades come in".  
: ''technically'' that acryonym means "the name for the package that all windows drivers for ''all'' hardware for the last two decades come in".  
: {{comment|(so yes, ASIO drivers are also WDM, in a 'technically' way that doesn't matter to this context)}}
: {{comment|(so yes, ASIO drivers are also WDM, in a 'technically' way that doesn't matter to this context)}}
: in this context we usually 'WDM drivers for sound cards that provide a Windows sound API' (basically anything other than ASIO)
: in this context we usually 'WDM drivers for sound cards that provide a Windows sound API' (basically anything other than ASIO)
Line 578: Line 588:
-->
-->


====Some history====
=====Some API history=====
<!--
<!--


Line 677: Line 687:




'''For context'''
{{zzz|For context|
ASIO is an API that is completely separate from windows sound APIs. 'Doing it entirely our own way because it's more controlled' was a good deal of the point. Speaking ASIO to an ASIO driver that directly controls its hardware might be called '''native ASIO'''.}}


ASIO is an API that is completely separate from windows sound APIs. 'Doing it entirely our own way because it's more controlled' was a good deal of the point
Speaking ASIO to an ASIO driver that directly controls its hardware might be called '''native ASIO'''.


{{zzz|For a little extra confusion,|drivers that speak ASIO might  
{{zzz|For a little extra confusion, drivers that speak ASIO might...|
* choose to expose ''only'' the ASIO API, because much of the point is ignoring all the and unknowns and variation that comes from windows APIs.
* choose to expose ''only'' the ASIO API, because much of the point is ignoring all the and unknowns and variation that comes from windows APIs.
* ...or, sometimes, choose to expose both ASIO and a windows API -- more flexible in theory ('you can just use it in windows'), though can also somewhat more confusing in use and ends up with a set of maybe-true, maybe-superstitions, such as
* ...or, sometimes, choose to expose both ASIO and a windows API -- more flexible in theory ('you can just use it in windows'), though can also somewhat more confusing in use and ends up with a set of maybe-true, maybe-superstitions, such as
Line 691: Line 700:
}}
}}


Given that context, you would think that "I speak ASIO" means "I provide ASIO capabilities for a specific audio device".




'''ASIO wrappers''' have a different goal from native ASIO
'''ASIO wrappers''' speak ASIO, but have a different goal.


ASIO wrappers open a regular (non-ASIO) sound card via a regular Windows sound API calls (in practice typically via WDM/KS or Core Audio's WASAPI),  
ASIO wrappers open a regular (non-ASIO) sound card via a regular Windows sound API calls (in practice typically via WDM/KS or Core Audio's WASAPI), force some specific settings (e.g. smaller buffers, and exclusive mode where possible),  
force some specific settings (e.g. smaller buffers, and exclusive mode where possible),  
then present that via the ASIO API. {{comment|(I guess you could also call them something like ASIO rerouters)}}.
then present that via the ASIO API.


Is this counter to ASIO's shortest-path-to-the-hardware principle? Yes.
Is this counter to ASIO's shortest-path-to-the-hardware principle? Yes.
Line 704: Line 714:


Is there still good reason to do it? Actually yes.
Is there still good reason to do it? Actually yes.




Line 751: Line 763:
* FlexASIO  [https://github.com/dechamps/FlexASIO]
* FlexASIO  [https://github.com/dechamps/FlexASIO]


 
* ASIO Link
: more focused on allowing {{imagesearch|asio link pro|more complex routing}}




Line 1,006: Line 1,019:


Confusingly, some Audio APIs are tied closely to the underlying driver API, and some are not.
Confusingly, some Audio APIs are tied closely to the underlying driver API, and some are not.
-->
===="Why are some things called loopback?"====
<!--
Loopback, in a wider sense, tends to mean 'wire an output to input, probably on the same device'.
Often, loopback is done to be able to record that output again.
: (...ideally without creating a bunch of feedback)
'''Windows audio devices called loopback'''
Windows-API devices with {{inlinecode (looppack)}} at the end of the name
are windows giving you a copy of whatever is being ''played'' on them.
: basically windows saying "hey I'm putting data to device, you want some too?"
: and distinguishing it from the recording the inputs that the device may (or may not) provide
{{comment|As loopback could be used to mean other things, even on exactly the the same device, a better name for that feature might have been 'playback tap'. Whatever.}}
It's a useful distinction - e.g. the Focusrite in front of me shows up as both
* Speakers    (Focusrite USB Audio) (loopback)
:: (I called it Speakers, because reasons)
* Analogue 1+2 (Focusrite USB Audio)
'''Other things called loopback'''
That Focusrite's settings also has something called loopback. What's that?
...same idea. You have the options
* analogue  - recording from it presents its analog inputs -- default, and what you would usually expect it to do
* playback  - recording from it presents what is being played to the device
* Custom mix - a mix of the above
Some effect boxes like the VT-3 also have loopback.
This one is more confusing, because it's thinking from its perspective.
Also, it's a sound card for your PC -- and the effects part and PC part are handled separately.
It's thinking about live performance, where the being-a-soundcard is primarily for the




Line 1,148: Line 1,215:
* zero latency ''does not exist'', a few milliseconds of relative offsets happens ''all over the place''
* zero latency ''does not exist'', a few milliseconds of relative offsets happens ''all over the place''


* amounts of ''added'' latency can matter, though
* amounts of ''added'' latency can matter, and can be made ''low''


* latency matters when hearing yourself live, or syncing to something live (e.g. looper pedals)
* latency matters when hearing yourself live, or syncing to something live (e.g. looper pedals)
Line 1,194: Line 1,261:


-->
-->
==API stuff==
==API stuff==
===Windows API tweaking===
===Windows API tweaking===
Line 1,341: Line 1,409:
===="Delay compensation?"====
===="Delay compensation?"====
<!--
<!--
Delay compensation (or 'latency conpensation') is not advanced magic sauce that lowers your latency.  
Delay compensation (or 'latency conpensation') is not advanced magic sauce that lowers your latency.  
It does ''nothing'' to lower interface latency,  
: Delay compensation does ''nothing'' to lower interface latency,  
and it does ''nothing'' for you live.
: Delay compensation does ''nothing'' for you live.
 
 
Delay compensation is some variation on the theme of
"specific music-producing/processing things have different amounts of implicit latency. When we know it for all, we can align it better ''when they play together''".
 
As a side effect of this feature ''reporting what it's measuring''
you may learn the latency of each, you may learn that disabling specific things, or simplifying specific processing, helps.
 
But by itself, this often means that the slowest element is sets the standard,
and everything else gets played later to align with it.




It is some variation on the theme of "knowing the latency of sound production, and and align it better",


Usually, that comes down to  
Usually, that comes down to  

Latest revision as of 23:23, 21 April 2024

The physical and human spects dealing with audio, video, and images

Vision and color perception: objectively describing color · the eyes and the brain · physics, numbers, and (non)linearity · color spaces · references, links, and unsorted stuff

Image: file formats · noise reduction · halftoning, dithering · illuminant correction · Image descriptors · Reverse image search · image feature and contour detection · OCR · Image - unsorted

Video: format notes · encoding notes · On display speed · Screen tearing and vsync


Audio physics and physiology: Sound physics and some human psychoacoustics · Descriptions used for sound and music

Noise stuff: Stray signals and noise · sound-related noise names · electronic non-coupled noise names · electronic coupled noise · ground loop · strategies to avoid coupled noise · Sampling, reproduction, and transmission distortions · (tape) noise reduction


Digital sound and processing: capture, storage, reproduction · on APIs (and latency) · programming and codecs · some glossary · Audio and signal processing - unsorted stuff

Music electronics: device voltage and impedance, audio and otherwise · amps and speakers · basic audio hacks · Simple ADCs and DACs · digital audio · multichannel and surround
On the stage side: microphones · studio and stage notes · Effects · sync


Electronic music:

Electronic music - musical terms
MIDI · Some history, ways of making noises · Gaming synth · microcontroller synth
Modular synth (eurorack, mostly):
sync · power supply · formats (physical, interconnects)
DAW: Ableton notes · MuLab notes · Mainstage notes


Unsorted: Visuals DIY · Signal analysis, modeling, processing (some audio, some more generic) · Music fingerprinting and identification

For more, see Category:Audio, video, images


Why latency exists (the long version)

Latency and physics

Latency in the real world exists because of distance and the speed of sound.

The speed of sound is 343 meters per second.

Which is roughly 3 milliseconds per meter, so purely physically:

a mic on a guitar cab has maybe 1ms to physically get sound from speaker to mic.
talking to someone closeby is easily 5ms
a small-to-moderate music practice space easily has maybe 15ms of delay from one wall to the other
opposite ends of a 15m bus would be 40ms
halfway across a sports field is easily 100ms

So distance alone is

why bands may well watch their drummers
why in larger spaces you may want to use headphones instead (but not bluetooth ones)
one of a few reasons orchestras have conductors


Some other context:

two frames in 60fps game are 17ms apart
two frames in 24fps movie are 42ms apart


In musical context


Latency in hardware

The nature of digital audio

Why larger buffers are generally useful

When smaller buffers are required

Input, output, round-trip, software? On reported and actual latency

Measuring latency

On drivers and APIs

Windows APIs

APIs in terms of latency

APIs that exist

Some API history

On ASIO wrappers

This article/section is a stub — some half-sorted notes, not necessarily checked, not necessarily correct. Feel free to ignore, or tell me about it.


💤 For context
ASIO is an API that is completely separate from windows sound APIs. 'Doing it entirely our own way because it's more controlled' was a good deal of the point. Speaking ASIO to an ASIO driver that directly controls its hardware might be called native ASIO.


💤 For a little extra confusion, drivers that speak ASIO might...
  • choose to expose only the ASIO API, because much of the point is ignoring all the and unknowns and variation that comes from windows APIs.
  • ...or, sometimes, choose to expose both ASIO and a windows API -- more flexible in theory ('you can just use it in windows'), though can also somewhat more confusing in use and ends up with a set of maybe-true, maybe-superstitions, such as
"it's better only when windows or some other software isn't also trying to use it"
or "don't set it as the windows out or it will refuse to open as ASIO"
or "don't set it as the windows out or windows out will stop when you open it as ASIO"
or something else.


Given that context, you would think that "I speak ASIO" means "I provide ASIO capabilities for a specific audio device".


ASIO wrappers speak ASIO, but have a different goal.

ASIO wrappers open a regular (non-ASIO) sound card via a regular Windows sound API calls (in practice typically via WDM/KS or Core Audio's WASAPI), force some specific settings (e.g. smaller buffers, and exclusive mode where possible), then present that via the ASIO API. (I guess you could also call them something like ASIO rerouters).

Is this counter to ASIO's shortest-path-to-the-hardware principle? Yes.

Will it get you latencies better than that the underlying API could give you anyway? No.

Is there still good reason to do it? Actually yes.



"If you could get these latencies from the underlying API anyway, why add a layer?"

Convenience, mostly.

  • it is easier for you to figure out the precise settings (small-buffer, possibly-exclusive) once,
in one place - the wrapper's settings (doesn't change with how you use it), rather than for every DAW-soundcard combination you have
that maybe you can save and restore later
there is a clean separation between "speaks ASIO" and "does various latency-lowering tricks"
in a way that anything that speaks ASIO can benefit from equally
  • using that wrapper might make explanations a lot easier.
Particularly towards people who just care for 'make it work decently' than reading up on decades of idiosyncratic programing history. (Per specific DAW where applicable)
  • In fact some DAWs speak mainly or only ASIO, because their approach is to tell you to either get ASIO hardware, or if not, figure out low latency in something external, and talk to that.
case in point: FL Studio, Cubase, MAGIX supply ASIO wrappers


There's a few more useful reasons hiding in the details, like

  • you can often force WASAPI cards down to maybe 5-10ms output latency without exclusive mode
...which means you don't have to dedicate a sound card to a DAW that only talks ASIO.
Which is good enough e.g. for when playing some piano on a laptop on the go, so pretty convenient
  • some ASIO wrappers can talk to on different sound cards for input and output, at the cost of slightly higher latency (will probably glitch at the lowest latencies), which DAWs talking native ASIO will typically refuse to do (for latency and glitch reasons).


As far as I can tell

  • ASIO4ALLv2 is a WDM/KS wrapper
needs to force exclusive mode
can talk to different sound cards for input and output
  • FL Studio ASIO (a.k.a. FLASIO) is a WASAPI wrapper
Comes with FL studio (including its demo), and is perfectly usable in other DAWs
can talk to different sound cards for input and output
  • "Generic Low Latency ASIO Driver" is similar to ASIO4ALL but with different options
Comes with Cubase
  • ASIO2WASAPI [2]
  • ASIO Link
more focused on allowing more complex routing


Some of them are multiclient - allow multiple ASIO clients to connect to the same ASIO device (at the cost of a little more latency than if you stay exclusive)

A few cases are only that, e.g. Steinberg's multiclient driver[4] seems nothing beyond ASIO in ASIO.



There is further software that happens to present its result as ASIO, sometimes more for options than for latency reasons

  • VOICEMEETER also presents its result as ASIO


On custom ASIO drivers

In theory, you can write an ASIO driver for any hardware.

It won't do you too much good if the hardware isn't designed for low latency, It won't be one installed automatically, and these days there may be less reason to do so (in that you can get similar latency with Core Audio/WASAPI).

...but e.g. the Kx project, that made a specific series of Sound Blaster cards more capable, also included ASIO support, and would let the cards do on the order of 5ms.

...and I have a handheld recorder that, while fairly cheap, happens to also act as a good low-latency ASIO sound card - when the drivers are installed, anyway.


Some audio interfaces can speak both a regular API and ASIO. It's more accomodating, but also potentially more confusing, because you cannot talk ASIO while the regular. And if windows decides it's first in the list of cards, that means you need to poke settings before it does what you think it should.

"So which API is best?"

"Why are some things called loopback?"

Linux APIs

Kernel level

Higher level

OSX APIs

This article/section is a stub — some half-sorted notes, not necessarily checked, not necessarily correct. Feel free to ignore, or tell me about it.

Because OSX was a relatively clean slate at the time, at lower levels there is primarily Core Audio (also present in iOS, slimmed down(verify)), which has been quite capable for a long while.


https://developer.apple.com/library/archive/documentation/MusicAudio/Conceptual/CoreAudioOverview/Introduction/Introduction.html

https://developer.apple.com/audio/

Lowering latency

In general

tl;dr

  • zero latency does not exist, a few milliseconds of relative offsets happens all over the place
  • amounts of added latency can matter, and can be made low
  • latency matters when hearing yourself live, or syncing to something live (e.g. looper pedals)
  • digital input, output, and/or processing have some latency
In ways that are (looking at forums) usually partly misunderstood


Decide how low is necessary

The basic steps

API stuff

Windows API tweaking

Use a sound API designed for lower latency

Use sound API settings that lower the latency: exclusive mode

Use sound API settings that lower the latency: smaller buffer sizes

Linux API tweaking

OSX API tweaking

DAW considerations

Considering effects

Higher sample rates?

"Delay compensation?"

End-to-end latency

Further considerations

Hardware bus?

On network latency

Unsorted