Visuals DIY, video synths

The physical and human spects dealing with audio, video, and images Image: file formats · image noise reduction · blur detection · halftoning, dithering · illuminant correction · Image descriptors · Reverse image search · image feature and contour detection · OCR · Image - unsorted Displays: Video display notes · before framebuffers · Simpler display types · Display DIY ·· Screen tearing and vsync · Arguments for 60fps / 60Hz in gaming‎‎ Video: file format notes · video encoding notes · Subtitle format notes Vision and color perception: physics, numbers, and (non)linearity · objectively describing color · the eyes and the brain · color spaces · references, links, and unsorted stuff · Lux and lumen Audio physics and perception: Sound physics and some human psychoacoustics · Descriptions used for sound and music · multichannel and surround · Sound level meter notes Noise related: Stray signals and noise · sound-related noise names · electronic non-coupled noise names · electronic coupled noise · ground loop · (tape) noise reduction · strategies to avoid coupled noise · emphasis · Sampling, reproduction, and transmission distortions Digital sound and processing: capture, storage, reproduction · on APIs (and latency) · programming and codecs · some glossary · audio noise reduction · Audio and signal processing - unsorted stuff Music electronics: device voltage and impedance, audio and otherwise · amps and speakers · basic audio hacks · Simple ADCs and DACs · digital audio On the stage side: microphones · what can I plug into what? · audio levels & technical gritty · devices you'll use · cables, connectors, adapters · effect hardware · sync Electronic music and DIY: approaches to making sounds · some history · interesting instruments · Gaming synth · Synth: MIDI · sync · Electrical components, small building blocks · VCO, LFO, DCO, DDS notes · microcontroller synth · audio platforms modular synth: formats (physical, interconnects) · physical · Learning from existing devices · eurorack power supply ·· Modules/makers of note: Mutable instruments DAW: audio plugins · Ableton notes · MuLab notes · Mainstage notes · VCV Rack notes Unsorted: Visuals DIY · Signal analysis, modeling, processing (some audio, some more generic) · Music fingerprinting and identification For more, see Category:Audio, video, images

Some technical background

VGA notes

You can mess with the pixel voltages directly.

Of the many pins, the only things that are truly required are

• Hsync
• Vsync

TTL signals (0..5V)

• red
• green
• blue

0V for black, 0.7V for full intensity

You will typically use hsync and vsync only to establish a specific resolution, and mess only with R, G, and B.

Ideally you also consider the back porch, front porch. Not doing so will CRTs look extra janky (which can be a feature) while LCDs may actually clean that up for you ...or have their auto adjust fail.

Feeding in audio is mostly just a diode away (to protect from negative voltages), also because it's roughly the right voltage ([Voltage_levels#Analog_audio_voltage_levels consumer audio is approx 0.3V RMS])

That said, baseband audio will mostly look like lots of horizontal lines, due mainly to audio rate being multiples slower than the pixel rate.

You can generate a a basic digital VGA signal from a microcontroller, e.g.

• AVR https://hackaday.com/2013/03/29/avr-vga-generator/
• PIC https://blogs.fsfe.org/pboddie/?p=1712
• ESP32 https://www.instructables.com/ESP32-Basic-PC-With-VGA-Output/
• RP2 https://github.com/tvlad1234/pico-vgaDisplay

More interesting things:

ewa justka, e.g. https://ewajustka.tumblr.com/post/141525276739/audio-video-synth-based-on-arduino-based-vga

http://www.jameshconnolly.com/rgb-vga-volt.html

When modes are sort of weird things

Let's assume for a moment that we're a CRT - TV, monitor, or similar.

We're bending a ray of electrons that light up the phosphors, line by line.

Hsync is the time the beam has to go to the next line.

Vsync is the time in which the beam goes from bottom right to top left for the next frame.

💤 And to get to more detail that is tangentially interesting

Both imply blanking intervals, times at which the beam is not beaming. (If you didn't, you would see little lines dragged back during hsync, and one big diagona for vsync.)

The blanking interval has a few more defined times, actually, namely the porches,

which are basically guard areas of zero signal that make it harder for position adjustments to start causing that issue (but on enough monitors, at the extremes of positioning this might still start to happen).

How does a video card know what a monitor supports?

Sometimes not. Early and simple monitors might support just one mode.

Some other monitors would only support a smallish number of modes. The hsync and vsync are almost a fingerprint of what mode it is, so a monitor could detect something it knew and switch to it.

This isn't perfect, because sometimes the closest mode it knows wouldn't be right, but a decent set of modes worked well enough.

In terms of user experience, this was finicky, because setting the wrong mode would give you no image. The OS would probably just choose the most standard low-resolution mode.

Implicitly, they left it up to us, the users, to trial and error what higher resolution might work as well.

And then, not all monitors supported higher resolutions in exactly the same way - timing details might vary a little, so the 'close mode is good mode' approach sort of stopped working. You could trial and error that too, if you like editing config files for a few hours.

Particularly EDID made things easier, reading out an EEPROM that contains information about supported resolutions and their timing.

Analog video hacking

✎ This article/section is a stub — some half-sorted notes, not necessarily checked, not necessarily correct. Feel free to ignore, or tell me about it.

Generating video

If you squint and consider it just a 'spitting out analog pixels', composite is broadly similar to VGA, but with a few differences due to its TV history.

This is of some interest to us in that 640x480@60's timings closely related to NTSC. This is by design, and potentially useful for some electronics projects in that with fairly little extra work you could output to either/both to a monitor and e.g. composite output.

Output to VGA or composite?

NTSC is 720x480@60i, PAL is 720x576@50i.

The timing of 640x480@60 VGA is close to NTSC composite, this was intentional at the time(verify).

and due to similar timings, even the sync signals are similar.

They're different enough that there is no passive conversions between the two, yet in theory if you design with the constraint of both in mind, you could make something that can output to either of the two at a time, or if you're clever, both at the same time.

Differences include:

• the image signal is luma and chroma (though you can get away with just luma)
• on TV there's a color burst thing,

a ~3.6MHz signal(verify) you have to generate at the right time (and takes some time away from that idle time),

• the sync is put in a little differently

but is well within your capabilities.

Microcontrollers don't have much memory, or spare time, to generate fine graphics.

You can have external ram, from a few hundred KB from SPI SRAM, to more from things like e.g. http://tinyvga.com/avr-sdram-vga or the later stages of http://www.lucidscience.com/pro-vga%20video%20generator-14.aspx. But even then, putting something interesting in that buffer is challenging because of the limited CPU in simple uCs, and the fact you're using much of the time outputting pixels. This is one of the reasons that video cards quickly moved to using RAMDACs.

(Mind you, the Due can do better, see e.g. https://stimmer.github.io/DueVGA/]

Just the hsync and vsync is fairly easy to do, in both VGA and composite.

For VGA you have separate R,G,B signals.

For the R,G,B most take one of three approaches:

• use PWM outputs for R,G,B

e.g. vgax uses two for just two of the colors

• use a digital port and resistor ladder for R,G,B

and a basic 2-bit resistor ladder, e.g. for 2-bit on each channel (sometimes cleverer)

e.g. [1]

e.g. [2]

• Monochrome is somewhat easier

e.g. http://tinyvga.com/avr-vga

In composite, monochrome is easier in that color composite is a little more timing-sensitive

VGA and TV:

http://www.serasidis.gr/circuits/AVR_VGA/avr_vga.htm

http://www.lucidscience.com/pro-vga%20video%20generator-1.aspx (dead link)

Note that for composite PAL/NTSC TV output (only) you may like things like the AD724

https://web.archive.org/web/20260110052117/https://os.mbed.com/forum/mbed/topic/2459/

https://electronics.stackexchange.com/questions/23579/whats-the-simplest-way-to-generate-a-vga-signal-for-a-totally-white-screen-pre

On sync and pixel speed - and hobby electronics applications

Most halfway decent resolutions have a pixel clock above 50MHz, and modern resolutions a few hundred MHz. (Yes, VGA can carry 1080p -- over a short wire, anyway.)

The 640x480 at 60Hz resolution is a bit of an industry standard, so you can expect even very old monitors will accept it. see timing details here, but note that it's clocked at 25.175MHz.

For microcontrollers like PICs and AVRs, even ~25MHz is too fast, so sending out all individual pixels can't be done directly.

There's also no AVR/PIC peripheral generic enough that we can fully delegate such output, so even for lower rates

you have to directly bit-bang GPIO,

the overhead of doing that at all means you can't practically do more than about half your own clock rate to start with,

and the rate during pixel output is fixed so you spend most CPU getting the timing right.

One saving grace is that the time that hsync+vsync blanks (meaning you don't need to send pixels) for the 640x480@60Hz mode is about ~30% of the overall time, which you can spend thinking of what the next frame will be.

The blanking region is spent with the RGB pins blanked.

On CRT, not doing that would put stripes all over the screen. LCDs are more forgiving if you don't actually blank, but blanking helps the 'auto adjust' feature to, well, work. Similar for the color burst in PAL and NTSC - you probably want to do this.

So what do we do on <25MHz microcontrollers?

HSync and VSync happen in the kHz range, so getting the monitor to go to the right mode is well within your capabilities here (possibly using timed interrupts, for accuracy).

It's only the pixels that go faster.

As it's supposed to be voltages back to back, and analog, screens don't care, or know, if you hold the voltage for what they would think is multiple pixels.

You'll just get wider pixels that way, and if you can accept that, just go at a multiple slower. you can easily change it at half or quarter the rate.

If you want to display a stored image, then the RAM you have limits the amount of pixels you can store anyway (a full screen of pixels would take ~300KB), so many people decide to also repeat lines, to get larger square pixels, and use less memory.

Showing text is easier, because it's effectively a lookup table of ~100 possible bitmaps.

For a lot of these tricks, look also to early TV-connected home computers - they did a lot of these things too, even if they usually separated video output from main work.

Another reason you can't get perfectly crisp images is that the an AVR at 16MHz or 20MHz isn't a divisor of 25.175, so the timing won't align so well, but call it retro and it's fine.

More on timing

The 640x480@60Hz mode describes just the visible pixels.

Timing-wise, you could see it as something like 800x524@60Hz (which is where almost all of that 25.175MHz comes from) with a whole bunch of pixels intentionally blanked.

From that view, horizontal lines are split like, in pixel-clock units:

• 96 sync
• 48 back porch
• 640 visible
• 16 front porch

Say you make an AVR go at 20MHz.

That same line, in terms of wallclock time, sums to ~636 clocks at 20MHz, which you can e.g. split up like 76+36+512+12 (which steals some time from the porch so that pixel generation can be regular)

Since an AVR can output a value every two clocks, this gives you an effective horizontal resolution of 256 pixels

You could also get some color, because AVR GPIO lets you set an 8-pin port at once, so with some resistor-ladder work you could do e.g. 2-bit color for the RGB channels (or basically 8-bit color if you're clever).

For a sense of magnitude: a pixel takes ~40ns, a line takes ~30us, a frame takes ~17ms

See e.g.

• http://wiki.osdev.org/Video_Signals_And_Timing

• http://tinyvga.com/vga-timing
• http://martin.hinner.info/vga/timing.html

• http://www.osdever.net/FreeVGA/vga/vgacrtc.htm

• https://www.artekit.eu/vga-output-using-a-36-pin-stm32/

• https://en.wikipedia.org/wiki/Video_Graphics_Array#Signal_timings

On slightly faster uCs

On things like STM32s, even the simpler STM32F103, we have

72MHz of speed,

more RAM for a framebuffer

This lets us

do 640x480 with more colors and/or detail,

manage 800x600@56Hz (36MHz pixel clock, conveniently half of a 72MHz STM32's main clock)

Particularly for the latter you now want to learn about DMA, and you still won't have enough RAM for a full res framebuffer, so projects seem to do either monochrome at higher res, or color at an effective 400x300 or so.

See e.g.

• https://hackaday.com/2015/06/07/better-vga-on-the-stm32f4/

• http://cliffle.com/blog/pushing-pixels/

• https://www.seanet.com/~karllunt/S4DVGA.html

• https://www.artekit.eu/vga-output-using-a-36-pin-stm32/

RP2

Interesting because

it's a order-of 100MHz CPU,

you can offload the signal transmission to the PIO

we have some RAM for a framebuffer

That leaves you with most of the CPU time to actually spend on the visuals.

Audio as image

You can put audio directly onto the color pins of VGA, while have something else (uC or computer) set the mode via sync signals.

Why it's darkish

For starters, consumer audio is roughly -0.3V ... 0.3V when loud, and VGA color in voltages are 0 .. 0.75V. Presumably the VGA clips out half that wave and the other part is probably often below 0.1V.

Biasing and amplifying (an op amp per channel) would help (and once you're at that you may be able to mix something for the third color too).

Why is audio horizontal stripes?

You get no nice patterns out of the box, mainly because anything not a nearly-exact multiple of 60 will walk around so fast it won't look coherent also meaning even if it is strongish signal, it will much less visible

Also, music and speech has most of its energy in the first few kHz, so most things at that pixel clock will be at most a few horizontal lines thick.

You can do a basic experiment, and figure out the math as you go:

Get an arduino or similar generating the sync pins - this should be simple (it's much slower than the pixels), and hook it to H and V lines.

Now hook up a tone generator - a website will do for basic experiments. You can pretty much wire audio directly into one or two of the R,G,B lines.

Now, consider that in 640x480@60, the screen updates sixty times per second. that means a 60Hz sine wave takes exactly one screen (1/60th of a second, ~17ms), and would look like a single overall vertical gradient.

At exactly 60Hz that gradient would be entirely still. Have it slightly off, e.g. 59 and 61Hz, and that gradient will seem to move around. (fiddling with a tone generator that also does sweeps, perhaps binaural combinations, may be nice here).

Multiples of 60Hz will be a multiple complete waves/gradients per frame, which'll look thinner because, well, it completes in the time of fewer lines for each full wave.

At higher frequencies thing start mattering in fewer lines.

Music is mostly in the few-kHz region, so this will be many multiples of complete waves per frame, and 99+% of them won't sync up with a frame. Or with a line, because the start of each line happens at 31kHz, above audible.

For two directly adjacent lines to differ, you need a signal faster than audible range (since a single line is ~31kHz). So actually most audio is there technically, just not in a very visually interesting way.

This also relates to why audio has no apparent vertical content -- to get a vertical line you would need periodicity of 31kHz, which audio-geared DACs intentionally filter out. Even if they wouldn't, 99+% of that content won't line up with a line.

If you want thin vertical content, that also implies it needs to pulse. e.g. a single pixel vertical line would be a ~40ns pulse (remember the pixel clock is ~25MHz) with ~30us (799px*~40ns) of nothing inbetween.

Modular synth adjacent

Visual output of CV

If you go very modular, then you may want to see your CV.

This can be as simple as a LED, or as complex as a

Level meter

LM3914 - linear, e.g. for level indication

LM3915 - 3db log, e.g. for VU

LM3916

Video modular

https://www.modulargrid.net/e/tags/view/48

https://www.modwiggler.com/forum/viewforum.php?f=48&sid=5cef807773e20d30ca802a5e3357e964

Video synths

Video synthesizers are anything that creates a video signal beyond just copying video input, and usually involves some ability to control it.

Whether this is prepared or live, audio-reactive reactive or knob-twiddled, will vary.

(Prepared or controlled is more of a guarantee of aesthetics, live and generative may need constant attention)

https://en.wikipedia.org/wiki/Video_synthesizer

Some software notes

Vsynth (library in Max/MSP/Jitter) https://www.kevinkripper.com/vsynth

VideoSync (for Ableton) https://www.showsync.com/videosync/

Hydra

https://hydra.ojack.xyz/

see also https://ojack.xyz/

Touch Designer

https://derivative.ca/

Lumen

https://lumen-app.com/

Synesthesia

https://synesthesia.live/

fragment flow

https://www.fragmentflow.com/

Euler VS

https://www.eulervs.com/

https://gerenm.net/visual-synthesis/

Some hardware notes

3TrinsRGB1

http://gieskes.nl/instruments/?file=3TrinsRGB1

HSS3jb

https://bleeplabs.com/hss3jb/

http://forum.vintagesynth.com/viewtopic.php?f=1&t=70757

http://www.sabinegruffat.com/Arduino-Video-Synth.html

http://hotchk155.blogspot.com/

Older hardware notes

Atari Video Music (C240)

• The board is fairly simple, mostly a few op amps, and a custom chip that does pretty much all the work.
• http://www.atarimuseum.com/videogames/dedicated/videomusic/videomusic.html
• https://www.youtube.com/watch?v=INnpnJvDXDg
• https://technabob.com/blog/2007/08/24/atari-video-music-forgotten-1970s-tech/#

Sandin Image Processor

https://en.wikipedia.org/wiki/Sandin_Image_Processor

Paik/Abe Synthesizer

http://www.medienkunstnetz.de/works/paik-abe-synthesizer/

https://njp.ma/video/video-synthesizer/

Scanimate systems https://www.youtube.com/watch?v=UHjkMThH0aE

Unsorted

https://www.hypertonal.net/cathodemer/

Software (inpiration mostly)

https://lumen-app.com/

https://airtightinteractive.com/demos/js/ruttetra/

https://www.youtube.com/watch?v=JsGf65d5TfM

Video wall

✎ This article/section is a stub — some half-sorted notes, not necessarily checked, not necessarily correct. Feel free to ignore, or tell me about it.