Video display notes: Difference between revisions

From Helpful
Jump to navigation Jump to search
 
(3 intermediate revisions by the same user not shown)
Line 782: Line 782:
<!--
<!--


What vsync adds
As the above notes, the graphics card pushes pixels to the framebuffer as fast as it can,
a monitor just shows what it gets.


 
We do want to start a frame at the top, so ''roughly'' speaking, ever frame (~16ms when at 60Hz)
As the above notes, we push pixels to the framebuffer as fast as we can, a monitor just shows what it gets (at what is almost always a fixed framerate = fixed schedule).
the video card signals it's sending a new frame.  
 
''Roughly'' speaking, every 16ms, the video card signals it's sending a new frame, and sends whatever data is in its framebuffer as it sends it.




And it has no inherent reason to care whatever else is happening.  
And it has no inherent reason to care whatever else is happening.  
 
If you changed the framebuffer while it was being sent out,
So if you changed that framebuffer while it was being sent out, the from the perspective of entire frames, it didn't sent full, individual ones.
: It is possible that it sent something that looks wrong because it wasn't fully done drawing (this further depends a little on ''how'' you are drawing)
: It is possible that it sent part of the previous and part of a next frame.
: It is possible that it sent part of the previous and part of a next frame.
: It is possible that it sent something that looks wrong because it wasn't fully done drawing (this further depends a little on ''how'' you are drawing)


It's the lowest latency to get things on screen.


It's lowest latency to get things on screen, but can look incorrect.
But can look incorrect.




Line 803: Line 802:
There are two common ingredients to the solution:
There are two common ingredients to the solution:


* vsync in the sense of waiting until we are between frames
* vsync  
:: The above describes the situation without vsync, or with vsync off. This is the most immediate, "whenever you want" / "as fast as possible" way.  
:: The above describes the situation without vsync, or '''with vsync off'''. This is the most immediate, "whenever you want" / "as fast as possible" way.  
:: '''With vsync on''', the change of the image to send from is closely tied to when we tell/know the monitor is between frames.
:: vsync describes the ability for the graphics card to tell that it is between frames
:: It usually amounts to the software ''waiting'' until the graphics card is between frames. That doesn't necessarily solve all the issues, though. In theory you could do all your drawing in that inbetween time, but it's not a lot of time.


* double buffering
* double buffering
Line 813: Line 811:




Double buffering with vsync off means ''switch as soon as we're done drawing''.
'''Double buffering with vsync off''' means ''swap buffers as soon as we're done drawing''.
It solves the incomplete-drawing problem, and is fast in the sense that it otherwise doesn't wait to get things on screen.  
It solves the incomplete-drawing problem, and is fast in the sense that it otherwise doesn't wait to get things on screen.  


But we still allow that to be in the middle of the monitor pixels being sent out, so it can switch images in the middle of a frame.  
But we still allow that to be in the middle of the monitor pixels being sent out, so it can switch images in the middle of a frame.  
'''Double buffering with vsync on''' means ''swap buffers the first vsync moment after we're done drawing''.


If those software flips are at a time unrelated to hardware, and/or images were very similar anyway (slow moving game, non-moving office document),
If those software flips are at a time unrelated to hardware, and/or images were very similar anyway (slow moving game, non-moving office document),
Line 837: Line 839:


Another benefit to vsync on is that we don't try to render faster than necessary.  
Another benefit to vsync on is that we don't try to render faster than necessary.  
60Hz screen? We will be sending out 60 images per second.
Have a 60Hz screen? We will be sending out 60 images per second.


...well, ''up to'' 60. Because we wait until a new frame, that means a very regular schedule that we have no immediate control over.  
...well, actually, ''up to'' 60.
Because we wait until a new frame, that means a very regular schedule that we have no immediate control over.  


At 60Hz, you need to finish a new frame within 16.67ms (1 second / 60).
So at 60Hz, you need to finish a new frame within 16.67ms (1 second / 60).
Did you take 16.8ms? Then you missed it, and the previous frame will stay on screen for another 16.67ms.
Did you take 16.8ms? Then you missed it, and the previous frame will stay on screen for another ~16ms.




Line 848: Line 851:
: 16.6ms (60fps)
: 16.6ms (60fps)
: 1 frame late means it has to come 33.3ms after the start instead, which you could say is momentarily 30fps (60/2)
: 1 frame late means it has to come 33.3ms after the start instead, which you could say is momentarily 30fps (60/2)
: 2 frames late is 50ms later, for 20fps (60/3)
: 2 frames late is 50ms later, for a momentary 20fps (60/3)
: 3 frames late is 66ms later, for 15fps (60/4)
: 3 frames late is 66ms later, for a momentary 15fps (60/4)
: ...and so on, but we're already in "looks terrible" territory.
: ...and so on, but we're already in "that looks terrible" territory.




Line 1,020: Line 1,023:
-->
-->


====Adaptive sync====
====Variable refresh rate====
<!--
<!--
{{stub}}
{{stub}}
Line 1,106: Line 1,109:


G-Sync apparently won't go below 30Hz which seems to mean your typical vsync-styl integer reduction
G-Sync apparently won't go below 30Hz which seems to mean your typical vsync-styl integer reduction
I have seen it called "adaptive sync",
but I've also seen a confusingly similar term "adaptive vsync" meanin "we block if we are above the display rate,
we allow tearing below" ...which sounds like frame limiting without vsync on.




Line 1,298: Line 1,309:
:: more so with vsync on
:: more so with vsync on
::: because that actually implies a mix of 30 and 60
::: because that actually implies a mix of 30 and 60
::: it's arguably better if it's consistently not managing 60, in which case it becomes a solid 30
::: it's arguably better if it's consistently not managing 60 (in which case it becomes a solid 30) than if it's ''sometimes'' managing (which is a mix)
 


* the difference in how it plays is tiny for almost all cases  
* the difference in how it plays is tiny for almost all cases  
:: but if it's your career, you care.
:: but if it's your career, you care.
:: and if it truly affects comfort, you care
:: and if it truly affects comfort, you care
 
:: and if you convince yourself of more than is real, you care (but I don't)




Line 1,317: Line 1,329:


In the CRT days there was an extra reason you might would care separately about Hz - see [[#Reproduction that flashes]] above.
In the CRT days there was an extra reason you might would care separately about Hz - see [[#Reproduction that flashes]] above.




Line 1,328: Line 1,339:


Including, but not limited to,  
Including, but not limited to,  
: a fully fixed framerate  
: a fully fixed framerate
: optical expectations
: optical expectations
: expectation in general
: expectation in general
Line 1,335: Line 1,346:




The last two are guided by cinematic languge - it suits most storytelling to not have too much happening most of the time,  
The last two are guided by cinematic language - it suits most storytelling to not have too much happening most of the time,  
and to have ''mostly'' slow movement.
and to have ''mostly'' slowish movement. It helps us parse most of what's happening, which is usually the point.
It helps us parse most of what's happening, which is often the point, at least in most movies.


Sure, this partly because movies have adapted to this restriction {{comment|(and an old one at that - 24fps was settled on in the 1930s or so, just as an average of the varying speeds from very early film that made it harder to show film)}}, but is partly true regardless.
Sure, this is ''partly'' because movies have adapted to this restriction {{comment|(and an old one at that - 24fps was settled in the 1930s or so, seemingly as the average of the varying speeds that were around then)}}.


{{comment}(Frame rate was also once directly proportional to cost - faster speeds meant proportionally more money spent on film stock. In the digital era this is much less relevant)}}
{{comment}(Frame rate was also once directly proportional to cost - faster speeds meant proportionally more money spent on film stock. In the digital era this is much less relevant)}}
Line 1,349: Line 1,359:
And no, movies at 24fps don't always look great - a 24fps action scene does ''not'' look quite as good as good as a higher speed one.
And no, movies at 24fps don't always look great - a 24fps action scene does ''not'' look quite as good as good as a higher speed one.
So action movies may well have faster framerates now.
So action movies may well have faster framerates now.




Line 1,356: Line 1,364:
'''Optics'''   
'''Optics'''   


Blur, depth of field, and other optical imperfections that happen in film sound bad,
Blur, [[depth of field]], and other optical imperfections that happen in film sound bad,
but we're so used to them that they help us parse what's happening.
but we're so used to them that they help us parse what's happening - depth of field helps us focus on the important bits by blurring out the less important stuff, motion blur tells us where we're going.  
: Depth of field helps us focus on the important bits.
: Motion blur tells us where we're going.  
Both can be used well to help us focus mostly on important things without us ever noticing.
Both can be used well to help us focus mostly on important things without us ever noticing.


Line 1,380: Line 1,386:
"You're saying 30fps with motion blur would be good?"
"You're saying 30fps with motion blur would be good?"


It might. And it would be a real consideration, were it not that that motion blur must be simulated,
It might be a real consideration, except that in the digital world motion blur must be simulated,
and doing that ''well'' is more costly to render than aiming for 60fps without motion blur.  
and doing that ''well'' is more costly to render than aiming for 60fps without motion blur.


So it's typically not worth it, ''especially'' not in games that go for twitchy shooting,
So it's typically not worth it, ''especially'' not in games that go for twitchy shooting
rather than cinematics (for a few different reasons, not just the amount of motion).
rather than cinematics (for a few different reasons, not just the amount of motion).


(It seems that some people get queasy from motion blur - possinly often because it tends to be inconsistent framerates)
(It seems that some people get queasy from motion blur - possibly often because it tends to be inconsistent framerates)




Line 1,394: Line 1,400:
The faster the movement is, the more noticeable a low framerate is.  
The faster the movement is, the more noticeable a low framerate is.  


Say, if that ball in the gif was not moving at half the speed, the jumpiness would be much less noticeable.
Say, if that ball in the GIF was not moving at half the speed, the jumpiness would be much less noticeable.
 
 
This is actually more complex topic. The fact that that ball is against a plain, static background lets us focus on it.
 
Games almost never have a static background.
: Much less have a static background  ''and'' have a fast moving object'', at the same time.
 
In games, things move relative to each other, and




This is actually more complex topic. The fact that that ball is against a plain, static background lets us focus on it,
few games have an empty, static background, fewer have a static background ''and'' a fast moving object at the same time.




Line 1,644: Line 1,644:




====On reaction time====
====On human reaction time====
<!--
<!--


Line 1,651: Line 1,651:


'''For things you do not or cannot anticipate''', reaction time is easily over 300ms and rarely below 200ms.  
'''For things you do not or cannot anticipate''', reaction time is easily over 300ms and rarely below 200ms.  
Formula one racers waiting for the light to change still take around 200ms.


Formula one racers waiting for the starting light are impressive by being closer to the 200ms end of that.
That's true for visuals - it can be a few dozen ms lower for audio and touch (150ms or so), presumably because it's less information to process{{verify}} or the biology just has fewer steps{{verify}}


...for visuals - it can be a few dozen ms lower for audio and touch, presumably because it's less information to process{{verify}}
When designing things with personal safety in mind, such as "braking distance for drivers", this guides the ''absolute minimum'' figure, usually with a lot of safety margin on top of this.
 
Because consider that at moderate speed, you have traveled 8 meters before you've processed enough to then spend another bunch of meters moving your foot on the brake pedal - it'll be a bunch more before your foot is pressing the brake hard enough, and the friction to slow down is actually happening. And even then the typical calculations are for ''cars'' which have a lot less momentum than trucks to get rid of)}}.


When designing things with personal safety in mind, such as "braking distance for drivers", this guides the ''absolute minimum'' figure, usually with a lot of safety margin on top of this.  {{comment|(Consider that's 8 meters traveled before you've even processed enough to ''start'' moving your foot towards the brake pedal - it'll be a bunch more before your foot is pressing the brake hard enough, and the friction to slow down is actually happening. And even then the typical calculations are for cars, not for trucks that have many times more momentum to get rid of)}}.


'''fully anticipated reactions''' can be much faster.
: consider rhythm games
:: they are on a fully regular pace, and you can work from memory, so you can estimate exactly
:: you can also negate the motor-control latency by starting early
:: you can negate some of the controller latency the same way
:: This requires feedback to train yourself, but people then start doing it intuitively, and well.


'''fully anticipated reactions''' can be much faster
: These games judge you at timing within a much smaller window
: e.g. when they are purely about motor control, which you can start early
:: where more than 30ms or so[https://gaming.stackexchange.com/questions/294060/which-bemani-rhythm-game-in-the-arcades-has-the-strictest-timing-window] may just be considered 'bad'
: for example, rhythm games are on a fully regular pace, and you can work from memory,
:: and expert players can get timing within 15ms pretty consistently.
: and you can negate some of the controller latency by hitting it early - you'll probably learn to do this intuitively
: These games judge you at timing within a much smaller window, where more than 30ms or so[https://gaming.stackexchange.com/questions/294060/which-bemani-rhythm-game-in-the-arcades-has-the-strictest-timing-window] may just be considered 'bad', and expert players can get timing within 15ms pretty consistently.






Most games are somewhere inbetween these two extreme.  
Most games are somewhere between these two extremes.


Yes, if someone is walking in a straight line, then yeah, you can figure out precisely where they'll be.
Yes, if someone is walking in a straight line, then yeah, you can figure out precisely where they'll be.

Latest revision as of 17:41, 25 June 2024

The physical and human spects dealing with audio, video, and images

Vision and color perception: objectively describing color · the eyes and the brain · physics, numbers, and (non)linearity · color spaces · references, links, and unsorted stuff

Image: file formats · noise reduction · halftoning, dithering · illuminant correction · Image descriptors · Reverse image search · image feature and contour detection · OCR · Image - unsorted

Video: file format notes · video encoding notes · On display speed · Screen tearing and vsync

Simpler display types · Video display notes · Display DIY


Audio physics and physiology: Sound physics and some human psychoacoustics · Descriptions used for sound and music

Noise stuff: Stray signals and noise · sound-related noise names · electronic non-coupled noise names · electronic coupled noise · ground loop · strategies to avoid coupled noise · Sampling, reproduction, and transmission distortions · (tape) noise reduction


Digital sound and processing: capture, storage, reproduction · on APIs (and latency) · programming and codecs · some glossary · Audio and signal processing - unsorted stuff

Music electronics: device voltage and impedance, audio and otherwise · amps and speakers · basic audio hacks · Simple ADCs and DACs · digital audio · multichannel and surround
On the stage side: microphones · studio and stage notes · Effects · sync


Electronic music:

Electronic music - musical terms
MIDI · Some history, ways of making noises · Gaming synth · microcontroller synth
Modular synth (eurorack, mostly):
sync · power supply · formats (physical, interconnects)
DAW: Ableton notes · MuLab notes · Mainstage notes


Unsorted: Visuals DIY · Signal analysis, modeling, processing (some audio, some more generic) · Music fingerprinting and identification

For more, see Category:Audio, video, images

Many-element, video display - TV and monitor notes (and a little film)

Backlit flat-panel displays

CCFL or LED backlight

https://nl.wikipedia.org/wiki/CCFL

Self-lit

OLED

This article/section is a stub — some half-sorted notes, not necessarily checked, not necessarily correct. Feel free to ignore, or tell me about it.

OLED are organic LEDs, which in itself party just a practical production detail, and really just LEDs. (...though you can get fancy in the production process, e.g. pricy see-through displays are often OLED with substate trickery(verify))

While OLED is also a thing in lighting, OLED usually comes up in the context of OLED displays.


OLED displays are mainly contrasted with backlit displays. OLEDs being off just emit no light at all (compared to pixels that block backlight, because it is hard to get those to block all light) So the blacks are blacker, you could go brighter at the same time, There are some other technical details why they tend to look a little crisper.

Viewing angles are also better, roughly because the light source is closer to the surface.


PMOLED versus AMOLED makes no difference to the light emission, just to the way we access them (Passive Matrix, Active Matrix).

AMOLED can can somewhat lower power, higher speed, and more options along that scale(verify), all of which makes them interesting for mobile uses. It also scales better to larger monitors.

POLED (and confusingly, pOLED is a trademark) uses a polymer instead of the glass, so is less likely to break but has other potential issues


QLED

On image persistence / burn-in

VFD

Vacuum Fluorescent Displays are vacuum tubes applied in a specific way - see Lightbulb_notes#VFDs for more details.



Some theory - on reproduction

Reproduction that flashes

Film
This article/section is a stub — some half-sorted notes, not necessarily checked, not necessarily correct. Feel free to ignore, or tell me about it.

Mechanical film projectors flash individual film frames while that film is being held entirely still, before advancing that film to the next (while no light is coming out) and repeating.

(see e.g. this and note that it moves so quickly that you see that the film is taken it happens so quickly that you don't even see it move. Separately, if you slow playback you can also see that it flashes twice before it advances the film - we'll get to why)

This requires a shutter, i.e. not letting through any light a moderate part of the time (specifically while it's advancing the film). We are counting on our eyes to sort of ignore that.


One significant design concept very relevant to this type of reproduction is the flicker fusion threshold, the "frequency at which intermittent light stimulus appears to be steady light" to our eyes because separately from actual image it's showing, it appearing smooth is, you know, nice.

Research shows that this varies somewhat with conditions, but in most conditions practical around showing people images, that's somewhere between 50Hz and 90Hz.


Since people are sensitive to flicker to varying degrees, and this can lead to eyestain and headaches, we aim towards the high end of that range whenever that is not hard to do.

In fact, we did so even with film. While film is 24fps and was initially shown at 24Hz flashes, movie projectors soon introduced two-blade and then three-blade shutters, showing each image two or three times before advancing, meaning that while they still only show 24 distinct images per second, they flash it twice or three times for a regular 48Hz or 72Hz flicker. No more detail, but a bunch less eyestrain.


As to what is actually being show, an arguably even more basic constraint is the rate of new images that we accept as fluid movement.

Anything under 10fps looks jerky and stilted
or at least like a choice.
western and eastern animations were rarely higher than 12, or 8 or 6 for the simpler/cheaper ones
around 20fps we start readily accepting it as continuous movement,
above 30 or 40fps it looks smooth,
and above that it keeps on looking a little better yet, with quickly diminishing returns



So why 24?

Film's 24 was not universal at the time, and has no strong significance then or now. It's just that when a standard was needed, the number 24 was a chosen balance between various aspects, like the fact that that's enough for fluid movement and relatively few scenes need higher, and the fact that film stock is expensive, and a standard for projection (adaptable or even multiple projectors would be too expensive for most cinemas).


The reason we still use 24fps today is more faceted, and doesn't really have a one-sentence answer.

But part of it is that making movies go faster is not always well received.

It seems that we associated 24fps to feels like movies, 50/60fps feels like shaky-cam home movies made by dad's camcorder (when those were still a thing) or sports broadcasts (which we did even though it reduced detail) with their tense, immediate, real-world associations. So higher, while technically better, was also associated with a specific aesthetic. It mat works well for action movies, yet less for others.

There is an argument that 24fps's sluggishness puts us more at ease, reminds us that it isn't real, seems associated with storytelling, a dreamlike state, memory recall.

Even if we can't put our finger on why, such senses persist.


CRT screens
This article/section is a stub — some half-sorted notes, not necessarily checked, not necessarily correct. Feel free to ignore, or tell me about it.

CRT monitors do something vaguely similar to movie projectors, in that they light up an image so-many times a second.


Where with film you light up the entire thing at once (maybe with some time with the shutter coming in and out, ignore that for now). a CRT light up one spot at a time - there is a beam constantly being dragged line by line across the screen -- look at slow motion footage like this.


Except what they light up is different. A film projector is just bouncing light off something white.

A CRT is pushing energy into phosphorescence - lighting up a pixel's worth of phosphor at a time. That phosphor has a softish onset, and retain light for a little while.

...but still mostly fallen off a millisecond or two later(verify), so they're definitely pulsing.

The largest reason that these pulsing phosphors don't look like harsh blinking is that our persistence of vision, combined with the fact that it's relatively bright, end up looking fairly constant. (you could say our eyes framerate sucks, though actually this is a poor name for our eyes's actual mechanics).


While TVs were fixed to 50Hz or 60Hz, primarily because they had to deal with one specific broadcast standard, most CRT monitors can be told to refresh at different rates.

There's a classic 60Hz mode that was easy to support, but people often preferred 72Hz or 75Hz or 85Hz or higher modes, primarily because they reduced eyestrain.


And yes, after working behind one of those faster-rate monitors and moving to a 60Hz monitor would be really noticeable. Because even when we accept it as smooth enough, we still perceive it as blinking.


To recap, in TVs and CRT monitors, we light up a line at a time (the line nature is not the only way to use a CRT, just the easiest way to fill the entire screen. For alternatives, see e.g. vector monitors, and CRT oscilloscopes), in fact a pixel at a time. Which happens so fast -- necessarily so -- that you would need a very fast camera to notice this. Take a loook at [1].


This means that there needs to be something that controls the left-to-right and top-to-bottom steering[2] - and because you're really just bending a stream it back and forth, there are times at which that would be a visible line (mostly horizontal-ish between lines, one diagonal between frames).

We solve that by just not emitting electrons just then, and call that the blanking intervals.

That's a lot of intricately timed things, and this is done within the set. In computers, there is a hsync and vsync pulses, which if I understand correctly are not so much control signals as... suggestions, interpreted by the monitor as "does that look like a timing of a mode I know? Okay, then I'll do the rest".(verify)


CRTs were driven relatively directly from the graphics card, in the sense that the values of the pixel we're beaming onto a pixel will be the value that is on the line at that very moment.


Later monitors were not tied to that mechanism, but there was still not that much reason to deviate.

You could add buffers, but why? It would cost more, be more complex, and probably a little slower.



How are CRT monitors different from CRT TVs?

In TVs, redraw speeds were basically set in stone, as were some decoding details.

When each part happened was still synchronized from the received broadcast signal, but the speed was basically fixed, as that made things easier.

On color TV there were some extra details, but a good deal worked the same way.

Early game consoles/computers just generated a TV signal, so that you could use the TV you already had. Which was slightly awkward to do, but a lot cheaper than something dedicated)


After that, CRT monitors started out as adapted CRT TVs, and it didn't take long at all before speed at which things are drawn was configurable. By the nineties it wasn't too unusual to drive a CRT monitor at 56, 60, 72, 75, perhaps 85, and sometimes 90(verify), 100, or 120Hz.

We also grew an increasing amount of resolutions that the monitor should be capable of displaying. Or rather, resolution-refresh combinations. Detecting and dealing that is a topic in and of itself.


Yet at the CRT level, they were driven much the same way - synchronization timing to let the monitor know when and how fast to sweep the beams around, and a stream of pixels passed through as they arrive on the wires.

So a bunch of the TV mechanism lived on into CRT monitors - and even into the flatscreen era.

Flatscreens

This article/section is a stub — some half-sorted notes, not necessarily checked, not necessarily correct. Feel free to ignore, or tell me about it.

For context: in film, and in CRTs, the mechanism that lights up the screen is the is the same mechanism as the one that shows you the image. And, as a result, blinky.


LCD-style flatscreens, the image updates and the lighting are now different mechanisms.

Basically, each pixel blocks light, and there is one big white area behind all the pixels.


If you take a high speed camera, you may still not see it flicker this part of the same slow motion video (note how the backlight appears constant even when the pixel update is crawling by) until you get really fast and specific.


So the difference between, say, a 60fps and 240fps monitor isn't in the lighting, it's how fast the light-blocking pixels in front of that constant backlight change. A 60fps monitor changes its pixels every 16ms (1/60 sec), a 240fps the latter every 4ms (1/240 sec). The light just stays on.

As such, while a CRT at 30Hz would look very blinky and be hard on the eyes, a flatscreen at 30fps updates looks choppier, but not blinkier or more eyestrainy.



On sending and updating pixels
This article/section is a stub — some half-sorted notes, not necessarily checked, not necessarily correct. Feel free to ignore, or tell me about it.


On pixel response time and blur

Vsync

Variable refresh rate

On perceiving

The framerate of our eyes

arguments for 60fps / 60Hz in gaming

On human reaction time

On end-to-end latency

Tracking objects?

On unintentional motion blur

On intentional motion blur

On resolution

On contrast ratio / dynamic range

see also

Visuals_DIY#Analog_video_notes