Video Reference Frames
Reference frames are decoded pictures retained so that other pictures can use them for prediction.
A decoder reconstructs pictures into memory. Some are kept because another picture will use their samples for prediction; these are reference pictures. Others are kept because their display time has not yet arrived.
One picture can have both roles. The decoded picture buffer tracks these uses so a surface is not released while either job still needs it.
A displayed picture can remain a reference
Consider a GOP where I0 is displayed first and P3 later predicts from it. Displaying I0 does not make its samples dispensable: P3's reconstruction still needs them.
After decoding P3, the decoder can retain both I0 and P3 for B1 and B2, which use the anchors around them. Once every required dependent has been decoded and the stream no longer retains I0 as a reference, that reference role can end.
A picture's lifetime is therefore set by the actual dependency instructions, not simply by whether the user has already seen it.
A reference can also wait for display
P3 may be decoded before B1 and B2 because they need it. Its samples already exist, but showing it immediately would put the film out of order.
| Picture state | Why memory remains occupied |
|---|---|
| Already displayed, still a reference | Future decoding needs its samples |
| Decoded ahead of display | Its presentation time has not arrived |
| Both reference and pending output | Both requirements apply to one picture |
| Neither role remains | The surface can become available for reuse |
These roles can overlap, so adding a reported reference count to a reported reorder count does not automatically give the number of distinct stored pictures.
Some reference pictures remain useful for longer stretches, such as an older view of a background temporarily covered by a foreground object. Codecs provide explicit rules for retaining and releasing the available references.
Count the samples in a surface
For an even-sized, tightly packed 8-bit 4:2:0 example, the main plane has width × height samples. Each of the two color-difference planes has half the width and half the height:
W × H + 2 × (W/2 × H/2) = 1.5 × W × H samples.
With one byte per sample:
| Grid | One picture's sample payload | Four pictures' payload |
|---|---|---|
| 1920 × 1080 | 3,110,400 bytes, 3.1104 MB | 12.4416 MB |
| 3840 × 2160 | 12,441,600 bytes, 12.4416 MB | 49.7664 MB |
The bytes are decoded sample payloads, not compressed packet sizes. A low-bitrate UHD stream still reconstructs a UHD surface.
If the same sampling grid uses 16-bit containers for its samples, one FHD picture requires 6,220,800 payload bytes and four require 24,883,200 bytes. A 10-bit sample can occupy such a 16-bit container, so valid precision and allocated storage width are different quantities.
Actual decoder allocations can add row alignment, a larger coded raster, additional working surfaces, and an output queue. The calculation isolates the visible-grid payload rather than estimating every implementation's total memory.
Connect lifetime to decoder capacity
More retained pictures need more surface storage. Larger pictures increase each surface's footprint. Codec level constraints describe relevant buffer resources along with picture size and throughput.
In Emby playback, this helps explain why recognizing the codec is only one part of source decoding. The active decoder must hold the pictures required by that bitstream's dependencies and presentation order.
When inspecting a reference count, establish whether it is a declared maximum, an encoder setting, or observed use. Individual blocks can choose among a smaller set at a particular moment. The dependency graph determines which decoded surfaces must actually stay alive.