Structural Similarity (SSIM) for Video

SSIM compares local luminance, contrast, and structure between decoded pictures and a reference.

Take a small reference patch with values 40,40,60,60. It contains a step from a darker region to a lighter one.

Now compare two reconstructions:

Patch Values Mean squared error from reference
A: brightness shifted 50,50,70,70 100
B: step flattened 50,50,50,50 100

Both have the same squared sample error. A retains the step and its contrast, while B removes it. Structural Similarity (SSIM) compares local statistical relationships that distinguish those results.

Separate mean, contrast and structure

The original SSIM approach uses three relationships:

Local mean: the average sample value. This is its luminance comparison; in this grayscale example it refers to code values, not physical display luminance.

Local contrast: the spread around that mean, described by variance or standard deviation. The step patch has variation; the flat patch has none.

Local structure: how corresponding deviations vary together, described by covariance. When both pictures put their below-mean and above-mean values in the same places, that correspondence is positive.

For the reference, the mean is 50. Its deviations are −10,−10,+10,+10. Using sample variance with denominator N−1:

(100 +100 +100 +100)/3 =133.333…

Patch A has mean 60 and the same deviations. Patch B has mean 50 but all its deviations are zero.

Calculate the combined comparison

For this illustration, use the original combined form:

SSIM =((2µxµy +C1)/(µx² +µy² +C1)) ×((2σxy +C2)/(σx² +σy² +C2))

Here µ denotes the local mean, σx² and σy² are variances, and σxy is covariance. Choose an 8-bit peak convention L = 255, with K1 =0.01 and K2 =0.03:

C1 =(K1L)² =6.5025, and C2 =(K2L)² =58.5225.

The constants stabilize comparisons when a mean or variation is small; two identical flat patches should not require dividing zero by zero.

Test patch Mean Variance Covariance with reference Calculated SSIM
A 60 133.333… 133.333… 0.983624…
B 50 0 0 0.305034…

A's contrast/structure factor is 1, with a smaller change in its mean factor. B's mean matches, but its variance and covariance describe the missing step.

Original four-value arithmetic using uniform local statistics and N−1 variance/covariance. It demonstrates the formula, not the output of a particular video tool's 2D windowing implementation.

Reverse the step

A third patch, 60,60,40,40, has the same mean and variance as the reference, but its deviations are opposite at every position. Its covariance is −133.333….

With the declared constants, its SSIM is approximately −0.640071. Negative covariance can therefore produce a negative index. SSIM is not universally a 0–1 percentage of retained picture quality.

Identical local content gives 1 in this original formulation. That identity case and the reversed-pattern example describe what the mathematical comparison is measuring.

Move from a patch to a video

A real implementation calculates local windows across each picture and combines their results. Window size and weighting, color/plane handling, scale preparation and frame aggregation all affect the number.

The author's suggested image scaling is one concrete example. Netflix's VMAF FAQ documents that its included SSIM applies empirical downsampling that differs from FFmpeg's SSIM filter. Equal input filenames do not make those preprocessing choices identical.

For a conversion comparison, align the picture content and times and declare the measurement geometry. Then examine local or per-frame results where detail changed, rather than relying only on a whole-film aggregate.

See Also

References