[Bug] HDR transcodes washed-out green colors in web browser - tonemap_opencl missing BT.2020 > BT.709 gamut

[solidsnakex37 56]

Running Emby 4.9.1.90 on Linux (Unraid, Intel Arc A380, QSV hardware transcoding). All HDR content transcoded for web browser playback comes out with a washed-out green/grey color cast. Direct play in the Emby Desktop app (MPV) looks completely correct on the same file. The issue is specific to the server-side transcode pipeline, not the client.

Setup

• Emby Server 4.9.1.90 on Linux (Unraid)
• Intel Arc A380 (QSV + OpenCL)
• ffmpeg: 5.1-emby_2023_06_25_p4
• Affected: all HDR content transcoded for browser (tested with DV HDR10+ and HDR10 sources)

Root Cause

The filter Emby generates for HDR to SDR transcoding is:

tonemap_opencl@f2=tonemap=hable:format=nv12:desat=0

The tonemap_opencl filter does the luminance mapping (PQ to SDR gamma) correctly but the BT.2020 to BT.709 color gamut conversion is never triggered because matrix, primaries, and transfer are not passed. The output pixels are correct in brightness but still in BT.2020 wide-gamut color space, then interpreted as BT.709 by the browser — which produces the green cast.

The bundled ffmpeg does support these parameters. The correct call should be:

tonemap_opencl=tonemap=hable:format=nv12:desat=0:matrix=bt709:primaries=bt709:transfer=bt709

Adding those three params via the Diagnostic Options Parameter Adjustment fixes the output completely. Before and after screenshots attached.

For comparison, Jellyfin's filter builder already emits all three params (from their public GitHub issues):

tonemap_opencl=format=nv12:p=bt709:t=bt709:m=bt709:tonemap=bt2390:peak=100:desat=0

Decompiled Emby.Server.MediaEncoding.dll and found two pipeline builders — PipelineBuilderVaapi.cs and PipelineBuilderQuickSync.cs. Comparing the two, the VAAPI Native branch in PipelineBuilderVaapi.cs correctly sets the gamut conversion properties on its filter wrapper:

tonemap_vaapi.primaries = bt709 tonemap_vaapi.transfer = bt709 tonemap_vaapi.matrix = bt709

PipelineBuilderQuickSync.cs does not. It sets format, tonemap, desat and param on the tonemap_opencl wrapper but never calls .primaries, .transfer or .matrix, which is why they're absent from the generated filter string.

The 4.10 beta changelog (from 4.10.0.1 onwards) includes the line:

"Fix Hardware Transcoding for HDR Video when Tone Mapping is enabled"

Can you clarify exactly what this covers? There are at least two distinct hardware tone mapping issues that have been reported on Linux/QSV, and it's not clear if this one line addresses one, both, or something else:

• The gamut conversion bug described above — tonemap_opencl missing matrix/primaries/transfer params, causing incorrect BT.2020 > BT.709 conversion and green/washed-out output
• A separate regression introduced between 4.9.1.90 and 4.9.3.0 — on the same hardware (Linux, Intel Arc A380, QSV+OpenCL), hardware tone mapping stopped working correctly in 4.9.x builds after 4.9.1.90. This is what originally forced a downgrade back to 4.9.1.90, where tone mapping at least functions (albeit with the gamut issue now identified). Whether these are related or separate code paths is unclear.

Knowing which issue(s) the 4.10 fix addresses would help determine whether upgrading to the 4.10 beta resolves both problems or just one of them.

Code finding

Decompiling Emby.Server.MediaEncoding.dll (the assembly that builds the filter graph) confirms that the tonemap_opencl filter wrapper in Emby.Ffmpeg.Lib.VideoFilters exposes set_matrix, set_primaries, and set_transfer property setters. The filter builder sets Tonemap, Format, and Desat when constructing the node but never calls the three gamut setters, so they're omitted from the emitted command. The fix is calling those three setters with bt709 when the source colorspace is BT.2020.

This affects all HDR content being transcoded — not just DV. DV just made it more obvious.

Workaround

In Administration > Transcoding > Diagnostic Options > Parameter Adjustment:

• Text to Replace: tonemap=hable:format=nv12:desat=0
• Replacement Text: tonemap=hable:format=nv12:desat=0:matrix=bt709:primaries=bt709:transfer=bt709

This works but doesn't survive an Emby restart, so it's not a long-term fix.

Logs attached

• ffmpeg-transcode-b5aa71dd-80e0-4477-9cac-fb28f5e465c7_1.txt Original broken transcode — shows Emby's default filter tonemap=hable:format=nv12:desat=0 with no gamut params, and the broken output stream tagged bt2020nc/bt2020/bt709 (mixed colorspace, incorrect)

  ---

  ffmpeg-transcode-da131b85-37f6-4b93-92af-6e252433ca15_1.txt Working fix — shows matrix=bt709:primaries=bt709:transfer=bt709 applied successfully, no errors, full QSV hardware pipeline intact, output correctly tagged h264, qsv(tv, bt709, progressive)

Before parameters added:

https://claude.ai/api/5fac2b87-d075-4cac-b784-d77a895fe492/files/b2a6b0d5-40fb-41b8-a1ad-c9e04c27158b/preview

After parameters added:

https://claude.ai/api/5fac2b87-d075-4cac-b784-d77a895fe492/files/60851e0b-6e02-442a-a234-162dd91c1194/preview

ffmpeg-transcode-b5aa71dd-80e0-4477-9cac-fb28f5e465c7_1.txt ffmpeg-transcode-da131b85-37f6-4b93-92af-6e252433ca15_1.txt

Edited 1 hour ago by solidsnakex37
more findings