Using libplacebo to Fix the Dolby Vision Profile 5 Transcoding Tonemapping Issue

[UnkNow404 3]

The current version cannot transcode DV Profile 5 content correctly. The transcoded video looks like this:

And it should be:

ffmpeg 5.0 support libplacebo to transcode DV Profile 5 to SDR correctly.

Command example:

ffmpeg -init_hw_device vulkan=vulkan -filter_hw_device vulkan -i ${VIDEO_FILE} -c:v hevc -crf 18 -c:a aac -b:a 320k -vf "hwupload,libplacebo=format=yuv420p:colorspace=bt709:color_primaries=bt709:color_trc=bt709,hwdownload,format=yuv420p,subtitles=${SUBTITLE_FILE}" ${OUTPUT}

 

[Luke 42888]

Hi, thanks. This is on our to do list for review. Thanks.

[CFC 31]

On 12/1/2025 at 9:48 AM, Luke said:

Hi, thanks. This is on our to do list for review. Thanks.

Hi, is this still in review, or has it been implemented?

My DV Profile 5 file does not transcode with proper color, but wanted to check whether it should be expected to at the moment.

Thanks!

[Luke 42888]

Hi, we will have a new ffmpeg build on the server beta channel in the near future, so stay tuned for that.

[lswang168 0]

Title

Dolby Vision Profile 5 sources transcode to green/purple: Emby inserts no tonemap for P5, while CUDA DOVI reshaping (jellyfin-ffmpeg tonemap_cuda) fixes it on identical hardware

Body

Environment

• Emby Server 4.9.5.0 (stable), Docker on a QNAP NAS (Premiere active)
• NVIDIA GTX 1050 Ti (Pascal, CC 6.1), driver 575.x, NVENC/NVDEC healthy
• Hardware tone mapping enabled and working for other HDR types

Symptom

Any Dolby Vision Profile 5 source (dv_profile: 5, dv_bl_signal_compatibility_id: 0 — base layer IPTPQc2, no HDR10/SDR compatibility layer) transcoded to 1080p SDR produces a fully green/magenta picture. Playback, speed and subtitles are fine — only color is wrong. Control group, same day, same GPU: HDR10, DV Profile 8.1 and Profile 7.6 all tone-map correctly (tonemap_cuda=hable appears in the ffmpeg-transcode logs).

Root cause observed

For P5 sources Emby's graph builder inserts no tonemap filter at all (verified from the ffmpeg-transcode logs). The IPT (ICtCp) data is then scaled and encoded as if it were ordinary YCbCr, hence green/purple.

What we tried that did NOT work (all verified empirically, same frame)

• EnableSoftwareToneMapping=true: byte-identical filter graph and byte-identical output frames (md5) for P5 — zero effect.
• Emby's built-in supertonemap filter on a P5 frame: pure black output.
• libplacebo/Vulkan route (forum topic 144597): not viable in our NVIDIA Docker setup — no Vulkan ICD manifests, runc container with manually bound /dev/nvidia* (no nvidia-container-runtime), and no NVIDIA driver libraries inside the GPU image.

What DOES work on the exact same hardware and frame

• jellyfin-ffmpeg 8.1.2 tonemap_cuda: colors fully correct. DOVI reshaping for P5 is baked into the CUDA kernel (jellyfin-ffmpeg#151, never upstreamed to FFmpeg).
• tonemapx (CPU software route): also correct.

The blocker is server-side graph generation

An ffmpeg binary with P5 reshaping is necessary but not sufficient: Emby decides the filter graph, and for P5 it decides on no tonemap. Swapping in jellyfin-ffmpeg wholesale is not viable for us either, because Emby's ffmpeg is a fork with private filters (e.g. supertonemap) that our working HDR10 / P8.1 / P7.6 paths depend on.

Request

1. Add Dolby Vision Profile 5 (IPTPQc2) reshaping to Emby's transcoding pipeline — ideally the CUDA route, since the kernel work is already proven by jellyfin-ffmpeg#151, with a CPU fallback.
2. Failing that, a supported/documented escape hatch to hand P5 transcodes to a custom ffmpeg without disturbing other codec paths.
3. Clarification: does the 4.10.0.24/25 beta changelog item Fix Hardware Transcoding for HDR Video when Tone Mapping is enabled affect DV Profile 5 at all?

Offer

Happy to beta-test on the affected hardware. We have a repeatable one-frame probe (a known P5 file, frame at 420s, compared against reference frames produced by the working tonemap_cuda pipeline), so any candidate build can be verified in about a minute.

Impact note

Direct play on DV-capable clients is unaffected; this only hits the transcode path (remote/cellular viewing). About 30 items / 10 titles in our library are affected.