Intel QSV/VAAPI detected manually, but Emby shows zero hardware codecs in docker. fd leak into ffdetect?

[eunux 0]

hi all. I hit a weird one this week, and after staring at it for longer than I want to admit, I wanted to write it up both as a breadcrumb for the next person googling and as a question for the Emby team: what is the right upstream fix so I can remove this wrapper?

short version: Intel QSV/VAAPI worked perfectly when I ran Emby's bundled `ffdetect` manually inside the container, but Emby's own hardware detection reported zero hardware codecs. The thing that fixed it was wrapping `ffmpeg`, `ffdetect`, and `ffprobe` so they close inherited file descriptors `3+` before execing the real Emby binaries.

environment:

-   Emby Server `4.9.5.0`
-   official `emby/embyserver` Docker image
-   Emby ffmpeg `5.1-emby_2023_06_25_p4`
-   container running in an Ubuntu `24.04.4` VM, kernel `6.8.0-124-generic`
-   Docker `29.6.0`, compose `5.1.4`
-   VM host is Proxmox `9.2.2`
-   Intel mobile 12th-gen iGPU, Alder Lake-P / Iris Xe class, Minisforum MS-01, passed through to the VM
-   `/dev/dri/renderD128` present in the VM and mounted into the container
-   Emby process running as uid/gid `1000`, with the render group added through `GIDLIST`
-   VAAPI in the container reports libva `1.22.0` and Intel iHD driver `25.2.4`

the symptom was not "the GPU is missing". The render node existed, permissions looked right, and manual probes worked. the exact bad state looked like this:

-   Emby's hardware codec inventory/API showed zero hardware codecs, or only software codecs like `x264`/`x265`.
-   the UI behaved as if hardware transcoding was not available.
-   Real playback could fall back to `VideoEncoder=x264` / `VideoEncoderIsHardware=false`.
-   meanwhile, running Emby's own detector by hand inside the same container and same image saw VAAPI/QSV just fine, for example `ffdetect ... vaencdec` and `ffdetect ... qsvencdec` returned the expected codec list.
-   the annoying bit was that Emby-spawned detection did not necessarily throw a clean "permission denied" style error. Would see log errors like `Hardware Detection`, `ffdetect_vaencdec`, `ffdetect_qsvencdec`, `CodecList`, `CodecInformation/Video`, `VideoEncoderIsHardware=false`, and "manual ffdetect works but Emby shows no hardware codecs".

things I tried before the fix:

-   Verified Proxmox PCI passthrough and that the guest saw the iGPU.
-   Verified `/dev/dri/card0` and `/dev/dri/renderD128` in the VM and container.
-   Checked uid/gid/render group mapping.
-   Moved to the official Emby image to remove linuxserver-specific variables.
-   Tested the bundled `ffmpeg`, `ffprobe`, and `ffdetect` directly.
-   Played with the container nofile limit, because I suspected a weird fd issue. That did not fix it.
-   Avoided privileged/seccomp changes; those were not needed.

the clue came from instrumenting the child processes that Emby was spawning. When Emby launched `ffdetect`, the child inherited an unrelated high-numbered fd from the parent EmbyServer process. In my repro it was fd `117`, pointing at an EmbyServer socket. When I launched the same binary manually with `docker exec`, that fd was not there and hardware detection worked. that was the tiny trapdoor in the floor.

the workaround image keeps the vendor binaries as `*.real`, then symlinks `/bin/ffmpeg`, `/bin/ffdetect`, and `/bin/ffprobe` to a tiny launcher:

#!/bin/sh
set -eu

name="${0##*/}"
real="/bin/${name}.real"

for fd_path in /proc/self/fd/*; do
  fd="${fd_path##*/}"
  case "$fd" in
    0|1|2) continue ;;
  esac
  eval "exec ${fd}>&-" 2>/dev/null || true
done

exec "$real" "$@"

after that, Emby's own hardware detection immediately reported the expected QSV and VAAPI codecs. In my case the inventory went from zero hardware codecs to 17 hardware codecs, including QuickSync H.264/H.265 decode and encode plus VAAPI entries. A real browser playback test of a 4K HEVC file then showed hardware decode and hardware encode, and the ffmpeg command line included `-init_hw_device qsv=...`, `hevc_qsv`, `vpp_qsv`, and `h264_qsv`. `intel_gpu_top` also showed Render/3D, Video, and VideoEnhance activity during the stream.

so, question for the team:

1. am I crazy? did I need to do any of this? I could not find another way to get GPU transcoding working from inside Emby, even though the same tools worked manually.
2. is Emby expected to launch the ffmpeg-family tools with arbitrary non-stdio file descriptors inherited from EmbyServer? If not, would the right fix be in the process launcher, making sure only stdin/stdout/stderr are inherited, or that all other fds are marked close-on-exec before spawning `ffdetect`/`ffmpeg`/`ffprobe`?

I am happy to test a build or provide more logs. the wrapper is an okay local bandage, but I would much rather remove it and run the stock image. Also entirely possible there is some very simple knob or obvious docker thing I missed, in which case I would love to be told that too.

[Luke 42607]

Hi, did you try our suggested Docker options?

https://hub.docker.com/r/emby/embyserver

[eunux 0]

i’m using docker. tried both the linuxserver and emby official images — same behaviour either way, which is partly why I moved to the official one, to rule out linuxserver-specific variables.
I did try the suggested options; that’s where I started, and the device/permission side is all correct: /dev/dri/renderD128 is mounted, GIDLIST includes the render group, and the container reports libva 1.22.0 + iHD 25.2.4. The proof it isn’t a setup/permissions problem is that running the bundled ffdetect by hand inside the exact same container and image returns the full codec list (vaencdec/qsvencdec both fine). So everything on the suggested-options page is already satisfied — manual detection succeeds, only Emby-spawned detection fails.
the difference between the two is a leaked file descriptor. when EmbyServer spawns ffdetect/ffmpeg/ffprobe, the child inherits an unrelated high-numbered fd from the parent — in my repro fd 117, pointing at an EmbyServer socket. a manual docker exec launch doesn’t carry that fd, which is why the same binary detects hardware fine by hand but reports zero codecs when Emby launches it. closing all fds ≥3 before exec is the only thing that fixes it: detection went straight from 0 → 17 hardware codecs (QSV H.264/H.265 decode+encode + VAAPI), and a real 4K HEVC playback then used hevc_qsv/vpp_qsv/h264_qsv with intel_gpu_top showing video-engine activity.
so the question is really about the launcher, not the docker options: is EmbyServer expected to spawn the ffmpeg-family tools with arbitrary non-stdio fds inherited? if not, the fix would be setting CLOEXEC on EmbyServer’s own sockets (or only passing stdin/stdout/stderr when forking the detector), which would let me ditch the wrapper.

or, perhaps even better, for someone to tell me why i chased these geese so far when the answer was right in front of me. 

Edited 1 hour ago by eunux