Intel QSV/VAAPI detected manually, but Emby shows zero hardware codecs in docker. fd leak into ffdetect?

[eunux 0]

hi all. I hit a weird one this week, and after staring at it for longer than I want to admit, I wanted to write it up both as a breadcrumb for the next person googling and as a question for the Emby team: what is the right upstream fix so I can remove this wrapper?

short version: Intel QSV/VAAPI worked perfectly when I ran Emby's bundled `ffdetect` manually inside the container, but Emby's own hardware detection reported zero hardware codecs. The thing that fixed it was wrapping `ffmpeg`, `ffdetect`, and `ffprobe` so they close inherited file descriptors `3+` before execing the real Emby binaries.

environment:

-   Emby Server `4.9.5.0`
-   official `emby/embyserver` Docker image
-   Emby ffmpeg `5.1-emby_2023_06_25_p4`
-   container running in an Ubuntu `24.04.4` VM, kernel `6.8.0-124-generic`
-   Docker `29.6.0`, compose `5.1.4`
-   VM host is Proxmox `9.2.2`
-   Intel mobile 12th-gen iGPU, Alder Lake-P / Iris Xe class, Minisforum MS-01, passed through to the VM
-   `/dev/dri/renderD128` present in the VM and mounted into the container
-   Emby process running as uid/gid `1000`, with the render group added through `GIDLIST`
-   VAAPI in the container reports libva `1.22.0` and Intel iHD driver `25.2.4`

the symptom was not "the GPU is missing". The render node existed, permissions looked right, and manual probes worked. the exact bad state looked like this:

-   Emby's hardware codec inventory/API showed zero hardware codecs, or only software codecs like `x264`/`x265`.
-   the UI behaved as if hardware transcoding was not available.
-   Real playback could fall back to `VideoEncoder=x264` / `VideoEncoderIsHardware=false`.
-   meanwhile, running Emby's own detector by hand inside the same container and same image saw VAAPI/QSV just fine, for example `ffdetect ... vaencdec` and `ffdetect ... qsvencdec` returned the expected codec list.
-   the annoying bit was that Emby-spawned detection did not necessarily throw a clean "permission denied" style error. Would see log errors like `Hardware Detection`, `ffdetect_vaencdec`, `ffdetect_qsvencdec`, `CodecList`, `CodecInformation/Video`, `VideoEncoderIsHardware=false`, and "manual ffdetect works but Emby shows no hardware codecs".

things I tried before the fix:

-   Verified Proxmox PCI passthrough and that the guest saw the iGPU.
-   Verified `/dev/dri/card0` and `/dev/dri/renderD128` in the VM and container.
-   Checked uid/gid/render group mapping.
-   Moved to the official Emby image to remove linuxserver-specific variables.
-   Tested the bundled `ffmpeg`, `ffprobe`, and `ffdetect` directly.
-   Played with the container nofile limit, because I suspected a weird fd issue. That did not fix it.
-   Avoided privileged/seccomp changes; those were not needed.

the clue came from instrumenting the child processes that Emby was spawning. When Emby launched `ffdetect`, the child inherited an unrelated high-numbered fd from the parent EmbyServer process. In my repro it was fd `117`, pointing at an EmbyServer socket. When I launched the same binary manually with `docker exec`, that fd was not there and hardware detection worked. that was the tiny trapdoor in the floor.

the workaround image keeps the vendor binaries as `*.real`, then symlinks `/bin/ffmpeg`, `/bin/ffdetect`, and `/bin/ffprobe` to a tiny launcher:

#!/bin/sh
set -eu

name="${0##*/}"
real="/bin/${name}.real"

for fd_path in /proc/self/fd/*; do
  fd="${fd_path##*/}"
  case "$fd" in
    0|1|2) continue ;;
  esac
  eval "exec ${fd}>&-" 2>/dev/null || true
done

exec "$real" "$@"

after that, Emby's own hardware detection immediately reported the expected QSV and VAAPI codecs. In my case the inventory went from zero hardware codecs to 17 hardware codecs, including QuickSync H.264/H.265 decode and encode plus VAAPI entries. A real browser playback test of a 4K HEVC file then showed hardware decode and hardware encode, and the ffmpeg command line included `-init_hw_device qsv=...`, `hevc_qsv`, `vpp_qsv`, and `h264_qsv`. `intel_gpu_top` also showed Render/3D, Video, and VideoEnhance activity during the stream.

so, question for the team:

1. am I crazy? did I need to do any of this? I could not find another way to get GPU transcoding working from inside Emby, even though the same tools worked manually.
2. is Emby expected to launch the ffmpeg-family tools with arbitrary non-stdio file descriptors inherited from EmbyServer? If not, would the right fix be in the process launcher, making sure only stdin/stdout/stderr are inherited, or that all other fds are marked close-on-exec before spawning `ffdetect`/`ffmpeg`/`ffprobe`?

I am happy to test a build or provide more logs. the wrapper is an okay local bandage, but I would much rather remove it and run the stock image. Also entirely possible there is some very simple knob or obvious docker thing I missed, in which case I would love to be told that too.