Live TV VAAPI transcode fails at stream start: "Access unit too large"

[bradford 10]

Live TV hardware transcoding fails at stream start. It failed in two of my three live starts. Emby then falls back to software.

Setup:

• Emby 4.10.0.40, linuxserver/emby Docker image, /dev/dri passed through.
• Intel i3-4330 (Haswell, HD 4600), VAAPI, i965 driver 2.4.1.
• Emby's ffmpeg: 5.1-emby_2023_06_25_p6.
• Tuner: HDHomeRun CONNECT (HDHR4-2US), US over-the-air. MPEG-2, 720p60 and 1080i, with closed captions.
• Client: Emby Web on a remote connection. Target was 640x360 at 616 kbps.

What happens:

• Emby picks the VAAPI MPEG-2 decoder and the VAAPI H.264 encoder.
• The decoder drops the first frames: "Invalid frame dimensions 0x0".
• The first encoded frame fails:
  h264_vaapi: Access unit too large: 8192 < 10264
h264_vaapi: Failed to write extra header 0: -28
Error submitting video frame to the encoder
• Emby logs "Previous transcoding attempt failed. Falling back to software transcoding."

The overflow grows with the dropped frames. 43 dropped frames gave 11960 bytes. 36 gave 10264. I think caption data from the dropped frames lands on the first frame.

What works:

• Library transcodes on VAAPI. No fallback.
• Some live TV starts. One stayed on VAAPI at about 25% of a core.
• The same encoder settings on a clean tuner capture. That encode succeeded with and without A53 captions.

Can you look at the first-frame header size on live streams? The full transcode logs attached.

Thanks,
Bradford

ffmpeg-transcode-fe7a4366-448e-4cbc-bd52-53e6c6438c81_1.txt ffmpeg-transcode-d3d074f3-59a7-41d9-a1a8-641b26bdaff0_1.txt ffmpeg-transcode-cf53136b-c9bc-43bb-933c-a0a2dd413997_1.txt ffmpeg-transcode-81c994e2-3772-4806-b10c-b234d3625a2c_1.txt ffmpeg-transcode-508dcfff-c488-478d-8c07-33aecd9aaacf_1.txt ffmpeg-transcode-46aafbce-69b4-4c40-9928-789d6a792169_1.txt