VideoToolbox H.264 Hardware Encoding Fails on Apple Silicon (M1 Ultra) - Invalid Profile Parameter

[rockinyp 6]

Environment

• Emby Server Version: 4.9.1.90 (also tested notification for 4.9.3.0 available)
• Operating System: macOS (Darwin 25.2.0 - macOS Sequoia)
• Hardware: Mac Studio M1 Ultra (20 cores, ARM64)
• FFmpeg Version: 5.1-emby_2023_06_25_p4

Problem Description

VideoToolbox H.264 hardware encoding fails immediately on transcode attempts. Emby falls back to software encoding (libx264), which works but uses CPU instead of the hardware encoder.

The root cause appears to be an invalid profile parameter being passed to the VideoToolbox encoder. Emby is sending profile=77 (the FFmpeg internal constant for "Main" profile) when VideoToolbox expects values in the range 0-5.

Settings

• Transcoding → VideoToolbox H.264 → Encoding Profile Limit: High Profile

Error from Logs

[h264_videotoolbox @ 0x145604300] Value 77.000000 for parameter 'profile' out of range [0 - 5]
[h264_videotoolbox @ 0x145604300] Error setting option profile to value 77.
Error initializing output stream 0:0 -- Error while opening encoder for output stream #0:0 - maybe incorrect parameters such as bit_rate, rate, width or height

FFmpeg Command Generated by Emby

/Applications/EmbyServer.app/Contents/MacOS/ffmpeg -loglevel +timing -y \
  -print_graphs_file "/Users/macstudio/emby-server/logs/ffmpeg-transcode-bd87cbd9-f7be-4e12-9f95-e53c30a9d65c_1graph.txt" \
  -copyts -start_at_zero -f mp4 -c:v:0 h264 -noautorotate \
  -i "/Volumes/media/Budgie Course/FLIGHTED/FLIGHTED - Sugar - 4 (Tube).mp4" \
  -filter_complex "[0:0]scale@f1=width=640:height=360[f1_out0]" \
  -map [f1_out0] -map 0:1 -sn \
  -c:v:0 h264_videotoolbox -b:v:0 746626 -g:v:0 72 -maxrate:v:0 746626 \
  -bufsize:v:0 1493252 -profile:v:0 77 -keyint_min:v:0 72 \
  -r:v:0 23.976024627685547 -c:a:0 copy ...

The problematic parameter is -profile:v:0 77.

Expected Behavior

Emby should translate "High Profile" to the correct VideoToolbox profile value. According to FFmpeg's VideoToolbox encoder, valid profile values are:

• 0 = Baseline
• 1 = Main
• 2 = High
• (or similar mapping)

Workaround

Emby automatically falls back to software encoding (libx264), which works correctly. The software encoder achieves ~40x realtime on the M1 Ultra, so playback is not affected. However, this defeats the purpose of having hardware encoding available.

Additional Context

This appears to be specific to Apple Silicon Macs using VideoToolbox. The profile parameter mapping that works for other hardware encoders (like Intel QSV or NVIDIA NVENC) doesn't translate correctly to VideoToolbox's expected values.

Relevant Log Sections

Codec Selection (shows hardware encoder being selected then failing)

>>>>>>  Hardware Encoders for h264
        [X] VideoToolbox H.264

>>>>>>  Selected Codecs
Encoder VideoToolbox H.264
        Max Bitrate: 781 Mbit/s
        Bit Depths: 8
        Profiles: Baseline Profile (Level 6.2), Main Profile (Level 6.2), High Profile (Level 6.2)

>>>>>  Non-Default Encoder Parameters
Warning EncoderParametersH264VideoToolbox.ProfileLimit: Original:  Actual: AvcProfileHigh

Fallback to Software Encoding

Info    Previous transcoding attempt failed. Falling back to software transcoding.

>>>>>>  FindVideoEncoder - MediaType: h264, UseHardwareCodecs: False, HWA-Mode: Advanced
Info    Checking: 'x264'
Info    Check successful - selecting 'x264'

Request

Could the profile parameter mapping for VideoToolbox H.264 be reviewed? The encoder expects different values than what Emby is currently passing.

Thank you!

[Luke 42821]

@rockinyp

 

Hi there, let's look at an example. Please attach the information requested in how to report a media playback issue. Thanks!

 

[BenLovell 0]

I'm having this issue with my M1 Mac Mini as well, however it's creating playback issues as it's an M1 and not an M1 Ultra. I've attached my log file.

ffmpeg-transcode-c26521ed-802c-4eb3-9048-5cfa18f5fb27_1.txt

Edited February 11 by BenLovell

[BenLovell 0]

@Luke@rockinyp

Sorry for the double post. I looked for a way to edit my previous post to tag the two of you. I'm having this issue as well. Please find my logs attached.

ffmpeg-transcode-c26521ed-802c-4eb3-9048-5cfa18f5fb27_1.txt

[Luke 42821]

@rockinyp  @BenLovell Have you updated to Emby Server 4.9.5? Are you still able to reproduce this?

[BenLovell 0]

@rockinyp@LukeThanks for following up! Yes. I'm running the latest server version (4.9.5.0) and it's still an issue. It happens pretty much every time transcription is involved. Please see the attached log file.

ffmpeg-transcode-acadc158-b3af-4373-85ae-2a490ac45b9d_1.txt

Edited Tuesday at 07:52 PM by BenLovell

[Luke 42821]

OK we are looking into it. Thanks.

[BenLovell 0]

Thanks! Let me know if there are any solutions you'd like me to try via betas, etc.

[rockinyp 6]

 

@LukeMy Mac Studio server is running Emby Server 4.9.5.0 and I can still reproduce the VideoToolbox H.264 failure.

Environment

• Emby Server: 4.9.5.0
• Hardware: Apple Silicon Mac Studio (M1 Ultra, arm64)
• OS: macOS 26.5.2
• Bundled FFmpeg: 5.1-emby_2023_06_25_p4

Reproduction

1. Play a 480i MPEG-2 DVR recording remotely at 1.808 Mbps, which requires H.264 video transcoding/deinterlacing.
2. Emby first launches FFmpeg with -c:v:0 h264_videotoolbox and -profile:v:0 77.
3. That VideoToolbox transcode exits with code 1.
4. Emby immediately retries with -c:v:0 libx264, and playback continues via software encoding.

This happened on July 29, 2026 at 07:19:05 EDT. The attached full server log and FFmpeg transcode log are downloaded from Emby’s Logs UI/API endpoint for that playback attempt.

Relevant server-log sequence:

07:19:05.149 h264_videotoolbox ... -profile:v:0 77 ... 07:19:05.422 Process exited with code 1 - Failed 07:19:05.430 libx264 ... 07:20:28.694 Process exited with code 0

So yes: H.264 VideoToolbox hardware encoding is still failing in 4.9.5.0 on this M1 Ultra, and Emby is falling back to CPU software encoding (libx264).

embyserver-63920966400.txt ffmpeg-transcode-42d2f5f9-55ef-4465-b729-27f5e85c5d80_1.txt