VideoToolbox H.264 Hardware Encoding Fails on Apple Silicon (M1 Ultra) - Invalid Profile Parameter

[rockinyp 6]

Environment

• Emby Server Version: 4.9.1.90 (also tested notification for 4.9.3.0 available)
• Operating System: macOS (Darwin 25.2.0 - macOS Sequoia)
• Hardware: Mac Studio M1 Ultra (20 cores, ARM64)
• FFmpeg Version: 5.1-emby_2023_06_25_p4

Problem Description

VideoToolbox H.264 hardware encoding fails immediately on transcode attempts. Emby falls back to software encoding (libx264), which works but uses CPU instead of the hardware encoder.

The root cause appears to be an invalid profile parameter being passed to the VideoToolbox encoder. Emby is sending profile=77 (the FFmpeg internal constant for "Main" profile) when VideoToolbox expects values in the range 0-5.

Settings

• Transcoding → VideoToolbox H.264 → Encoding Profile Limit: High Profile

Error from Logs

[h264_videotoolbox @ 0x145604300] Value 77.000000 for parameter 'profile' out of range [0 - 5]
[h264_videotoolbox @ 0x145604300] Error setting option profile to value 77.
Error initializing output stream 0:0 -- Error while opening encoder for output stream #0:0 - maybe incorrect parameters such as bit_rate, rate, width or height

FFmpeg Command Generated by Emby

/Applications/EmbyServer.app/Contents/MacOS/ffmpeg -loglevel +timing -y \
  -print_graphs_file "/Users/macstudio/emby-server/logs/ffmpeg-transcode-bd87cbd9-f7be-4e12-9f95-e53c30a9d65c_1graph.txt" \
  -copyts -start_at_zero -f mp4 -c:v:0 h264 -noautorotate \
  -i "/Volumes/media/Budgie Course/FLIGHTED/FLIGHTED - Sugar - 4 (Tube).mp4" \
  -filter_complex "[0:0]scale@f1=width=640:height=360[f1_out0]" \
  -map [f1_out0] -map 0:1 -sn \
  -c:v:0 h264_videotoolbox -b:v:0 746626 -g:v:0 72 -maxrate:v:0 746626 \
  -bufsize:v:0 1493252 -profile:v:0 77 -keyint_min:v:0 72 \
  -r:v:0 23.976024627685547 -c:a:0 copy ...

The problematic parameter is -profile:v:0 77.

Expected Behavior

Emby should translate "High Profile" to the correct VideoToolbox profile value. According to FFmpeg's VideoToolbox encoder, valid profile values are:

• 0 = Baseline
• 1 = Main
• 2 = High
• (or similar mapping)

Workaround

Emby automatically falls back to software encoding (libx264), which works correctly. The software encoder achieves ~40x realtime on the M1 Ultra, so playback is not affected. However, this defeats the purpose of having hardware encoding available.

Additional Context

This appears to be specific to Apple Silicon Macs using VideoToolbox. The profile parameter mapping that works for other hardware encoders (like Intel QSV or NVIDIA NVENC) doesn't translate correctly to VideoToolbox's expected values.

Relevant Log Sections

Codec Selection (shows hardware encoder being selected then failing)

>>>>>>  Hardware Encoders for h264
        [X] VideoToolbox H.264

>>>>>>  Selected Codecs
Encoder VideoToolbox H.264
        Max Bitrate: 781 Mbit/s
        Bit Depths: 8
        Profiles: Baseline Profile (Level 6.2), Main Profile (Level 6.2), High Profile (Level 6.2)

>>>>>  Non-Default Encoder Parameters
Warning EncoderParametersH264VideoToolbox.ProfileLimit: Original:  Actual: AvcProfileHigh

Fallback to Software Encoding

Info    Previous transcoding attempt failed. Falling back to software transcoding.

>>>>>>  FindVideoEncoder - MediaType: h264, UseHardwareCodecs: False, HWA-Mode: Advanced
Info    Checking: 'x264'
Info    Check successful - selecting 'x264'

Request

Could the profile parameter mapping for VideoToolbox H.264 be reviewed? The encoder expects different values than what Emby is currently passing.

Thank you!

[Luke 42077]

@rockinyp

 

Hi there, let's look at an example. Please attach the information requested in how to report a media playback issue. Thanks!

 

[BenLovell 0]

I'm having this issue with my M1 Mac Mini as well, however it's creating playback issues as it's an M1 and not an M1 Ultra. I've attached my log file.

ffmpeg-transcode-c26521ed-802c-4eb3-9048-5cfa18f5fb27_1.txt

Edited February 11 by BenLovell

[BenLovell 0]

@Luke@rockinyp

Sorry for the double post. I looked for a way to edit my previous post to tag the two of you. I'm having this issue as well. Please find my logs attached.

ffmpeg-transcode-c26521ed-802c-4eb3-9048-5cfa18f5fb27_1.txt