HLS transcoding for Apple's native player: fMP4 segments, and an HDR + SDR variant pair

[vdatanet 1647]

Hi,

I'm building a tvOS client that uses the native Apple player (AVPlayer/AVFoundation) rather than a bundled software decoder. On Apple TV that's the only path that gets you hardware decoding, Match Dynamic Range and Match Frame Rate — but it also means I'm bound by what AVFoundation accepts, and that's where I've run into a wall.

What I observe

My device profile advertises HLS transcoding with an MP4 container:

{
  "Type": "Video",
  "Protocol": "hls",
  "Container": "mp4",
  "VideoCodec": "hevc,h264",
  "AudioCodec": "aac,ac3,alac,eac3,flac"
}

I then POST to /Items/{id}/PlaybackInfo, follow the returned TranscodingUrl to the master playlist, and fetch the first variant playlist. What comes back is MPEG-2 TS: there is no #EXT-X-MAP:URI="..." initialisation segment, and every media segment URI ends in .ts. The "Container": "mp4" in my transcoding profile doesn't appear to change the segment format.

Why this blocks a native Apple client

Apple's HLS Authoring Specification for Apple Devices is explicit on this point:

The container format for HEVC video MUST be fMP4.

MPEG-2 TS is only sanctioned for H.264. HDR10, HLG and Dolby Vision all ride on HEVC, so they inherit the same restriction. In practice, on an Apple TV 4K, that means:

• HEVC can't be delivered over HLS at all, so everything has to be re-encoded to H.264;
• HDR and Dolby Vision sources get tone-mapped down to SDR;
• and since the player never sees the real video range or frame rate, tvOS can't switch the display — Match Dynamic Range and Match Frame Rate stay inert.

The net effect is that a 4K HDR HEVC file the Apple TV would decode in hardware, untouched, instead becomes a full SDR H.264 transcode: more work for the server, and a visibly worse picture on the TV.

But what about clients that can't handle HDR?

That's the concern I've seen raised whenever HDR delivery comes up, and it's a fair one. The answer is the reason the HLS multivariant playlist exists in the first place, and the authoring specification states it directly:

For backward compatibility, SDR streams MUST be provided.

You don't choose between HDR and SDR — you publish both, described side by side in the master playlist. Each #EXT-X-STREAM-INF carries the attributes a player needs in order to judge it: CODECS, RESOLUTION, FRAME-RATE and VIDEO-RANGE. Every client reads that list and takes only what it can actually decode and display. A player with no HDR support never selects the VIDEO-RANGE=PQ variant, because it can see from the manifest that it shouldn't; it takes the SDR one instead. Nothing has to be decided server-side, no client has to be sniffed or profiled, and no client ends up holding a stream it can't play.

This is also why the description matters as much as the segments do. On Apple platforms the player decides what a stream is before it downloads a single segment: it reads those attributes and uses them both to rule variants in or out and to pre-configure the display. If the HDR variant isn't described in the master playlist, tvOS has nothing to act on and the display-mode switch never fires, however good the segments underneath happen to be.

The request

Two things, and the second is what makes the first safe:

1. fMP4 segments. When a client's transcoding profile asks for HLS with an mp4 container, emit fMP4/CMAF segments instead of MPEG-2 TS — ffmpeg's -hls_segment_type fmp4, which writes an initialisation segment referenced by #EXT-X-MAP:URI="init.mp4" and .m4s media segments. Gating it on the container the client declares, or behind a server setting, leaves today's TS output as the default, so nothing that currently works changes.
2. An HDR + SDR variant pair in the master playlist for HDR and Dolby Vision sources: the HDR variant (VIDEO-RANGE=PQ, carrying the Dolby Vision codec string where it applies) alongside an SDR companion, each fully described. One URL then serves both a 4K HDR television and a device that can't take HDR, with no second trip through PlaybackInfo and no guesswork about the client.

Questions

1. Is any of this already achievable and I've simply missed it — a server setting, or a different property in the device profile?
2. If not, are fMP4 segment output and the HDR/SDR variant pair things you'd consider?

Happy to test against a 4K HDR/DV library and report back with player-side measurements: variant chosen, video range as the player reports it, and whether the display mode switches.

Thanks.

[Luke 42840]

Try asking for m4s as the segment container. you can get fmp4. you can't get both hdr and sdr together in the same manifest.

[vdatanet 1647]

Thanks for the pointer — I went and measured it, and the result is worth reporting, because it changes what I'm asking for.

First, a correction to my own post. I said the transcoding output was MPEG-2 TS with .ts segments. Against Emby 4.9.5.0 that doesn't reproduce. With my existing profile — "Container": "mp4", the one I quoted — the variant playlist already contains #EXT-X-MAP:URI= and the media segments are 0.mp4. That's fMP4. So request #1 in my original post is already satisfied, and I withdraw it. Either something changed since I took that measurement or I misread my own capture; either way the record should be straight.

I did try m4s as you suggested. It works, but as far as I can tell it only changes the file extension — 0.m4s instead of 0.mp4 — with #EXT-X-MAP present in both cases. Same container, different name. If it's meant to do something more than that, I'd be glad to know what to look for.

What actually blocks the native Apple player is the manifest description, not the segments.

Here is the multivariant playlist Emby produces for an HDR10 source (Emby's own metadata for the item reports VideoRange: "HDR 10", ExtendedVideoType: "Hdr10"

#EXT-X-STREAM-INF:BANDWIDTH=26581586,AVERAGE-BANDWIDTH=22151322,RESOLUTION=3840x1600,FRAME-RATE=23.976

There is no VIDEO-RANGE attribute and no CODECS attribute.

This is the part that matters on Apple platforms, and it's independent of what the segments contain. AVPlayer decides what a stream is before it downloads a single segment: it reads the #EXT-X-STREAM-INF attributes to rule variants in or out and to pre-configure the display. With no VIDEO-RANGE=PQ and no codec string to inspect, tvOS has nothing to act on, so Match Dynamic Range never fires and the TV stays in SDR — even if the elementary stream underneath is perfectly good HDR HEVC. CODECS is also what lets a player reject a variant it can't decode without downloading it first; without it, every client has to guess.

For contrast, this is what a correctly described HDR variant looks like — VIDEO-RANGE, a codec string, and Dolby Vision signalled via SUPPLEMENTAL-CODECS:

#EXT-X-STREAM-INF:BANDWIDTH=44081437,VIDEO-RANGE=PQ,CODECS="hvc1.2.4.L150.B0,ac-3",SUPPLEMENTAL-CODECS="dvh1.08.06/db1p",RESOLUTION=3840x2160,FRAME-RATE=23.976

To pre-empt the obvious objection: for a lot of my library Emby decides to burn in subtitles (SubtitleMethod=Encode, TranscodeReasons=SubtitleCodecNotSupported), and a burn-in forces a full re-encode through format=pix_fmts=yuv420p — 8-bit, so HDR is gone regardless of the source. For those items, omitting VIDEO-RANGE would be entirely honest: you shouldn't advertise PQ if what you're sending is SDR.

So I removed the burn-in entirely (it turned out to come from a remembered per-item subtitle selection on my user account, not from anything you do by default) and re-measured. Across the first 10 HDR10 and Dolby Vision items in my library, with no burn-in on any of them — TranscodeReasons=ContainerNotSupported on its own, no subtitle filter in the chain — none gets a VIDEO-RANGE attribute. The attribute isn't missing because the pipeline tone-mapped; it's missing because it never gets written.

The server does receive the information, for what it's worth: with Property: "VideoRange" on the CodecProfile (I had been using Jellyfin's VideoRangeType, which Emby ignores — that one's on me), the transcoding URL comes back carrying hevc-videorange=SDR,DOVIWithSDR,HLG,…,DOVIWithHDR10,DOVI. So Emby knows the client accepts HDR. It just doesn't say anything about it in the manifest.

On "you can't get both HDR and SDR together in the same manifest": understood, and I want to be clear that I'm not asking you to change that. The HLS Authoring Specification does ask for an SDR companion, but a single well-described variant would already solve my problem. If the manifest said VIDEO-RANGE=PQ and carried a CODECS string, an Apple TV 4K would configure the display correctly and a client that can't handle HDR would see, from the manifest alone, that it shouldn't select it. That's a much smaller change than restructuring into a variant pair, and it's the one I actually need.

So the revised request is just this: populate VIDEO-RANGE and CODECS on #EXT-X-STREAM-INF (and SUPPLEMENTAL-CODECS where Dolby Vision applies), reflecting what the stream really is.

And the stream really is HDR. I pulled the #EXT-X-MAP init segment and the first media segment, concatenated them and ran ffprobe — once on an HDR10 title, once on a Dolby Vision one:

HDR10 → hevc (Main) · 3840x1600 · yuv420p · smpte2084 · bt2020 · bt2020nc
DOVI  → hevc (Main) · 3840x2160 · yuv420p · smpte2084 · bt2020 · bt2020nc

smpte2084 is PQ and the primaries are BT.2020 — Emby is sending me HDR right now. It just doesn't say so in the manifest, and by the time the player has a segment in hand it's too late: on Apple platforms the display mode is decided up front, from the attributes. So this isn't a request to change what the transcoder produces. The bytes are already right. It's a request to describe them.

Four questions:

1. Is the absence of VIDEO-RANGE and CODECS deliberate, or just not implemented? Is there a server setting or a device-profile property that turns it on that I've missed?
2. Would you also consider keeping 10-bit? The output is hevc (Main) at yuv420p — 8-bit — while carrying PQ and BT.2020 tags. PQ at 8 bits bands visibly, and HDR10 is defined as 10-bit. The source is Main 10, so the depth is being dropped in the transcode rather than being a limitation of the input. I notice the ffmpeg command line asks for -profile:v:0 main10 while the filter chain ahead of it has already done format=pix_fmts=yuv420p, which looks like the two ends disagreeing.
3. The Dolby Vision title comes out the same as the HDR10 one: PQ and BT.2020, but no RPU or DV configuration record left in the stream. Is DV expected to survive a transcode, or is dropping to HDR10 base the intended behaviour?
4. If the attributes were added, is there anything else on the Emby side that would stop tvOS from switching display mode?

One footnote, in case it's useful to anyone else: this took a while to measure because most of my HDR titles were failing with HTTP 500 — Error starting ffmpeg. That turned out to be a burn-in of a subrip track that my account had remembered per-item, which puts subtitles=filename='<the same 18 GB mkv>' in the filter chain; that filter scans the whole container before emitting a frame, my media sits on a nearly-full ZFS pool doing ~34 MB/s, and the encode timed out. The same command without the subtitle filter produces its first segment in 14 seconds. Entirely my storage's problem, not a bug on your side — but worth knowing that a burn-in on a very large file can fail outright rather than just run slowly.

Happy to test any build against a 4K HDR/DV library and report back with player-side measurements: the variant chosen, the video range as AVPlayer reports it, and whether the display mode actually switches on an Apple TV 4K.

[Luke 42840]

The server does populate video-range and codecs already.