HLS transcoding for Apple's native player: fMP4 segments, and an HDR + SDR variant pair

[vdatanet 1647]

Hi,

I'm building a tvOS client that uses the native Apple player (AVPlayer/AVFoundation) rather than a bundled software decoder. On Apple TV that's the only path that gets you hardware decoding, Match Dynamic Range and Match Frame Rate — but it also means I'm bound by what AVFoundation accepts, and that's where I've run into a wall.

What I observe

My device profile advertises HLS transcoding with an MP4 container:

{
  "Type": "Video",
  "Protocol": "hls",
  "Container": "mp4",
  "VideoCodec": "hevc,h264",
  "AudioCodec": "aac,ac3,alac,eac3,flac"
}

I then POST to /Items/{id}/PlaybackInfo, follow the returned TranscodingUrl to the master playlist, and fetch the first variant playlist. What comes back is MPEG-2 TS: there is no #EXT-X-MAP:URI="..." initialisation segment, and every media segment URI ends in .ts. The "Container": "mp4" in my transcoding profile doesn't appear to change the segment format.

Why this blocks a native Apple client

Apple's HLS Authoring Specification for Apple Devices is explicit on this point:

The container format for HEVC video MUST be fMP4.

MPEG-2 TS is only sanctioned for H.264. HDR10, HLG and Dolby Vision all ride on HEVC, so they inherit the same restriction. In practice, on an Apple TV 4K, that means:

• HEVC can't be delivered over HLS at all, so everything has to be re-encoded to H.264;
• HDR and Dolby Vision sources get tone-mapped down to SDR;
• and since the player never sees the real video range or frame rate, tvOS can't switch the display — Match Dynamic Range and Match Frame Rate stay inert.

The net effect is that a 4K HDR HEVC file the Apple TV would decode in hardware, untouched, instead becomes a full SDR H.264 transcode: more work for the server, and a visibly worse picture on the TV.

But what about clients that can't handle HDR?

That's the concern I've seen raised whenever HDR delivery comes up, and it's a fair one. The answer is the reason the HLS multivariant playlist exists in the first place, and the authoring specification states it directly:

For backward compatibility, SDR streams MUST be provided.

You don't choose between HDR and SDR — you publish both, described side by side in the master playlist. Each #EXT-X-STREAM-INF carries the attributes a player needs in order to judge it: CODECS, RESOLUTION, FRAME-RATE and VIDEO-RANGE. Every client reads that list and takes only what it can actually decode and display. A player with no HDR support never selects the VIDEO-RANGE=PQ variant, because it can see from the manifest that it shouldn't; it takes the SDR one instead. Nothing has to be decided server-side, no client has to be sniffed or profiled, and no client ends up holding a stream it can't play.

This is also why the description matters as much as the segments do. On Apple platforms the player decides what a stream is before it downloads a single segment: it reads those attributes and uses them both to rule variants in or out and to pre-configure the display. If the HDR variant isn't described in the master playlist, tvOS has nothing to act on and the display-mode switch never fires, however good the segments underneath happen to be.

The request

Two things, and the second is what makes the first safe:

1. fMP4 segments. When a client's transcoding profile asks for HLS with an mp4 container, emit fMP4/CMAF segments instead of MPEG-2 TS — ffmpeg's -hls_segment_type fmp4, which writes an initialisation segment referenced by #EXT-X-MAP:URI="init.mp4" and .m4s media segments. Gating it on the container the client declares, or behind a server setting, leaves today's TS output as the default, so nothing that currently works changes.
2. An HDR + SDR variant pair in the master playlist for HDR and Dolby Vision sources: the HDR variant (VIDEO-RANGE=PQ, carrying the Dolby Vision codec string where it applies) alongside an SDR companion, each fully described. One URL then serves both a 4K HDR television and a device that can't take HDR, with no second trip through PlaybackInfo and no guesswork about the client.

Questions

1. Is any of this already achievable and I've simply missed it — a server setting, or a different property in the device profile?
2. If not, are fMP4 segment output and the HDR/SDR variant pair things you'd consider?

Happy to test against a 4K HDR/DV library and report back with player-side measurements: variant chosen, video range as the player reports it, and whether the display mode switches.

Thanks.

[Luke 42840]

Try asking for m4s as the segment container. you can get fmp4. you can't get both hdr and sdr together in the same manifest.