HLS transcoding for Apple's native player: fMP4 segments, and an HDR + SDR variant pair

[vdatanet 1659]

Hi,

I'm building a tvOS client that uses the native Apple player (AVPlayer/AVFoundation) rather than a bundled software decoder. On Apple TV that's the only path that gets you hardware decoding, Match Dynamic Range and Match Frame Rate — but it also means I'm bound by what AVFoundation accepts, and that's where I've run into a wall.

What I observe

My device profile advertises HLS transcoding with an MP4 container:

{
  "Type": "Video",
  "Protocol": "hls",
  "Container": "mp4",
  "VideoCodec": "hevc,h264",
  "AudioCodec": "aac,ac3,alac,eac3,flac"
}

I then POST to /Items/{id}/PlaybackInfo, follow the returned TranscodingUrl to the master playlist, and fetch the first variant playlist. What comes back is MPEG-2 TS: there is no #EXT-X-MAP:URI="..." initialisation segment, and every media segment URI ends in .ts. The "Container": "mp4" in my transcoding profile doesn't appear to change the segment format.

Why this blocks a native Apple client

Apple's HLS Authoring Specification for Apple Devices is explicit on this point:

The container format for HEVC video MUST be fMP4.

MPEG-2 TS is only sanctioned for H.264. HDR10, HLG and Dolby Vision all ride on HEVC, so they inherit the same restriction. In practice, on an Apple TV 4K, that means:

• HEVC can't be delivered over HLS at all, so everything has to be re-encoded to H.264;
• HDR and Dolby Vision sources get tone-mapped down to SDR;
• and since the player never sees the real video range or frame rate, tvOS can't switch the display — Match Dynamic Range and Match Frame Rate stay inert.

The net effect is that a 4K HDR HEVC file the Apple TV would decode in hardware, untouched, instead becomes a full SDR H.264 transcode: more work for the server, and a visibly worse picture on the TV.

But what about clients that can't handle HDR?

That's the concern I've seen raised whenever HDR delivery comes up, and it's a fair one. The answer is the reason the HLS multivariant playlist exists in the first place, and the authoring specification states it directly:

For backward compatibility, SDR streams MUST be provided.

You don't choose between HDR and SDR — you publish both, described side by side in the master playlist. Each #EXT-X-STREAM-INF carries the attributes a player needs in order to judge it: CODECS, RESOLUTION, FRAME-RATE and VIDEO-RANGE. Every client reads that list and takes only what it can actually decode and display. A player with no HDR support never selects the VIDEO-RANGE=PQ variant, because it can see from the manifest that it shouldn't; it takes the SDR one instead. Nothing has to be decided server-side, no client has to be sniffed or profiled, and no client ends up holding a stream it can't play.

This is also why the description matters as much as the segments do. On Apple platforms the player decides what a stream is before it downloads a single segment: it reads those attributes and uses them both to rule variants in or out and to pre-configure the display. If the HDR variant isn't described in the master playlist, tvOS has nothing to act on and the display-mode switch never fires, however good the segments underneath happen to be.

The request

Two things, and the second is what makes the first safe:

1. fMP4 segments. When a client's transcoding profile asks for HLS with an mp4 container, emit fMP4/CMAF segments instead of MPEG-2 TS — ffmpeg's -hls_segment_type fmp4, which writes an initialisation segment referenced by #EXT-X-MAP:URI="init.mp4" and .m4s media segments. Gating it on the container the client declares, or behind a server setting, leaves today's TS output as the default, so nothing that currently works changes.
2. An HDR + SDR variant pair in the master playlist for HDR and Dolby Vision sources: the HDR variant (VIDEO-RANGE=PQ, carrying the Dolby Vision codec string where it applies) alongside an SDR companion, each fully described. One URL then serves both a 4K HDR television and a device that can't take HDR, with no second trip through PlaybackInfo and no guesswork about the client.

Questions

1. Is any of this already achievable and I've simply missed it — a server setting, or a different property in the device profile?
2. If not, are fMP4 segment output and the HDR/SDR variant pair things you'd consider?

Happy to test against a 4K HDR/DV library and report back with player-side measurements: variant chosen, video range as the player reports it, and whether the display mode switches.

Thanks.

[Luke 42938]

Try asking for m4s as the segment container. you can get fmp4. you can't get both hdr and sdr together in the same manifest.

[vdatanet 1659]

Thanks for the pointer — I went and measured it, and the result is worth reporting, because it changes what I'm asking for.

First, a correction to my own post. I said the transcoding output was MPEG-2 TS with .ts segments. Against Emby 4.9.5.0 that doesn't reproduce. With my existing profile — "Container": "mp4", the one I quoted — the variant playlist already contains #EXT-X-MAP:URI= and the media segments are 0.mp4. That's fMP4. So request #1 in my original post is already satisfied, and I withdraw it. Either something changed since I took that measurement or I misread my own capture; either way the record should be straight.

I did try m4s as you suggested. It works, but as far as I can tell it only changes the file extension — 0.m4s instead of 0.mp4 — with #EXT-X-MAP present in both cases. Same container, different name. If it's meant to do something more than that, I'd be glad to know what to look for.

What actually blocks the native Apple player is the manifest description, not the segments.

Here is the multivariant playlist Emby produces for an HDR10 source (Emby's own metadata for the item reports VideoRange: "HDR 10", ExtendedVideoType: "Hdr10"

#EXT-X-STREAM-INF:BANDWIDTH=26581586,AVERAGE-BANDWIDTH=22151322,RESOLUTION=3840x1600,FRAME-RATE=23.976

There is no VIDEO-RANGE attribute and no CODECS attribute.

This is the part that matters on Apple platforms, and it's independent of what the segments contain. AVPlayer decides what a stream is before it downloads a single segment: it reads the #EXT-X-STREAM-INF attributes to rule variants in or out and to pre-configure the display. With no VIDEO-RANGE=PQ and no codec string to inspect, tvOS has nothing to act on, so Match Dynamic Range never fires and the TV stays in SDR — even if the elementary stream underneath is perfectly good HDR HEVC. CODECS is also what lets a player reject a variant it can't decode without downloading it first; without it, every client has to guess.

For contrast, this is what a correctly described HDR variant looks like — VIDEO-RANGE, a codec string, and Dolby Vision signalled via SUPPLEMENTAL-CODECS:

#EXT-X-STREAM-INF:BANDWIDTH=44081437,VIDEO-RANGE=PQ,CODECS="hvc1.2.4.L150.B0,ac-3",SUPPLEMENTAL-CODECS="dvh1.08.06/db1p",RESOLUTION=3840x2160,FRAME-RATE=23.976

To pre-empt the obvious objection: for a lot of my library Emby decides to burn in subtitles (SubtitleMethod=Encode, TranscodeReasons=SubtitleCodecNotSupported), and a burn-in forces a full re-encode through format=pix_fmts=yuv420p — 8-bit, so HDR is gone regardless of the source. For those items, omitting VIDEO-RANGE would be entirely honest: you shouldn't advertise PQ if what you're sending is SDR.

So I removed the burn-in entirely (it turned out to come from a remembered per-item subtitle selection on my user account, not from anything you do by default) and re-measured. Across the first 10 HDR10 and Dolby Vision items in my library, with no burn-in on any of them — TranscodeReasons=ContainerNotSupported on its own, no subtitle filter in the chain — none gets a VIDEO-RANGE attribute. The attribute isn't missing because the pipeline tone-mapped; it's missing because it never gets written.

The server does receive the information, for what it's worth: with Property: "VideoRange" on the CodecProfile (I had been using Jellyfin's VideoRangeType, which Emby ignores — that one's on me), the transcoding URL comes back carrying hevc-videorange=SDR,DOVIWithSDR,HLG,…,DOVIWithHDR10,DOVI. So Emby knows the client accepts HDR. It just doesn't say anything about it in the manifest.

On "you can't get both HDR and SDR together in the same manifest": understood, and I want to be clear that I'm not asking you to change that. The HLS Authoring Specification does ask for an SDR companion, but a single well-described variant would already solve my problem. If the manifest said VIDEO-RANGE=PQ and carried a CODECS string, an Apple TV 4K would configure the display correctly and a client that can't handle HDR would see, from the manifest alone, that it shouldn't select it. That's a much smaller change than restructuring into a variant pair, and it's the one I actually need.

So the revised request is just this: populate VIDEO-RANGE and CODECS on #EXT-X-STREAM-INF (and SUPPLEMENTAL-CODECS where Dolby Vision applies), reflecting what the stream really is.

And the stream really is HDR. I pulled the #EXT-X-MAP init segment and the first media segment, concatenated them and ran ffprobe — once on an HDR10 title, once on a Dolby Vision one:

HDR10 → hevc (Main) · 3840x1600 · yuv420p · smpte2084 · bt2020 · bt2020nc
DOVI  → hevc (Main) · 3840x2160 · yuv420p · smpte2084 · bt2020 · bt2020nc

smpte2084 is PQ and the primaries are BT.2020 — Emby is sending me HDR right now. It just doesn't say so in the manifest, and by the time the player has a segment in hand it's too late: on Apple platforms the display mode is decided up front, from the attributes. So this isn't a request to change what the transcoder produces. The bytes are already right. It's a request to describe them.

Four questions:

1. Is the absence of VIDEO-RANGE and CODECS deliberate, or just not implemented? Is there a server setting or a device-profile property that turns it on that I've missed?
2. Would you also consider keeping 10-bit? The output is hevc (Main) at yuv420p — 8-bit — while carrying PQ and BT.2020 tags. PQ at 8 bits bands visibly, and HDR10 is defined as 10-bit. The source is Main 10, so the depth is being dropped in the transcode rather than being a limitation of the input. I notice the ffmpeg command line asks for -profile:v:0 main10 while the filter chain ahead of it has already done format=pix_fmts=yuv420p, which looks like the two ends disagreeing.
3. The Dolby Vision title comes out the same as the HDR10 one: PQ and BT.2020, but no RPU or DV configuration record left in the stream. Is DV expected to survive a transcode, or is dropping to HDR10 base the intended behaviour?
4. If the attributes were added, is there anything else on the Emby side that would stop tvOS from switching display mode?

One footnote, in case it's useful to anyone else: this took a while to measure because most of my HDR titles were failing with HTTP 500 — Error starting ffmpeg. That turned out to be a burn-in of a subrip track that my account had remembered per-item, which puts subtitles=filename='<the same 18 GB mkv>' in the filter chain; that filter scans the whole container before emitting a frame, my media sits on a nearly-full ZFS pool doing ~34 MB/s, and the encode timed out. The same command without the subtitle filter produces its first segment in 14 seconds. Entirely my storage's problem, not a bug on your side — but worth knowing that a burn-in on a very large file can fail outright rather than just run slowly.

Happy to test any build against a 4K HDR/DV library and report back with player-side measurements: the variant chosen, the video range as AVPlayer reports it, and whether the display mode actually switches on an Apple TV 4K.

[Luke 42938]

The server does populate video-range and codecs already.

[vdatanet 1659]

You're right — the server does populate it, and I can now show you exactly why it never reached me. There are two separate problems, one of them mine.

Mine first. My CodecProfile was written with Jellyfin's range enum names (HDR10, DOVIWithHDR10…). Emby compares against the literal VideoRange string on the stream, which for these files is "HDR 10" — with a space. Your log says so plainly:

Profile: VideoCodecProfile, DirectPlay=false. Reason=Atrium tvOS.VideoRange
Condition: EqualsAny. ConditionValue: SDR|DOVIWithSDR|HLG|…|HDR10|…|DOVI

So the server correctly concluded this client can't take HDR, and went off to tone-map (tonemap_opencl=tonemap=mobius). Omitting VIDEO-RANGE in that situation is the right thing to do, and my earlier post was wrong to read it as a missing attribute. Apologies.

Now the part that isn't mine. When I fix the profile to "SDR|HDR 10|DolbyVision", PlaybackInfo accepts it — it even offers direct play for the title. But the TranscodingUrl it hands back has lost the space:

Sent in the profile:  Value = "SDR|HDR 10|DolbyVision"
Came back in the URL: hevc-videorange=SDR,HDR10,DolbyVision

And the manifest generator compares that against "HDR 10" again, so it fails a second time and tone-maps anyway. The two halves of the same request disagree with each other.

Putting the space back by hand is enough to prove it. Requesting the same master playlist with hevc-videorange=SDR,HDR%2010,DolbyVision:

#EXT-X-STREAM-INF:BANDWIDTH=26581586,AVERAGE-BANDWIDTH=22151322,VIDEO-RANGE=PQ,RESOLUTION=3840x1600,FRAME-RATE=23.976

There it is. VIDEO-RANGE=PQ, exactly as you said. So this is a serialisation bug, not a missing feature — and from a client's point of view it's unreachable, because the value that passes the profile check is the one the URL builder mangles.

Two things still don't work even once the attribute appears:

1. The segment endpoint throws. With the space present in the URL, /videos/{id}/hls1/main/init.mp4 returns HTTP 400 with an unhandled System.ArgumentException: The index is equal to or greater than the length of the array, or the number of elements in the dictionary is greater than the available space from index to the end of the destination array. So the correct value breaks the other end of the same round trip. That's why no client can work around this on its own.
2. CODECS is still absent from the #EXT-X-STREAM-INF line even when VIDEO-RANGE=PQ is there. That one matters for a different reason: it's what lets a player rule a variant in or out before downloading anything, and on Apple platforms it feeds the initial decision alongside the range.

Environment, in case it's version-specific: Emby 4.9.5.0 on Linux from the .deb, source is HEVC Main 10, yuv420p10le, bt2020nc/bt2020/smpte2084, in an MKV (so the container forces the HLS path rather than direct play).

One last observation while I'm here, since it's the root of my original mistake: the ProfileConditionValue enum only exposes VideoRange, so a client has to match on a free-form string whose exact spelling — spaces included — isn't documented anywhere I could find. Meanwhile the stream itself carries ExtendedVideoType (Hdr10, Hdr10Plus, HyperLogGamma, DolbyVision) and ExtendedVideoSubType, which are properly typed but can't be used in a profile condition. Being able to condition on those would have made this whole class of bug impossible.

Happy to re-test any of this against a fix.

[AJV20 69]

is there a testflight for this new app?

[vdatanet 1659]

5 hours ago, AJV20 said:

is there a testflight for this new app?

Thanks for the interest — genuinely glad to hear it.

Where things stand, honestly: Atrium is on the App Store today and it's free, but right now it speaks Jellyfin only. Version 1.1 is with Apple for review as I write this, and Emby support is planned for 1.2, which I've already started work on.

I can't give you a date, and I'd rather not invent one. 1.2 doesn't have a frozen scope yet, and on top of that every release goes through Apple's review queue, which isn't mine to control.

One thing I'd rather say up front than have you discover it: in the first Emby release, HDR content will be tone-mapped by the server instead of played as HDR. That isn't the app being lazy — Emby's HLS manifest doesn't carry the VIDEO-RANGE and CODECS attributes that Apple's player reads to switch the TV into HDR mode, so letting the server map it down is the option that actually looks correct on screen. Everything else is meant to behave as it does on Jellyfin. If that changes on Emby's side, the app is built to take advantage of it the same day.

https://apps.apple.com/app/id6777638751

Edited Sunday at 09:48 AM by vdatanet

[AJV20 69]

17 hours ago, vdatanet said:

Thanks for the interest — genuinely glad to hear it.

Where things stand, honestly: Atrium is on the App Store today and it's free, but right now it speaks Jellyfin only. Version 1.1 is with Apple for review as I write this, and Emby support is planned for 1.2, which I've already started work on.

I can't give you a date, and I'd rather not invent one. 1.2 doesn't have a frozen scope yet, and on top of that every release goes through Apple's review queue, which isn't mine to control.

One thing I'd rather say up front than have you discover it: in the first Emby release, HDR content will be tone-mapped by the server instead of played as HDR. That isn't the app being lazy — Emby's HLS manifest doesn't carry the VIDEO-RANGE and CODECS attributes that Apple's player reads to switch the TV into HDR mode, so letting the server map it down is the option that actually looks correct on screen. Everything else is meant to behave as it does on Jellyfin. If that changes on Emby's side, the app is built to take advantage of it the same day.

https://apps.apple.com/app/id6777638751

Any plans for iOS support?

[vdatanet 1659]

11 minutes ago, AJV20 said:

Any plans for iOS support?

No plans for iOS at the moment — Apple TV is my only target for now. The app is built specifically around the tvOS experience (focus engine, remote, big-screen playback), so an iOS version would be a separate project rather than a quick port. Thanks for asking!

[abat119 13]

Just wanted to say a huge thanks for this! I only ever use the native player on my Apple TV and the emby implementation - while pretty good - is missing a few things when it comes to 4K playback.

Regarding iOS playback - I use the safari web interface and that uses Apple's native player on iPad / iPhone - so that works and looks great. No need for a seperate app - keep focus on making this the best Native Apple TV Emby client 

(Emby team - maybe implement this app into the mainstream one? With blessings from the author of course  )

Edited 2 hours ago by abat119

[Geordie 14]

@vdatanet

Have you ever thought about using the open source AetherEngine?

It does on-device remuxing to specs-complaint Apple-friendly HLS and uses the platform-native AVPlayer with all the privileges that entails.

https://github.com/superuser404notfound/AetherEngine

That said, Emby’s HLS implementation still needs some minor improvements (it’s way better than when i last tried it years ago). Any improvements would be welcome.

 

[vdatanet 1659]

17 minutes ago, Geordie said:

Have you ever thought about using the open source AetherEngine?

Thanks for the pointer — I hadn't come across this one, and I went and read it properly, because I
wanted it to be the answer.

For my case it isn't, and I'd rather be precise about why than vague. Its README is explicit that a
non-live remote .m3u8 handed to the loopback path gets rerouted onto the AVPlayer bypass: the
bundled FFmpeg is built without network support, so the playlist can't be demuxed locally, and as
they put it, remote HLS is AVPlayer's native domain anyway. That's a reasonable call rather than an
oversight — but it means that pointed at an Emby transcoding manifest, the engine hands the same
attribute-less #EXT-X-STREAM-INF to the same AVPlayer I'm already using, and the display stays in
SDR for the same reason. The on-device remux is real; it applies when the source is something FFmpeg
can open, not when it's already a remote HLS playlist.

Where it would change things is by a route I hadn't considered, so the pointer was worth it anyway:
skipping the HLS path altogether and direct-playing the original file.

Once the CodecProfile uses the range spelling Emby actually compares against — the literal "HDR 10"
from the MediaStream, with the space, on Property: "VideoRange" — Emby stops deciding to tone-map
and offers direct play on these titles. I couldn't use that, because AVPlayer won't open an MKV. An
engine that demuxes with FFmpeg and decodes with VideoToolbox can. So the value would be as a way
around the manifest rather than through it, with the HDR coming from the original file untouched:
10-bit, and the Dolby Vision RPU still in place. I've written that down as a real option.

It isn't a small one, to be fair to it. LGPL-3.0 with an App Store exception — well drafted, and it
does cover the store's signing and re-linking restrictions, but only for the binary; the source
obligations stay. Bundling FFmpeg. And on that path subtitle rendering moves out of the native tvOS
picker and into the app's own overlay, which is a chunk of work I've already done once. It's a
question for a later version.

It's also why the two attributes are still what I'd rather have. VIDEO-RANGE and CODECS exist
precisely so a client doesn't have to carry its own demuxer to find out what it's being sent — any
client using the platform player benefits, and only the ones shipping FFmpeg can route around their
absence.

On "Emby's HLS implementation still needs some minor improvements" — which ones did you hit? The two
I ran into are that the range value loses its space somewhere between PlaybackInfo and the
transcoding URL, so the manifest generator never matches it; and that putting the space back by hand
does produce VIDEO-RANGE=PQ, but then the segment endpoint answers 400. I owe this thread the full
measurements on both and will post them. If you hit the same things from a different pipeline, or
something else entirely, I'd like to know — you've clearly looked at this more recently than most.

[Geordie 14]

Why would you point it at Emby’s manifest though?

The engine will ingest media files directly (mkv, avi, ts etc) so as far as the server is concerned, you’ll be direct playing via http. 

The engine will do what the server does (mkv to fMP4 + HLS to AVPlayer) directly on device and where not possible due to an older or unsupported codec, it will use ffmpeg software decode. 

Also, AVPlayer does DV to HDR to SDR tone mapping natively during playback. You don’t need to transcode to play DV on an SDR monitor (just need to skip the master playlist and play the media playlist directly instead).

Lots of apps already use AetherEngine like Moonfin, Plezy, Silo, Sodalite… etc. 

I use Moonfin for Emby and Sodalite for Jellyfin.

I do hope the web client gets better with HLS though as I prefer it to third party apps. It’s more power efficient on mobile.

 

Edited 1 hour ago by Geordie

[vdatanet 1659]

You were right about the route, and I checked rather than take it on trust — Moonfin direct plays
these files and the device does the remux, exactly as you described. The approach works.

It isn't the one I want for this app, though, and that's a goal of mine rather than a verdict on
the engine. What I'm building leans deliberately on two things: the native tvOS player, and the
server's own stream. No bundled engine, no external dependencies. That's a line I've drawn for
myself, and it's why the app behaves the way it does — the system track picker, the system chapter
menu, all of it comes free as long as I stay on the platform player.

That combination already gives me real HDR wherever the manifest describes the stream. Here it
doesn't yet, and that's the whole of the gap: VIDEO-RANGE and CODECS on #EXT-X-STREAM-INF, and the
player handles the rest by itself.

On the Dolby Vision point — playing the media playlist directly is the one thing I can't do, because
the master is where the display switch comes from. Skip it and the TV never leaves SDR, whatever the
segments contain.

Still curious which HLS problems you ran into, if you get a moment.