Auto-Selection of different Versions?

[Lessaj 467]

My browser doesn't even support HEVC and yet always picks the 4k HDR in HEVC rather than the 1080p in H264.

[softworkz 5068]

16 minutes ago, Lessaj said:

My browser doesn't even support HEVC and yet always picks the 4k HDR in HEVC rather than the 1080p in H264.

Which one is that?

What result do you get when visiting this page? https://cconcolato.github.io/media-mime-support/#other_video_codecs

AFAIR, we're using the same method for checking.

[Lessaj 467]

15 minutes ago, softworkz said:

Which one is that?

What result do you get when visiting this page? https://cconcolato.github.io/media-mime-support/#other_video_codecs

AFAIR, we're using the same method for checking.

Chromium 125.0.6422.60 on Linux. These are the only HEVC entries on that page.

I recall seeing this page linked before as well https://ott.dolby.com/codec_test/index.html

[softworkz 5068]

@Lessaj- you say that you see the server deliver as HEVC even though your browser doesn't support it. What happens in that case? Playback fails?

[Lessaj 467]

Just now, softworkz said:

@Lessaj- you say that you see the server deliver as HEVC even though your browser doesn't support it. What happens in that case? Playback fails?

It transcodes and plays successfully in that case, which isn't an issue since I have the power to handle it and I'm watching within my local network 95% of the time, but certainly for my less tech-savvy friends who just press play not realizing it auto selected the 4k (which could be a large remux) and whom I know also do not have a 4K HDR display it makes a lot more sense for it have selected the 1080p version that would have likely direct played or at most remuxed the audio.

[softworkz 5068]

1 minute ago, Lessaj said:

It transcodes and plays successfully in that case, which isn't an issue since I have the power to handle it 

Wait - you said that it would remux or directplay HEVC even though your browser doesn't support it, no?

[Lessaj 467]

3 minutes ago, softworkz said:

Wait - you said that it would remux or directplay HEVC even though your browser doesn't support it, no?

It would transcode the HEVC file. The 1080p version is generally h264, so that would direct play or remux the audio. In this example when I navigated to the episode it automatically selected the 4k HDR which required a transcode to play, the 1080p just remuxes the audio.

Edited May 31, 2024 by Lessaj

[softworkz 5068]

@Lessaj- So your issue is about cases where you have multiple versions of the same library item and you want to have a different version chosen automatically for playback, right?

@Lukemay correct me if I'm wrong, but I'm not aware that we would have any automatism for choosing different versions for playback. That's a manual choice one can make, but I don't see any reasonable way to make automatic choices in this regard because support for multiple versions of an item is a multi-purpose feature which is being used in many more ways than just for providing different qualities. It can be also used for having versions with different content like "director's cut vs. theatre", "original vs. digitally restored/remastered", "rated vs. uncut" or whatever a user would think of or want to use it for. That's something that Emby cannot know about and the reason why Emby cannot reasonable/reliably make any automatic selection between versions.

[Lessaj 467]

2 minutes ago, softworkz said:

@Lessaj- So your issue is about cases where you have multiple versions of the same library item and you want to have a different version chosen automatically for playback, right?

Yes but I wouldn't word it specifically like that, I would word it as I would like it to evaluate the capabilities of the client to pick the version that is most likely to require the least amount of work to play back, direct playing is likely preferred in general. At the end of the day as the end user I still have the ability to pick a different version to play and it will do what it needs to play it back but let's say for example I'm not on the item detail page, just on a movie library page or a TV show season list. I don't know until I've gone into the item that multiple versions are grouped together, if I just hit play from one of those screens (not the item detail page) it's going to (most likely) play back the highest bitrate item, even though it would end up requiring a transcode to do that, when there's another version that could be direct played or remuxed. I understand that this is likely challenging to achieve so I do remind my friends, especially if I see them playing back a 4K HDR that's transcoding/tone mapping to check before hitting play but again, not the most tech savvy of folks.   I have plenty of bandwidth and transcoding power to handle it, but not everyone will.

[softworkz 5068]

@LessajI have split out this into a separate topic because it's unrelated and confusing inside the original one.

[softworkz 5068]

5 minutes ago, Lessaj said:

Yes but I wouldn't word it specifically like that, I would word it as I would like it to evaluate the capabilities of the client to pick the version that is most likely to require the least amount of work to play back, direct playing is likely preferred in general. At the end of the day as the end user I still have the ability to pick a different version to play and it will do what it needs to play it back but let's say for example I'm not on the item detail page, just on a movie library page or a TV show season list. I don't know until I've gone into the item that multiple versions are grouped together, if I just hit play from one of those screens (not the item detail page) it's going to (most likely) play back the highest bitrate item, even though it would end up requiring a transcode to do that, when there's another version that could be direct played or remuxed. I understand that this is likely challenging to achieve so I do remind my friends, especially if I see them playing back a 4K HDR that's transcoding/tone mapping to check before hitting play but again, not the most tech savvy of folks.   I have plenty of bandwidth and transcoding power to handle it, but not everyone will.

I see that this makes sense as a use case, but unless we have an ability to mark alternate versions of an item as being like "same content just different encoding", we cannot start to do auto-selections of "item versions". That's just a logical problem, not a technical one. 

Technically it's still challenging because it would require us to assess "what would need to happen to play version a, b or c" by applying some kind of "weights" to each required processing and then choosing the most lightweight one. Not always trivial but doable by theory.

Yet, what stands in the way is the multi-purpose nature of the multi-version feature as mentioned.

[Lessaj 467]

I understand, and I can live with it in its current iteration because I have the horse power to throw at it and I've informed my friends that it will do this so they should check first before hitting play. I also inform them to use theater on their computers as it's more likely to direct play compared to their browser, which saves both of us some bandwidth if HEVC transcoding is required. 99% of the time when they have told me they have trouble playing it's been on their end, their connection isn't fast enough. I regularly review my reverse proxy logs to evaluate for performance issues and I can see in some cases that delivery of TS segments can exceed 3 seconds for several requests in a row so they are likely experiencing buffering and not telling me about it but that's on them for not checking the version first.  I run a double reverse proxy (within my local network as well as on a geographically close VPS) so I can evaluate the different times reported to deliver a segment by the emby server, the internal reverse proxy, and the external reverse proxy to narrow that down and slap them on the wrist.

The main benefit of auto selecting a version that is more likely to direct play is just to save network bandwidth for remote users really, a 65 Mbit HEVC can easily be 100 Mbit H264 which just about every local network should have no issue playing barring slow wifi speeds so for local users it doesn't really matter too much, and you can't necessarily enforce it for remote users with bitrate caps because you could have a high bitrate 1080p H264 that would play fine without needing to be transcoded at all. While I am curious if even a very basic "can this device direct play the video track in this version" could be implemented in the case of a multi versioned item where as I showed earlier my browser is simply not capable of direct playing content with that encoding but it can handle the other version, once you go beyond 2 items it would likely require some kind of weighting system as you mentioned to be able to solve that. I don't think a UI to pick a version before playing if not on an item page is a solution, it would likely be clunky to show enough information to make the determination but would also be really inconvenient when playing back a show and having to pick the version on every episode.

In a few weeks I'll be traveling from NA to Europe which will be a great opportunity to test my remote capabilities, I can't rely on my friends to give me useful information or understand how to use the tools I would use to evaluate performance.

[softworkz 5068]

8 minutes ago, Lessaj said:

a 65 Mbit HEVC can easily be 100 Mbit H264

Do you still see this happening? I thought we had fixed this from happening some time ago (like a year)...

[softworkz 5068]

13 minutes ago, Lessaj said:

I can see in some cases that delivery of TS segments can exceed 3 seconds for several requests in a row so they are likely experiencing buffering and not telling me about it

It depends on how far their point of playback is away from the download edge and whether segment downloads are strictly sequential or overlapping.

[Lessaj 467]

18 minutes ago, softworkz said:

Do you still see this happening? I thought we had fixed this from happening some time ago (like a year)...

Yes I believe so, this is what is reported on my dashboard and within the user sessions page. TS chunks are averaging probably 38-40 MB so for a movie that's 107 minutes (6420 seconds) that's 2140 chunks which works out to 85.6GB but the original file is 50.4GB.

EDIT: Added ffmpeg transcode log as well.

 

ffmpeg-transcode-6c207ad7-ce84-4211-af28-507ed6388c2d_1.txt

Edited June 1, 2024 by Lessaj

[Lessaj 467]

1 minute ago, softworkz said:

It depends on how far their point of playback is away from the download edge and whether segment downloads are strictly sequential or overlapping.

Yes I have trouble evaluating this sometimes but if I see a few chunks take say 8 seconds and then it goes back to 2 or so they're probably fine, and sometimes chunks are really small if there's a dimly lit scene or a black transition so it's possible for it to catch up.

[Happy2Play 9781]

13 minutes ago, softworkz said:

23 minutes ago, Lessaj said:

a 65 Mbit HEVC can easily be 100 Mbit H264

Do you still see this happening? I thought we had fixed this from happening some time ago (like a year)...

Not as much as it used to as it is not doubling bitrate if possible.

[softworkz 5068]

12 minutes ago, Lessaj said:

Yes I believe so, this is what is reported on my dashboard and within the user sessions page. TS chunks are averaging probably 38-40 MB so for a movie that's 107 minutes (6420 seconds) that's 2140 chunks which works out to 85.6GB but the original file is 50.4GB.

101 Mbps for a H264 HLS stream is definitely not healthy, not even for 4k. 

I think there's still some crude logic which assumes that it takes double (earlier) or 1.5-times the bandwidth for an H264 stream to match the quality of an HEVC stream, but this doesn't take into account the fact that HEVC is 10but whereas H264 output is not HDR and neither the plane packing which may be 444 or 422 while it's always 420 for H264 (making it impossible to match HEVC not matter how much bandwidth you allow).

That needs adjustment. @Luke

Edited June 1, 2024 by softworkz

[softworkz 5068]

8 minutes ago, softworkz said:

That needs adjustment. @Luke

More specifically: A formula rather than just a factor change . Besides the implications I mentioned above, it also needs discrimination between high and low bandwidth orders.

Edited June 1, 2024 by softworkz

[visproduction 315]

This is a tricky problem.

• Top commercial sites might use Adaptive Bitrate Streaming (ABR), that requires all media to be multiplied to different bitrate versions.  This takes extra encoding time and uses a lot of storage.
• What's the ideal Emby user experience to help fix this bandwidth mismatch?

             1)  Auto test for user bandwidth and in some way mark media that will probably not work with the tested limited bandwidth.  Perhaps a pop-up alert to warn the user that media playback may fail, because of a slow connection.  This limit bandwidth switch can be reset by the user with a manual retest, whenever the user thinks the bandwidth should have improved.  This test will probably never be perfect.  There are too many variables with remote connections and server encoding settings.  Finally, if Emby adds this test and switch, where to put it all to be user friendly?

             2) Give the user an easy on/off filter button option to only find media that would work with their current connection bandwidth or an easy filter switch to find only .mp4. aac media and other acceptable .mkv media resolutions at 720P or 480P or below some maximum bitrate.

             3) If any media has multiple bit-rate versions, make the acceptable bitrate be the default playback when the bandwidth tests slow.  I think this might already be working with Emby.  It seems that remote user request with limited Wifi speeds already seems to pick a 720P version if there is also a high bitrate 1080P version.  Does someone know if this is already part of Emby?

 Hope that helps.
===

ABR info of interest: 
 https://cloudinary.com/blog/video_optimization_part_ii_adaptive_bitrate_streaming_of_multiple_codecs
 https://demo.cloudinary.com/video-player/
 https://imagekit.io/blog/adaptive-bitrate-streaming/
 https://blog.vidizmo.com/the-ultimate-guide-to-video-bitrate-for-streaming#Is_There_an_Optimum_Video_Bitrate_for_Streaming?

PDF studies
 https://arxiv.org/pdf/2104.01104

PDF large server solution to test and auto switch bitrates.  Not practical for Emby but shows the complexity to resolve this problem with code examples.
 https://www.akamai.com/site/en/documents/research-paper/improving-bitrate-adaptation-in-the-dash-reference-player.pdf

Edited June 1, 2024 by visproduction

[softworkz 5068]

@visproductionYou fell for a misconception regarding the way Emby is working. Bandwidth is just one of some dozen different factors based on which Emby is determining the best way for delivering a media file to a specific client app, according to the device capabilities, client app settings and server settings.

Due to this high adaptability and configurability, it cannot work like a streaming service with a range of pre-transcoded versions for each item.

[crusher11 1101]

On 6/1/2024 at 7:09 AM, softworkz said:

Luke may correct me if I'm wrong, but I'm not aware that we would have any automatism for choosing different versions for playback.

It's been said repeatedly that Emby selects the highest-quality version that will direct play, or the highest-quality version overall if none will direct play, because the multi-version feature is only intended to be used for different qualities, not different cuts. Comes up every time someone using it for cuts wants to be able to select the best cut as the default.

Even as it is this is flawed, because it will give itself a bigger transcoding job if all versions require a transcode.

[Lessaj 467]

2 hours ago, crusher11 said:

It's been said repeatedly that Emby selects the highest-quality version that will direct play, or the highest-quality version overall if none will direct play, because the multi-version feature is only intended to be used for different qualities, not different cuts. Comes up every time someone using it for cuts wants to be able to select the best cut as the default.

Even as it is this is flawed, because it will give itself a bigger transcoding job if all versions require a transcode.

If that is true then that might be why it's auto selecting the highest bitrate version because my browser still requires the audio to be remuxed on the 1080p version, so both versions are not able to be direct played anyway. I know it can direct play the 1080p video stream but without a weighting system in place it's not able to make the determination that it's the least amount of work to playback.

However as discovered in this thread the H264 conversion bitrate is still higher than it should be for this content, and I think adjusting that would at least help towards reducing the network bandwidth, whether that be local or remote.