Adaptive streaming

[ebr 14904]

4 hours ago, neik said:

Or transcoding high bitrate stuff where the server struggles to keep up

Actually, I think implementing a real-time adaptive transcoding algorithm/process would increase the necessary resources at the server, not lessen them.  But, this is still a good request for unpredictable bandwidth situations.

[kikinjo 162]

This works on plex for serveral years already.

[Spaceboy 2493]

but isnt that because everything streamed on plex is via plex. Which model i don't think many of us have any interest in. We're here because we want to avoid that breach of privacy

[neik 835]

On 1/4/2022 at 5:55 PM, Spaceboy said:

but isnt that because everything streamed on plex is via plex. Which model i don't think many of us have any interest in. We're here because we want to avoid that breach of privacy

Definitely, I don't want to go through their servers but I don't think that is a requirement for this (but also no developer here).
Considering all the continues improvements @softworkz (btw. great work there, looking forward to the subs hw acceleration) is doing to HW transcoding this becomes even more interesting implement.

Don't know how it works with Plex but on Emby whenever you change the quality it stops, starts a new transcode and after (HW & file dependent) a little bit of time it moves on.
With this implemented I would think of Emby handle these changes like Youtube where playback doesn't get stopped when changing resolution.

I would like to see this, even if it's optional for the user but I also know this is still far ahead in the future (unfortunately).

[softworkz 3326]

4 hours ago, neik said:

I would like to see this, even if it's optional for the user but I also know this is still far ahead in the future (unfortunately).

It wouldn't be that far away when we would do it in the "normal" way, like it is intended to be done (when live streaming to many clients).

That would mean that you simply do transcoding with multiple outputs of different quality. But as @ebr mentioned: this would bind a huge amount of server resources.

Assume, you have a 4k file and the client requests 1080 and subtitle burn-in. Now let's say, we want to deliver three qualities for adaptive streaming.
We might then transcode to a 720, a 1080 and a 4k output. Given the fact that we have a 4k output, we need to do all video processing in 4k (e.g. tone mapping, subtitle overlay) as the primary output, and the scale down for the lower quality outputs at the end of processing. 
Normally, when transcoding for just a single output quality, we would doe the scaling to the target size first, and do all processing afterwards only.

That means in total, that it's not just the three simultaneous encodings that add to the whole picture - it also means a huge increase in CPU/GPU usage - in most cases, for nothing (you watch transcoded to 1080 while the server is creating 4k and 720 versions that might never get used.

The next consideration here would be to possibly generate the alternate quality segments on request only. This would be possible,  but there's one caveat:
For multiple quality renditions of a certain video, there are some pretty strict uniformity requirements on the outputs. Those are easy to fulfill when using a triple-output transcode, but it quickly gets difficult when you want to create the different outputs separately.

The biggest drawback though, is that in many cases, you can't mix it up with the original, so the original file can't be one of the qualities that are offered, because it doesn't fulfill the required criteria of similarities with the others. And that's finally making me doubt, whether HLS renditions is the best method to handle this.

The alternative would be to do hard switches, similar as when changing quality, but this would need a tight reduction of the restart delay time first (accidentally we got a few things in the lab for this..). 

There are a number of HLS improvements that need to come before this, though. Then we'll revisited the subject.

[rbjtech 4223]

The on-demand transcode stream sounds interesting.  Manually flipping transcode resolutions today I'm only seeing maybe a 10th of a second pause between the changes - so if a 2nd stream is 'ready and waiting' then maybe that would be acceptable - it's certainly better than the stream stalling due to lack of bandwidth in any case.

 

[softworkz 3326]

2 hours ago, rbjtech said:

Manually flipping transcode resolutions today I'm only seeing maybe a 10th of a second pause between the changes

What do you mean by that - where do you "flip" resolutions and how?

[rbjtech 4223]

9 hours ago, softworkz said:

What do you mean by that - where do you "flip" resolutions and how?

Manually on the client - in my case the web browser.    I just pick a resolution (up or down) and it continues after the very brief pause.

[softworkz 3326]

17 minutes ago, rbjtech said:

Manually on the client - in my case the web browser.    I just pick a resolution (up or down) and it continues after the very brief pause.

With a transcoding restart, creating a new ffmpeg log?

[rbjtech 4223]

49 minutes ago, softworkz said:

With a transcoding restart, creating a new ffmpeg log?

yes - I've sent you all the logs in a zip via PM - manually going up in quality and then back down again.  Going down causes a double pause.

[softworkz 3326]

Thanks for the logs. I have about 500-800ms time for switching with a similar video. Are you sure you get 100ms? That would be really fast  

[rbjtech 4223]

10 minutes ago, softworkz said:

Thanks for the logs. I have about 500-800ms time for switching with a similar video. Are you sure you get 100ms? That would be really fast  

Hmm.. looking at the logs that does appear to show it being around the 450-500ms mark - but it seems faster than that for some of the transitions - I have no way of timing it (other than the logs) but it's definitely quicker than 500ms - that is a long pause that would be very noticeable in video terms.  Possibly 200-300ms - maybe 100ms was a bit optimistic

 

btw - This is transcoding to a Samsung 980 Pro NVME using an Intel 12700K and UHD770 iGPU hardware - so it's probably near the top of the hardware stack with regards to expected transcoding performance..?

[softworkz 3326]

9 minutes ago, rbjtech said:

btw - This is transcoding to a Samsung 980 Pro NVME using an Intel 12700K and UHD770 iGPU hardware - so it's probably near the top of the hardware stack with regards to expected transcoding performance..?

Crucial CT1000P2SSD8, 11700 - that's a bit slower than yours, but when you say 400 or 500 then it matches anyway.

19 minutes ago, rbjtech said:

I have no way of timing it

Screen recording - but no need to go for that, I didn't do it either.  

[Q-Droid 634]

2 hours ago, rbjtech said:

Hmm.. looking at the logs that does appear to show it being around the 450-500ms mark - but it seems faster than that for some of the transitions - I have no way of timing it (other than the logs) but it's definitely quicker than 500ms - that is a long pause that would be very noticeable in video terms.  Possibly 200-300ms - maybe 100ms was a bit optimistic

 

btw - This is transcoding to a Samsung 980 Pro NVME using an Intel 12700K and UHD770 iGPU hardware - so it's probably near the top of the hardware stack with regards to expected transcoding performance..?

2 hours ago, softworkz said:

Crucial CT1000P2SSD8, 11700 - that's a bit slower than yours, but when you say 400 or 500 then it matches anyway.

Screen recording - but no need to go for that, I didn't do it either.  

Is the playback buffer a factor in making the perceived switch time shorter?

 

[softworkz 3326]

20 minutes ago, Q-Droid said:

Is the playback buffer a factor in making the perceived switch time shorter?

What do you exactly mean by "playback buffer"?

Throttling => no

Buffering in the client => yes

[Q-Droid 634]

2 minutes ago, softworkz said:

What do you exactly mean by "playback buffer"?

Throttling => no

Buffering in the client => yes

Yes, the portion of the stream already buffered by the client. It could make the transition feel shorter than the time it takes the server to make the change. 

 

[softworkz 3326]

2 hours ago, Q-Droid said:

Yes, the portion of the stream already buffered by the client. It could make the transition feel shorter than the time it takes the server to make the change. 

You can't simply reduce the buffer because that might give you an early picture but it also increases the risk of getting "hangs" way more often and especially on playback start.

A simple trick would be to continue playing the current until everything is ready and set up to start playing the next instantly. But while the idea is simple, this would be pretty difficult to implement. 

We have some pretty cool things to come with regards to playback startup, and I'd love to tell about it, but I can't post the recipes before it's even out.

[crusher11 851]

Yeah, I assumed that's what Q meant, playing the extant buffer while loading up the new stream.

[Q-Droid 634]

24 minutes ago, softworkz said:

You can't simply reduce the buffer because that might give you an early picture but it also increases the risk of getting "hangs" way more often and especially on playback start.

A simple trick would be to continue playing the current until everything is ready and set up to start playing the next instantly. But while the idea is simple, this would be pretty difficult to implement. 

We have some pretty cool things to come with regards to playback startup, and I'd love to tell about it, but I can't post the recipes before it's even out.

I wasn't proposing a solution or approach. I was merely suggesting that the reason for the perceived very brief pause vs. the longer system time spent changing resolution could be due to the playback buffer on the client. If that can't easily be leveraged to smooth out the transition then it doesn't matter. 

 

[softworkz 3326]

8 hours ago, Q-Droid said:

I wasn't proposing a solution or approach. I was merely suggesting that the reason for the perceived very brief pause vs. the longer system time spent changing resolution could be due to the playback buffer on the client. If that can't easily be leveraged to smooth out the transition then it doesn't matter. 

Reducing the playback startup time for "everything" (many many details add up to this) - is of course the way that makes the real difference and as mentioned, there are things we have in the lab already.

For the other idea - to be honest - I hadn't thought about this before: Letting an existing video continue playing until the next video is ready to start instantly. It wouldn't be an effective improvement but definitely a subjective improvement, because it would avoid the blackout between stop prev and start next.

Though, the problem about this is that the HLS playback implementation and the HTML video element cannot handle two different streams simultaneously (like the currently playing and the next-up video). The stream details will almost always be different and this can't be matched or handled in parallel.
The only way this could be done would be by using two player cores and maybe even two HTML video elements (maybe just one, though). The playback implementation is already pretty complex, and handling two player cores would be really challenging - and first of all: error-prone.
 

I still like the idea, though..

[crusher11 851]

This brings me to something else I've been thinking about: I use Vantage Point, and even locally I get a brief spinning wheel between the trailer and the classification, and then another spinning wheel between the classification and the bumper.

I was going to add a feature request for queuing these up, so that they play immediately one after the other, loading each in the background as the prior clip(s) play(s). But it seems that's not doable?

[softworkz 3326]

It would require a re-write of large parts of the htmljs player implementation which is used by many clients as well, and that means it would be a quite a way to get it working always and everywhere and in all kinds of playback situations. It would be painful for us and all beta users until it's settled. But still doable. As always, the question would be whether it's worth the effort. But sure - go ahead and post a feature request!

[neik 835]

On 5/3/2022 at 3:48 PM, softworkz said:

Assume, you have a 4k file and the client requests 1080 and subtitle burn-in. Now let's say, we want to deliver three qualities for adaptive streaming.
We might then transcode to a 720, a 1080 and a 4k output.

From Emby's perspective you probably need to take 4k to 4k transcoding into consideration but this is I would say quite an edge case - why would someone transcode 4k to 4k in the first place?

I think this idea could go hand in hand with an improvement of the "auto" quality setting in Emby.
(Please correct me if I'm wrong)
Right now, Emby does a speed check on the client and submits it to the server when a playback is started. Once the playback started the speed (transcode bitrate) doesn't change anymore for the whole playback.
What about having a speed check more often (even during playback) and Emby then adapting to the current bitrate available - a conservative approach is surely needed in the bitrate change. If the bitrate goes up to a point where it matches or exceeds the bitrate of the original file then transcoding could even be stopped and the original file served to the client.
This would make the "auto" setting somehow like the "auto" of YouTube, which sometimes start with 480 or 720 but quite quickly goes up to 1080p or even more - depending on available resolution/HW.

Thank you for the quite deep insights and the time you take to do write profound responses.  

[ebr 14904]

6 hours ago, neik said:

like the "auto" of YouTube

The thing to realize there is that YT already has pre-created multiple bitrates of the same item and everyone is playing the same one so this only has to happen once.

With Emby it is a very different story as everyone has their own media and all of these conversions would have to be done on the fly.  That is a very different and much more complex problem than simply switching to a different pre-created stream during playback (which is already built-in to the HLS protocol).

[softworkz 3326]

6 hours ago, neik said:

From Emby's perspective you probably need to take 4k to 4k transcoding into consideration but this is I would say quite an edge case - why would someone transcode 4k to 4k in the first place?

• Video codec not supported by client
• tone mapping required
• subtitle burn-in required
• container not seekable remotely

just those that come to my mind immediately

6 hours ago, neik said:

Right now, Emby does a speed check on the client and submits it to the server when a playback is started. Once the playback started the speed (transcode bitrate) doesn't change anymore for the whole playback.

Correct

6 hours ago, neik said:

What about having a speed check more often (even during playback) and Emby then adapting to the current bitrate available - a conservative approach is surely needed in the bitrate change. If the bitrate goes up to a point where it matches or exceeds the bitrate of the original file then transcoding could even be stopped and the original file served to the client.

• Automatically?
  It can be pretty difficult to find rules for this behavior which satisfy everyone
  .
• Would the user really want it in that situation?
  Maybe a user doesn't want to switch to 4k even though the device would be capable - but the display is so small that it's pointless. The user might have another device where this would be desired, though - could make configuration complex
  .
• Would the user find a short interruption acceptable?
  Maybe an interruption would be just an annoyance and the change undesired anyway. How to decide?
  .
• A switch from transcoded to original can also change subtitle appearance, which may be desired or not
  .
• How to measure the bandwidth and how quickly to take action?
  Maybe a user is just in a lift or a radio blind spot and bandwidth will be ok afterwards

I'm not naming these as blocking arguments. Just to illustrate things which could come into play here, and that's just a few of many.

I'm trying to think about how I would want it. And I'm often coming to fear that an automatic switching could make things just worse in some situations. Streaming clients doing adaptive bandwidths are often switching around, causing them to remain stalled even longer than shorter.

We'll definitely need to do some more research on this at the technical side about what would be possible and might make sense. The very first step, though is about the HLS improvements I mentioned above which are needed to do most of those things.

Meanwhile - I wonder whether we might try to start with something totally simple and minimal :

• When playback hangs/stalls or is about to stall, display a message box on screen which tells something like
  "Slow connection, do you wish to switch to a lower bandwidth?"
• There's a yes/no, yes is focused, so it can easily be confirmed (by clicking OK on the remote or keyboard, etc.)
• It goes only down in bandwidth, not up, because the primary annoyance is when playback hangs, which is also the situation that is easy to assess
• Before somebody will say: "Uh, a message box would be pretty annoying" - let me clarify: this should only be shown in situations where it's clear that playback would hang within the next few seconds. 

Anyway, this might be a good entry point into the subject, but we got other things to come which....well..you'll see..