Incorrect Detection of Interlaced Video?

[markyp 38]

Sorry for missing the debate here...

Just to clear up how the 2 files were generated...

I took T1, a nearly 7Gb recording taken from Live TV last week (England versus Finland Football game broadcast on ITV) 

I copied and renamed the file as T1 for ease...

For T2 I used the following command:

ffmpeg -i T1.TS -t 3600 -c copy T2.TS

for T3 I used the following command:

ffmpeg -ss 8200 -i T1.TS -t 3600 -c copy T3.TS

I hope this helps...

 

[Guest]

8 hours ago, markyp said:

Sorry for missing the debate here...

Just to clear up how the 2 files were generated...

I took T1, a nearly 7Gb recording taken from Live TV last week (England versus Finland Football game broadcast on ITV) 

I copied and renamed the file as T1 for ease...

For T2 I used the following command:

ffmpeg -i T1.TS -t 3600 -c copy T2.TS

for T3 I used the following command:

ffmpeg -ss 8200 -i T1.TS -t 3600 -c copy T3.TS

I hope this helps...

 

Does the original file/recording have both interlaced and progressive video? It seems to me that the beginning (the commercials) is progressive, and when the game is playing, the video is interlaced. Is that correct?

Edited October 22, 2024 by generiq

[markyp 38]

14 hours ago, generiq said:

Does the original file/recording have both interlaced and progressive video? It seems to me that the beginning (the commercials) is progressive, and when the game is playing, the video is interlaced. Is that correct?

Correct!

[Guest]

@softworkz @Luke static, predetermined deinterlacing options are not going to work here. Player dynamic deinterlacing is required. Deinterlacing progressive video will have negligible impact, but interlaced video that isn't deinterlaced is horrible. For the players that support dynamic deinterlacing, you really should consider using that. In cases like this, deinterlacing will activate even when the server can't get it right. @markyp is deliberately enabling deinterlacing for all content to get around this.

[softworkz 5066]

@markyp- Would you be able to cut out a slightly larger segment which includes a switch between interlaced and non-interlaced?

[Guest]

20 minutes ago, softworkz said:

@markyp- Would you be able to cut out a slightly larger segment which includes a switch between interlaced and non-interlaced?

When I play t2, deinterlacing gets triggered in the middle.

Portable mpv log.txt

[  35.041][v][auto_profiles] Re-evaluating auto profile Deinterlace
[  35.041][i][auto_profiles] Applying auto profile: Deinterlace
[  35.049][d][cplayer] Run command: apply-profile, flags=64, args=[name="Deinterlace", mode="apply"]
[  35.050][v][cplayer] Applying profile 'Deinterlace'...
[  35.050][v][cplayer] Setting option 'hwdec' = 'd3d11va-copy' (flags = 4)
[  35.050][v][vd] Looking at hwdec h264-d3d11va-copy...
[  35.086][v][vd] Trying hardware decoding via h264-d3d11va-copy.
[  35.087][v][cplayer] Setting option 'vf' = 'bwdif' (flags = 4)
[  35.089][v][vf] User filter list:
[  35.089][v][vf]   bwdif (bwdif.00)

 

[softworkz 5066]

You are right. In the video above, I had probed just 500 frames. But when I probe all ~1500 frames, there are in fact two periods with interlaced frames:

[Guest]

5 minutes ago, softworkz said:

You are right. In the video above, I had probed just 500 frames. But when I probe all ~1500 frames, there are in fact two periods with interlaced frames:

And if that were a live stream, deinterlacing wouldn't be active either

 

[Carlo 4560]

Couple comments
MPEGTS allows mixed interlaced and progressive content by marking the TTF or BFF flags correctly in each PID.

Unless you've extensively tested the capture output of any hardware/software combination against a known source broadcast test stream you don't know if the capture is true or not. A quick search will find examples of this on Stack Exchange, Intel forums, etc.  IE (Stack Exchange): "Using ffprobe, polling every 1sec on a known interlaced RTP stream only detected interlace 30% of the time. Using analyzeduration set as high as 120 seconds made no difference."

If true, that would make me highly suspect of the technique used to determine interlaced frames as well as if it's using the information in the transport stream correctly if at all. Any manipulation, processing or saving to files as well as metaInfo generated for said transport stream by the same library or toolkit would also be highly suspect and require verification IMHO.

Others have shown similar differences of streams when comparing a transport stream directly vs one re-streamed using ffmpeg.

Some personal testing of examples demonstrated by others does show similar results and leaves me skeptical of how the ff tools handle many things. 

 

Edited October 23, 2024 by Carlo

[Guest]

21 minutes ago, Carlo said:

Couple comments
MPEGTS allows mixed interlaced and progressive content by marking the TTF or BFF flags correctly in each PID.

Unless you've extensively tested the capture output of any hardware/software combination against a known source broadcast test stream you don't know if the capture is true or not. A quick search will find examples of this on Stack Exchange, Intel forums, etc.  IE (Stack Exchange): "Using ffprobe, polling every 1sec on a known interlaced RTP stream only detected interlace 30% of the time. Using analyzeduration set as high as 120 seconds made no difference."

If true, that would make me highly suspect of the technique used to determine interlaced frames as well as if it's using the information in the transport stream correctly if at all. Any manipulation, processing or saving to files as well as metaInfo generated for said transport stream by the same library or toolkit would also be highly suspect and require verification IMHO.

Others have shown similar differences of streams when comparing a transport stream directly vs one re-streamed using ffmpeg.

Some personal testing of examples demonstrated by others does show similar results and leaves me skeptical of how the ff tools handle many things. 

 

In this case, Emby is incorrectly identifying the video as not-interlaced in its entirety. Leading to no deinterlacing at all. A 100% failure. Using a frame by frame detection (as in the case of mpv) ensures a much higher degree of success. In my use, I've yet to see it fail. A false positive is more desirable than a false negative. As shown by @markyp deliberately applying deinterlacing of ALL video, 

But, if ffprobe is not to be trusted, what would be the alternative? And if pre-scanning is to be employed, you would still need to scan the entire video, or as we see here the result is all but useless. Having the player actively monitor the frames as it plays, still ensures a more reliable outcome. Especially if watching a live stream where scanning ahead isn't possible. 

[softworkz 5066]

42 minutes ago, Carlo said:

MPEGTS allows mixed interlaced and progressive content by marking the TTF or BFF flags correctly in each PID.

There's no such information at the PID level. There you only have a stream type id (which tells the coded but not whether it's interlaced) and various descriptors, but there's no descriptor for intelaced state.

In all cases (MPEG-2, H264, H265), the information about interlacing is only in the bitstream of the codecs itself. For example, in case of H.264, there's an SPS (sequence parameter set) which defines parameters for a range of frames. There are multiple modes, for example when the SPS has the frame_mbs_only_flag unset, all frames in this set are progressive frames. If it's set then it depends on the field flag of each frame (= field if set).

 

 

16 minutes ago, generiq said:

In this case, Emby is incorrectly identifying the video as not-interlaced in its entirety. Leading to no deinterlacing at all. A 100% failure. 

55 minutes ago, Carlo said:

"Using ffprobe, polling every 1sec on a known interlaced RTP stream only detected interlace 30% of the time.

My friends, you are still missing the point: It is impossible to get a valid yes/no answer when the content is mixed. What do you expect to happen? Continue scanning the file until you find a frame with different interlacing than the initial frames? And then declare the file as non-interlaced when the first frames were interlaced? Or declare the file as interlaced when the first frames were non-interlaced? It's always wrong in either case. That's why there's no point in scanning further through the file.

The ffprobe results must rather be seen as only valid for the start of the sequence it is probing.

55 minutes ago, Carlo said:

Using analyzeduration set as high as 120 seconds made no difference."

Correct, it doesn't. Increasing probesize or analyzeduration only affects ffprobe's detection of things it hasn't detected yet, but if it has made a detection regarding a certain type of information then it's done with the detection of that type.

[Guest]

3 minutes ago, softworkz said:

 What do you expect to happen? Continue scanning the file until you find a frame with different interlacing than the initial frames? 

This is exactly what the player will do.

For the server detection, other than scanning the entire file, there isn't much else you can do.

What i am suggesting is not use the server result to apply deinterlacing. Where possible, let the player make the decision. 

You see that Mark is setting deinterlacing to yes. This means he's defeating what Emby is choosing. In doing so, all video he plays, deinterlacing is being applied.

[softworkz 5066]

This topic is about Emby's detection, not video playback in the Emby Windows beta. No need to worry about the latter.  

[Guest]

9 minutes ago, softworkz said:

This topic is about Emby's detection, not video playback in the Emby Windows beta. No need to worry about the latter.  

I was actually referring to all emby apps.

[Carlo 4560]

5 hours ago, generiq said:

In this case, Emby is incorrectly identifying the video as not-interlaced in its entirety. Leading to no deinterlacing at all. A 100% failure. Using a frame by frame detection (as in the case of mpv) ensures a much higher degree of success. In my use, I've yet to see it fail. A false positive is more desirable than a false negative. As shown by @markyp deliberately applying deinterlacing of ALL video, 

  Keep in mind if this is a live stream, Emby server can only go by what it has received before writing/sending/transcoding the stream. I'd think it only right to have the metadata based on what it encounters during the very first part of the transport stream.

IMHO, any de-interlacing should be left to the client to handle if direct streaming. Most TVs purchased in the last 10 to 15 years will handle deinterlacing and scaling on the fly as needed. Same with most dedicated streaming boxes. If a client can't handle the stream or needs transcoding that's where I think the bigger issue is.

5 hours ago, softworkz said:

There's no such information at the PID level. There you only have a stream type id (which tells the coded but not whether it's interlaced) and various descriptors, but there's no descriptor for intelaced state.

My bad mentioning PID. I looked at '''mfxVideoParam.mfx.frameInfo.PicStruct = MFX_PICSTRUCT_FIELD_TFF (or BFF)" and read PID not PIC

5 hours ago, softworkz said:

In all cases (MPEG-2, H264, H265), the information about interlacing is only in the bitstream of the codecs itself. For example, in case of H.264, there's an SPS (sequence parameter set) which defines parameters for a range of frames. There are multiple modes, for example when the SPS has the frame_mbs_only_flag unset, all frames in this set are progressive frames. If it's set then it depends on the field flag of each frame (= field if set).

As mentioned above, I think the only logical thing to do is mark it based on what's encountered at the start of the stream. Technically, I guess we could update the metadata at completion to add a comment "Mixed Interlaced and Progressive frames" or similar. Other than having a human readable comment, it does nothing.

4 hours ago, generiq said:

This is exactly what the player will do.

For the server detection, other than scanning the entire file, there isn't much else you can do.

What i am suggesting is not use the server result to apply deinterlacing. Where possible, let the player make the decision. 

You see that Mark is setting deinterlacing to yes. This means he's defeating what Emby is choosing. In doing so, all video he plays, deinterlacing is being applied.

I agree the player should be responsible for deinterlacing if they are able to direct stream. The client deinterlacing IMHO, shouldn't be based on some upfront or global header, but instead be done dynamically based on the frame it's processing.

What I find more important is when transcoding is needed by the server and deinterlacing is needed. If deinterlacing doesn't use a motion-adaptive/compensated method, you can lose temporal resolution, have less fluid motion, get flickering or combing artifacts, etc that make watching something like sports difficult. It's a difficult situation to handle on the backend.

[softworkz 5066]

46 minutes ago, Carlo said:

Keep in mind if this is a live stream, Emby server can only go by what it has received before writing/sending/transcoding the stream. I'd think it only right to have the metadata based on what it encounters during the very first part of the transport stream.

Yes, currently for server transcoding ffmpeg needs to be started with certain parameters and you'd need to stop and restart in order to make a change.

With TVnext, there's more control, since the stream goes through the server which parses the codec data and ffmpeg is used only in a packet-by-packet manner, so it would be possible to stop and reconfigure when there's a change.

But the problem is still at the client side. While the RFC doesn't restrict it, the Apple authoring spec demands HLS video to be non-interlaced, and this had led to the situation that not all HLS client support interlaced video (even when they support it in other cases - eg. TVs). 

56 minutes ago, Carlo said:

What I find more important is when transcoding is needed by the server and deinterlacing is needed. If deinterlacing doesn't use a motion-adaptive/compensated method, you can lose temporal resolution, have less fluid motion, get flickering or combing artifacts, etc that make watching something like sports difficult. It's a difficult situation to handle on the backend.

The server applies deinterlacing with "send_frame" output (when it applies deinterlacing for transcoding). This makes it resilient to mixed streams, because full frames can't be deinterlaced and when interlaced frames ("fields") appear, they are deinterlaced as frames, which means: the frame rate never changes. And that's the really important part, because a player internally can easily handle framerate changes as a result to its internal processing, but streaming video content with framerate changes will get many players in trouble.
(not those like mpv of course, but many others).

[softworkz 5066]

BTW, this is one of the things that needs to be done when resuming work and testing on TVnext: get the information into device profiles of clients, whether they support interlaced video over HLS (and for which codecs) as well as whether HEVC is supported with mpegts segments.

[markyp 38]

Thanks All,

So where i think we've got to is that we shouldn't rely on the Server to detect interlaced or progressive content, and that it should be left to the client application?

in that case, do we need to do some work on the Server side to remove the current implementation, and leave this to the clients?

[Guest]

6 minutes ago, markyp said:

Thanks All,

So where i think we've got to is that we shouldn't rely on the Server to detect interlaced or progressive content, and that it should be left to the client application?

in that case, do we need to do some work on the Server side to remove the current implementation, and leave this to the clients?

Right now, the only option we have in emby to guarantee deinterlacing is to set it manually in the settings as you've already done in Theater. Or if you use the desktop app, you can disable it in the settings and set it to Auto in the mpv.conf. Then it will be dynamic. This is what I did. Unfortunately, an mpv.conf won't work with the new app. I will still be using mpv as my primary as Theater will likely never give me everything that I require. Emby's handling of deinterlaced video is far from desirable. Hopefully they will change this, but I've talked about this for years. I consider it to be broken, but our options have been reduced further. 

[softworkz 5066]

I think we should calm down a bit on this topic. We're not talking about something that has been caused or emphasized by a recent change (neither to TV broadcast, nor to Emby software). Such switching of stream formats may be more frequently seen in the US and in the UK, but it's definitely not something new. And on the Emby side, nothing has changed either.
When thinking back through all the many past years, I can't even remember any substantial issue with Emby Server and such tv recordings with varying interlaced states. Not in the transcoding area at least.
I really don't see a big alarm here. If it was a big thing, it would have frequently come up before. I also have an explanation for that: normally, when you record a TV broadcast, the DVR software records and saves individual programs to disk as separate files. And no TV broadcaster employs a monkey to let them arbitrarily switch back-and-forthe between interlaced or non-interlaced. So, switching during commercial breaks is absolutely plausible, but you'll hardly find any program where it switches video format right in the middle of a show - if at all, then from one show to another show.
Normally, the DVR saves recordings like one file per show/broadcast and the result is that the frames being probed at the beginning are the relevant ones for this show. The worst thing that can happen there is that the appearance of commercials could be suboptimal and that's it.

There are takeaways for TVnext, for which it was good to have the discussion. For client playback, I would not go that far as to always activate deinterlacing unconditionally as that may have adverse effects and we also have the hardware filtering cases where we need to manage deinterlacing ourselves.
So, the most reasonable logic in client players for activation of deinterlacing that I can think of is to enable it..

• either when Emby Server has detected it as interlaced
  or
• when there's a combination of codec and container where it's known that all or parts could be interlaced
  (#1 candidates of course H.262/4/5 + MPEGTS)

Additional combinations can be added if needed, but otherwise there's not much more to it.

[Carlo 4560]

I agree @softworkzthis isn't anything major but we probably should do a couple things.  For example, a recording is being made that starts recording 5 minutes early which will say is interlaced. The program broadcast itself is progressive, then 5 minutes post recording which goes back to interlaced. While not common place yet it might be a good idea to probe the recording a couple minutes into the recording (based on the EPG) and use this for the file header which we could modify after completion of recording, Another method could be to probe the stream or note the frame markings for interlace every X amount IE 4:39 seconds (avoid 5 minutes). Then simply mark the recording based on the number of polls.   That would at least justify why it's marked one way or another. This is really nothing but a justification of how the recording is marked.

Technically both HLS & SRT require progressive, while MPEG-DASH, RTMP & WebRTC allow interfaced content.

[softworkz 5066]

56 minutes ago, Carlo said:

  For example, a recording is being made that starts recording 5 minutes early which will say is interlaced. The program broadcast itself is progressive, then 5 minutes post recording which goes back to interlaced

I had thought about the exact same example. For TVnext we don't need to do any probing. Instead we are writing "live" metadata information, i.e. right in the moment when the data passes through. That means we can have precise information for everything that gets recorded or watched - for "free". Scanning files on disk is expensive, but not when the data is passing by anyway and you need to parse it anyway. In the end such format changes can even be useful for determining program start/end and possibly cutting out (or jumping over) commercials.  

[markyp 38]

19 hours ago, softworkz said:

I had thought about the exact same example. For TVnext we don't need to do any probing. Instead we are writing "live" metadata information, i.e. right in the moment when the data passes through. That means we can have precise information for everything that gets recorded or watched - for "free". Scanning files on disk is expensive, but not when the data is passing by anyway and you need to parse it anyway. In the end such format changes can even be useful for determining program start/end and possibly cutting out (or jumping over) commercials.  

Having just becoming aware of what TVnext was looking to achieve, it certainly will be interesting on the Live TV front with Emby!  

All I'm looking for is seamless playback of my content, without having to resort to changing options etc... So, I want to play 4k content at 23fps in HDR on my TV, but also watch progressive/interlaced/mixed content at 50fps on the same TV without dropping frames.

However i also want this done to the best quality and most efficient depending on the hardware available - now in my case my server has a beefy CPU but a not-so-beefy GPU, a GTX960. My main clients all have decent GPU's in them though in an effort to extract the best quality - and all run Windows, but i realise I'm not the norm!!

I want to utilise the hardware I've got to extract the best quality possible to maximise the enjoyment of media playback!

Edited October 25, 2024 by markyp

[Carlo 4560]

If all your clients are windows w/4K support using Ethernet, using Emby Theater with the bitrate set at the highest should hardly ever need to transcode.

[markyp 38]

12 hours ago, Carlo said:

If all your clients are windows w/4K support using Ethernet, using Emby Theater with the bitrate set at the highest should hardly ever need to transcode.

No they don't - but they need to deinterlace!