Jump to content

Recommended Posts

Posted (edited)

WOW! I found this article which goes into detail about Title sequences, and the algorithms which are used  to find these time stamps in each episode (/Netflix). It looks like they have a computer scan the stream. crazy man! They use Machine Learning. Awesome!

https://www.diva-portal.org/smash/get/diva2:1412560/FULLTEXT01.pdf

introexample.thumb.png.d53fa010c2cda14c168cd2e99c4a1436.pngintroexample2.png.29574c2182ff4211f51ee9276879acde.png

Edited by chef
  • Like 3
Posted
8 minutes ago, chef said:

WOW! I found this article which goes into detail about Title sequences, and the algorithms which are used  to find these time stamps in each episode (/Netflix). It looks like they have a computer scan the stream. crazy man! They use Machine Learning. Awesome!

https://www.diva-portal.org/smash/get/diva2:1412560/FULLTEXT01.pdf

I'm actually a little surprised. I would have thought they would just force their content providers to provide the information with every single episode.

Posted (edited)

It would also seem, that they use subtitle start time to analyze where the intro might have stopped.

 

It seems like they can make assumptions about how long the title sequence was  based on how much time passed in the stream, and when subtitles are first time-stamped.

 

Use the average title length of 35.6 seconds, and subtract it from the subtitle timestamp.

1. first_sub_timestamp - 35.6s <=0 then we can assume the intro is 35.6 seconds and starts right away.

show the "skip intro" option.

 

2. first_sub_timestamp - 35.6s >= 35.6s then we can assume there was a season recap prior to the title sequence.

wait for the result of equation 2 before showing the "skip into" option.

 

I think something like this would allow for the 15.9 seconds deviation mentioned in the results from the article.

 

perhaps allow users to mark the show with either "long title sequence" or "short title sequence" in the metadata. 35.6s, and 19.7s (35.6-15.9 = 19.7) respectively. 

 

The threshold for episodes could be updated, and tweek'd with server releases.

 

Edited by chef
samuelqwe
Posted
4 hours ago, ebr said:

The issue is that the method depends on you having that theme song in your library.

I see, I had misunderstood earlier then.

knitowskimedia
Posted

Why not use chapters for this? Chapters labeled "opening", "op", "ending", "ed", "recap" or other similar things. You could even allow the user add the chapter names that are skippable for a particular show. That wouldn't require any fancy file comparison. This feature also wouldn't be as big a deal if there was a next chapter button for tv episodes.

  • Like 1
Posted

The issue I see is, a lot of users not having proper chapter naming in their files.

Posted
7 hours ago, knitowskimedia said:

Why not use chapters for this? Chapters labeled "opening", "op", "ending", "ed", "recap" or other similar things. You could even allow the user add the chapter names that are skippable for a particular show. That wouldn't require any fancy file comparison. This feature also wouldn't be as big a deal if there was a next chapter button for tv episodes.

The problem is most of all TV shows comes without chapters

miniliQuid
Posted

But even within shows, the moment the intro starts can differ from episode to episode right? Wouldn't this feature be near impossible to implement unless you have your own shows chaptered perfectly (and I guess labeled so it knows which chapter is the intro)?

PenkethBoy
Posted

no - fingerprinting the files - or the first say 5-10 mins - would give you what you need 

i believe plex do this or something similar

its kind of mute as a couple of clicks of ffwd and you are past most intros for a series - the investment in time for the server to do the work for you and the coding to make it happen are probably considerable compared to a couple of clicks. But people want this as they dont have to do anything or think about it - i get it but not a high priority for me.

  • Like 2
miniliQuid
Posted
3 minutes ago, PenkethBoy said:

no - fingerprinting the files - or the first say 5-10 mins - would give you what you need 

i believe plex do this or something similar

its kind of mute as a couple of clicks of ffwd and you are past most intros for a series - the investment in time for the server to do the work for you and the coding to make it happen are probably considerable compared to a couple of clicks. But people want this as they dont have to do anything or think about it - i get it but not a high priority for me.

ah, yeah I never thought much about this feature, sometimes I even enjoy the intro, sometimes I just tap FF 2 or 3 times to skip it depending on the show :D

  • Haha 1
Posted
1 hour ago, miniliQuid said:

But even within shows, the moment the intro starts can differ from episode to episode right? Wouldn't this feature be near impossible to implement unless you have your own shows chaptered perfectly (and I guess labeled so it knows which chapter is the intro)?

From the article posted above, there is an 86% chance that each episode in a season has the same intro length, and start position within episodes.  Don't know how relavant any of that info is to figuring out this feature.

knitowskimedia
Posted
On 11/15/2020 at 8:28 PM, Gae082 said:

The problem is most of all TV shows comes without chapters

I personally don't have a single TV show without chapters, but that's not a huge sample size. Is there a reason why we can't get a next chapter button on the player, for both movies and TV episodes?

  • Like 1
Posted (edited)

@Luke did you know that ffmpeg has a silence detect option for audio? 

http://underpop.online.fr/f/ffmpeg/help/silencedetect.htm.gz#:~:text=Detect silence in an audio,duration are expressed in seconds.

Does Emby  have this option? Because I think this could be the answer.

ffmpeg -i VIDEO_AUDIO.aac -af silencedetect=n=-50dB:d=1 

 

What if:

1. Make a scheduled task that would search Episode types in the library

2. Using the information from the SVT PDF Neural Network AI Scan paper above, we clip a 1 min piece of an episode file using ffmpeg

3. Scan that 1 minute piece of audio for the first time silence is detected (this is the end of the intro sequence), mark the end

4. Movie the time-stamp back 35.6s (or closest duration available) and mark the beginning.

 

It doesn't have to be perfect. We don't  scan the first episode of a season, because it will almost always contain either a longer intro, or recap, and we should show the user the intro anyway.

We only have to scan the second episode of each season, because there is an 86% chance that each following episode will follow suit. 

If its not perfect... well ... then it will, most always,  be 86% good enough. LOL.

 

 

 

Edited by chef
  • Like 1
Posted
32 minutes ago, chef said:

@Luke did you know that ffmpeg has a silence detect option for audio? 

Does our Emby version have this option? Because I think this could be the answer.


ffmpeg -i VIDEO_AUDIO.aac -af silencedetect=n=-50dB:d=1 

 

What if:

1. Make a scheduled task that would search Episode types in the library

2. Using the information from the SVT PDF Neural Network AI Scan paper above, we clip a 1 min piece of an episode file using ffmpeg

3. Scan that 1 minute piece of audio for the first time silence is detected (this is the end of the intro sequence), mark the end

4. Movie the time-stamp back 35.6s (or closest duration available) and mark the beginning.

 

It doesn't have to be perfect. We don't  scan the first episode of a season, because it will almost always contain either a longer intro, or recap, and we should show the user the intro anyway.

We only have to scan the second episode of each season, because there is an 86% chance that each following episode will follow suit. 

If its not perfect... well ... then it will, most always,  be 86% good enough. LOL.

 

 

 

I remove comercials, remux to HEVC and move via MCEBuddy. Will this still work?

Posted (edited)
7 minutes ago, Sammy said:

I remove comercials, remux to HEVC and move via MCEBuddy. Will this still work?

In theory, all we are doing is estimating a time duration for metadata.

So, this doesn't edit your video file, it would look at a one minute duration of audio clipped from your (processed) file to see if it could find the silence between the intro ending, and the episode starting.

If it found silence, it could add a tag to your metatdata file (xml/nfo) for <introEnd>.

Then, (with a little science/math) move the duration back 35.6 seconds and tag the metadata <introStart> with the duration left over since the beginning.

It isn't perfect, but I'm actually testing the ffmpeg command lines right now to see if it even works at all...

Edited by chef
Posted (edited)

OH MY! It works...

Latest Superhero GIFs | Gfycat

silencePNG.png.a599743c4ec4c947f0a2dd0b4180efbd.png

 

So... ... wow. 

This is the command line in ffmpeg to scan the clip (and it works on video files):

        private static string ScanSilence(string input)
        {
            return $"-i {input} -af silencedetect=noise=-30dB:d=0.5 -f null -";
        }

 

Edited by chef
  • Like 2
Posted (edited)

After some further testing, I can confirm that these silence breaks in the audio do correspond with and can be used to estimate intros.

For instance, in the log below we see a timestamp of 8 seconds in the fifth silence break of the stream:

silenceTest1.png.1f9d3565443340d4bcfce789a8567b78.png

 

which corresponds perfectly to a proper break in one of the episodes intros (seen below when the episode shows a second of nothing before the episode starts). 

silenceTest2.thumb.png.4379fbf1888ab90c1c360e037e640bdf.png

 

Also, a clip doesn't have to be extracted from the episode (like I had initially thought). The silence command will scan the entire episode in a matter of a second. Which means, if we get this correct, we could scan all episodes in an entire library in a couple seconds.

Edited by chef
  • Like 1
PenkethBoy
Posted

@chef- just doing some testing as well - and it depends on the video - some moody ones with lots of smoldering looks have numerous silences :)

so we need to set a limit of say 10 mins for the scan - do you have the commands for that for ffmpeg?

  • Like 1
Posted
1 minute ago, PenkethBoy said:

@chef- just doing some testing as well - and it depends on the video - some moody ones with lots of smoldering looks have numerous silences :)

so we need to set a limit of say 10 mins for the scan - do you have the commands for that for ffmpeg?

I believe that this will extract a one minute clip

        private static string CreateVideoClip1(string input)
        {
            return $"-ss 00:00:00 -i \"{input}\" -ss 00:00:00 -t 01:00:00 -c copy VideoClip2.mp4";
          
        }

can we use the "-ss" command to tell ffmpeg to scan the first minute of the file?

PenkethBoy
Posted (edited)

have you tried this on a stereo audio track?

getting loads of errors

 

ok - dodgy file - ignore

Edited by PenkethBoy
PenkethBoy
Posted
2 minutes ago, chef said:

I believe that this will extract a one minute clip


        private static string CreateVideoClip1(string input)
        {
            return $"-ss 00:00:00 -i \"{input}\" -ss 00:00:00 -t 01:00:00 -c copy VideoClip2.mp4";
          
        }

can we use the "-ss" command to tell ffmpeg to scan the first minute of the file?

not sure will try in a bit

Posted
Just now, PenkethBoy said:

not sure will try in a bit

or "-t"

-t duration (input/output)

When used as an input option (before -i), limit the duration of data read from the input file.

When used as an output option (before an output url), stop writing the output after its duration reaches duration.

duration must be a time duration specification, see (ffmpeg-utils)the Time duration section in the ffmpeg-utils(1) manual.

-to and -t are mutually exclusive and -t has priority.

PenkethBoy
Posted

what i am seeing is that even a very brief pause in dialog is getting picked up as a "silence" any way to mave a minimum duration or similar?

also not getting much of a match or so many silences not sure at moment how you would pick out the intro - or you would have so many markers as it would get kind of pointless

  • Like 1
PenkethBoy
Posted

this is prob why plex went with a fingerprint match?

  • Like 1
Posted (edited)

I am starting to see this as well...

But here is one for @Luke (...because his form picture is... Sheldon..)

But the break is right there... 

silenceTest4.thumb.png.6f0e9abbe6045265315f4b435b174f58.pngsilenceTest3.png.f8862070f66699a179df155eb80f6438.png

 

Maybe we need to adjust the decibels in the command line that register silence. 

I still think this idea might lead to something.... maybe LOL!

 

Edited by chef
Guest
This topic is now closed to further replies.
×
×
  • Create New...