chef 3788 Posted November 13, 2020 Posted November 13, 2020 (edited) WOW! I found this article which goes into detail about Title sequences, and the algorithms which are used to find these time stamps in each episode (/Netflix). It looks like they have a computer scan the stream. crazy man! They use Machine Learning. Awesome! https://www.diva-portal.org/smash/get/diva2:1412560/FULLTEXT01.pdf Edited November 13, 2020 by chef 3
Luke 38525 Posted November 13, 2020 Posted November 13, 2020 8 minutes ago, chef said: WOW! I found this article which goes into detail about Title sequences, and the algorithms which are used to find these time stamps in each episode (/Netflix). It looks like they have a computer scan the stream. crazy man! They use Machine Learning. Awesome! https://www.diva-portal.org/smash/get/diva2:1412560/FULLTEXT01.pdf I'm actually a little surprised. I would have thought they would just force their content providers to provide the information with every single episode.
chef 3788 Posted November 13, 2020 Posted November 13, 2020 (edited) It would also seem, that they use subtitle start time to analyze where the intro might have stopped. It seems like they can make assumptions about how long the title sequence was based on how much time passed in the stream, and when subtitles are first time-stamped. Use the average title length of 35.6 seconds, and subtract it from the subtitle timestamp. 1. first_sub_timestamp - 35.6s <=0 then we can assume the intro is 35.6 seconds and starts right away. show the "skip intro" option. 2. first_sub_timestamp - 35.6s >= 35.6s then we can assume there was a season recap prior to the title sequence. wait for the result of equation 2 before showing the "skip into" option. I think something like this would allow for the 15.9 seconds deviation mentioned in the results from the article. perhaps allow users to mark the show with either "long title sequence" or "short title sequence" in the metadata. 35.6s, and 19.7s (35.6-15.9 = 19.7) respectively. The threshold for episodes could be updated, and tweek'd with server releases. Edited November 13, 2020 by chef
samuelqwe 316 Posted November 13, 2020 Posted November 13, 2020 4 hours ago, ebr said: The issue is that the method depends on you having that theme song in your library. I see, I had misunderstood earlier then.
knitowskimedia 9 Posted November 15, 2020 Posted November 15, 2020 Why not use chapters for this? Chapters labeled "opening", "op", "ending", "ed", "recap" or other similar things. You could even allow the user add the chapter names that are skippable for a particular show. That wouldn't require any fancy file comparison. This feature also wouldn't be as big a deal if there was a next chapter button for tv episodes. 1
neik 860 Posted November 15, 2020 Posted November 15, 2020 The issue I see is, a lot of users not having proper chapter naming in their files.
Gae082 0 Posted November 16, 2020 Posted November 16, 2020 7 hours ago, knitowskimedia said: Why not use chapters for this? Chapters labeled "opening", "op", "ending", "ed", "recap" or other similar things. You could even allow the user add the chapter names that are skippable for a particular show. That wouldn't require any fancy file comparison. This feature also wouldn't be as big a deal if there was a next chapter button for tv episodes. The problem is most of all TV shows comes without chapters
miniliQuid 87 Posted November 18, 2020 Posted November 18, 2020 But even within shows, the moment the intro starts can differ from episode to episode right? Wouldn't this feature be near impossible to implement unless you have your own shows chaptered perfectly (and I guess labeled so it knows which chapter is the intro)?
PenkethBoy 2066 Posted November 18, 2020 Posted November 18, 2020 no - fingerprinting the files - or the first say 5-10 mins - would give you what you need i believe plex do this or something similar its kind of mute as a couple of clicks of ffwd and you are past most intros for a series - the investment in time for the server to do the work for you and the coding to make it happen are probably considerable compared to a couple of clicks. But people want this as they dont have to do anything or think about it - i get it but not a high priority for me. 2
miniliQuid 87 Posted November 18, 2020 Posted November 18, 2020 3 minutes ago, PenkethBoy said: no - fingerprinting the files - or the first say 5-10 mins - would give you what you need i believe plex do this or something similar its kind of mute as a couple of clicks of ffwd and you are past most intros for a series - the investment in time for the server to do the work for you and the coding to make it happen are probably considerable compared to a couple of clicks. But people want this as they dont have to do anything or think about it - i get it but not a high priority for me. ah, yeah I never thought much about this feature, sometimes I even enjoy the intro, sometimes I just tap FF 2 or 3 times to skip it depending on the show 1
chef 3788 Posted November 18, 2020 Posted November 18, 2020 1 hour ago, miniliQuid said: But even within shows, the moment the intro starts can differ from episode to episode right? Wouldn't this feature be near impossible to implement unless you have your own shows chaptered perfectly (and I guess labeled so it knows which chapter is the intro)? From the article posted above, there is an 86% chance that each episode in a season has the same intro length, and start position within episodes. Don't know how relavant any of that info is to figuring out this feature.
knitowskimedia 9 Posted November 18, 2020 Posted November 18, 2020 On 11/15/2020 at 8:28 PM, Gae082 said: The problem is most of all TV shows comes without chapters I personally don't have a single TV show without chapters, but that's not a huge sample size. Is there a reason why we can't get a next chapter button on the player, for both movies and TV episodes? 1
chef 3788 Posted November 19, 2020 Posted November 19, 2020 (edited) @Luke did you know that ffmpeg has a silence detect option for audio? http://underpop.online.fr/f/ffmpeg/help/silencedetect.htm.gz#:~:text=Detect silence in an audio,duration are expressed in seconds. Does Emby have this option? Because I think this could be the answer. ffmpeg -i VIDEO_AUDIO.aac -af silencedetect=n=-50dB:d=1 What if: 1. Make a scheduled task that would search Episode types in the library 2. Using the information from the SVT PDF Neural Network AI Scan paper above, we clip a 1 min piece of an episode file using ffmpeg 3. Scan that 1 minute piece of audio for the first time silence is detected (this is the end of the intro sequence), mark the end 4. Movie the time-stamp back 35.6s (or closest duration available) and mark the beginning. It doesn't have to be perfect. We don't scan the first episode of a season, because it will almost always contain either a longer intro, or recap, and we should show the user the intro anyway. We only have to scan the second episode of each season, because there is an 86% chance that each following episode will follow suit. If its not perfect... well ... then it will, most always, be 86% good enough. LOL. Edited November 19, 2020 by chef 1
Sammy 767 Posted November 19, 2020 Posted November 19, 2020 32 minutes ago, chef said: @Luke did you know that ffmpeg has a silence detect option for audio? Does our Emby version have this option? Because I think this could be the answer. ffmpeg -i VIDEO_AUDIO.aac -af silencedetect=n=-50dB:d=1 What if: 1. Make a scheduled task that would search Episode types in the library 2. Using the information from the SVT PDF Neural Network AI Scan paper above, we clip a 1 min piece of an episode file using ffmpeg 3. Scan that 1 minute piece of audio for the first time silence is detected (this is the end of the intro sequence), mark the end 4. Movie the time-stamp back 35.6s (or closest duration available) and mark the beginning. It doesn't have to be perfect. We don't scan the first episode of a season, because it will almost always contain either a longer intro, or recap, and we should show the user the intro anyway. We only have to scan the second episode of each season, because there is an 86% chance that each following episode will follow suit. If its not perfect... well ... then it will, most always, be 86% good enough. LOL. I remove comercials, remux to HEVC and move via MCEBuddy. Will this still work?
chef 3788 Posted November 19, 2020 Posted November 19, 2020 (edited) 7 minutes ago, Sammy said: I remove comercials, remux to HEVC and move via MCEBuddy. Will this still work? In theory, all we are doing is estimating a time duration for metadata. So, this doesn't edit your video file, it would look at a one minute duration of audio clipped from your (processed) file to see if it could find the silence between the intro ending, and the episode starting. If it found silence, it could add a tag to your metatdata file (xml/nfo) for <introEnd>. Then, (with a little science/math) move the duration back 35.6 seconds and tag the metadata <introStart> with the duration left over since the beginning. It isn't perfect, but I'm actually testing the ffmpeg command lines right now to see if it even works at all... Edited November 19, 2020 by chef
chef 3788 Posted November 19, 2020 Posted November 19, 2020 (edited) OH MY! It works... So... ... wow. This is the command line in ffmpeg to scan the clip (and it works on video files): private static string ScanSilence(string input) { return $"-i {input} -af silencedetect=noise=-30dB:d=0.5 -f null -"; } Edited November 19, 2020 by chef 2
chef 3788 Posted November 19, 2020 Posted November 19, 2020 (edited) After some further testing, I can confirm that these silence breaks in the audio do correspond with and can be used to estimate intros. For instance, in the log below we see a timestamp of 8 seconds in the fifth silence break of the stream: which corresponds perfectly to a proper break in one of the episodes intros (seen below when the episode shows a second of nothing before the episode starts). Also, a clip doesn't have to be extracted from the episode (like I had initially thought). The silence command will scan the entire episode in a matter of a second. Which means, if we get this correct, we could scan all episodes in an entire library in a couple seconds. Edited November 19, 2020 by chef 1
PenkethBoy 2066 Posted November 19, 2020 Posted November 19, 2020 @chef- just doing some testing as well - and it depends on the video - some moody ones with lots of smoldering looks have numerous silences so we need to set a limit of say 10 mins for the scan - do you have the commands for that for ffmpeg? 1
chef 3788 Posted November 19, 2020 Posted November 19, 2020 1 minute ago, PenkethBoy said: @chef- just doing some testing as well - and it depends on the video - some moody ones with lots of smoldering looks have numerous silences so we need to set a limit of say 10 mins for the scan - do you have the commands for that for ffmpeg? I believe that this will extract a one minute clip private static string CreateVideoClip1(string input) { return $"-ss 00:00:00 -i \"{input}\" -ss 00:00:00 -t 01:00:00 -c copy VideoClip2.mp4"; } can we use the "-ss" command to tell ffmpeg to scan the first minute of the file?
PenkethBoy 2066 Posted November 19, 2020 Posted November 19, 2020 (edited) have you tried this on a stereo audio track? getting loads of errors ok - dodgy file - ignore Edited November 19, 2020 by PenkethBoy
PenkethBoy 2066 Posted November 19, 2020 Posted November 19, 2020 2 minutes ago, chef said: I believe that this will extract a one minute clip private static string CreateVideoClip1(string input) { return $"-ss 00:00:00 -i \"{input}\" -ss 00:00:00 -t 01:00:00 -c copy VideoClip2.mp4"; } can we use the "-ss" command to tell ffmpeg to scan the first minute of the file? not sure will try in a bit
chef 3788 Posted November 19, 2020 Posted November 19, 2020 Just now, PenkethBoy said: not sure will try in a bit or "-t" -t duration (input/output) When used as an input option (before -i), limit the duration of data read from the input file. When used as an output option (before an output url), stop writing the output after its duration reaches duration. duration must be a time duration specification, see (ffmpeg-utils)the Time duration section in the ffmpeg-utils(1) manual. -to and -t are mutually exclusive and -t has priority.
PenkethBoy 2066 Posted November 19, 2020 Posted November 19, 2020 what i am seeing is that even a very brief pause in dialog is getting picked up as a "silence" any way to mave a minimum duration or similar? also not getting much of a match or so many silences not sure at moment how you would pick out the intro - or you would have so many markers as it would get kind of pointless 1
PenkethBoy 2066 Posted November 19, 2020 Posted November 19, 2020 this is prob why plex went with a fingerprint match? 1
chef 3788 Posted November 19, 2020 Posted November 19, 2020 (edited) I am starting to see this as well... But here is one for @Luke (...because his form picture is... Sheldon..) But the break is right there... Maybe we need to adjust the decibels in the command line that register silence. I still think this idea might lead to something.... maybe LOL! Edited November 19, 2020 by chef
Recommended Posts