Completed Show Intro Skip Option

[chef 3745]

1 hour ago, jptestung said:

Hi, I tried to read a couple of posts from this topic but it's huge, and I can't find what I'm looking for. What I understand is that the dev team is working on this feature, a scheduled task will browse the media files and determine intro/end timestamps. But what I don't understand is how it will decide intro/end. Is it based on the music? Credits detection ? Is it based on an open source library or custom video/audio analysis code ?

Using chroma printing technology, you can  encode a snipit of audio from an episode (10 min). Each episode in a season is chromaprinted. We use seasons because some series change their intros per season.

Once the print is taken, we take one print from an episode, and slide it over another, until they align, and we match the print. This tells us that the audio is comparable. This is the intro for both episodes. We take the timestamp for the stream and make that data available as an endpoint in the Emby api. 

 

Each episode in a season is compared with every other episode until the algorithm is able to make a match. 

Edited December 26, 2020 by chef

[Deathsquirrel 741]

2 hours ago, chef said:

Using chroma printing technology, you can  encode a snipit of audio from an episode (10 min). Each episode in a season is chromaprinted. We use seasons because some series change their intros per season.

Once the print is taken, we take one print from an episode, and slide it over another, until they align, and we match the print. This tells us that the audio is comparable. This is the intro for both episodes. We take the timestamp for the stream and make that data available as an endpoint in the Emby api. 

 

Each episode in a season is compared with every other episode until the algorithm is able to make a match. 

I'm curious how well this is working for shows like The Simpsons, which have three different intro lengths that they use throughout a season.

[chef 3745]

43 minutes ago, Deathsquirrel said:

I'm curious how well this is working for shows like The Simpsons, which have three different intro lengths that they use throughout a season.

I would also be interested in that. Because each episode is compared to every other episode in the season, it is possible, that if there are two intros that are the same, these will compare and there will be a result for both episodes. However, if there is only one episode in the season with a distinctly different intro, it is possible that only the parts of the theme which match another version of it will have results.

Does that make sense? If the Simpsons theme intro is extended for one episode only, then only the parts of the theme that is comparable will get a result. 

An intro time stamp will be generated, but in that particular instance, the tine stamp for the end of the intro would be marked at the last position in the stream where the compatible episode theme would match. 

However,  if there were two episodes with extended intros, we would get a match.

Edited December 26, 2020 by chef

[chef 3745]

@TeamB

I was able to get the output stream from ffmpeg, and hold the data in a memory stream, and pipe it into fpcalc. 

But, the only thing I need to figure out now is how to stop one error I get when piping the stream.

The error is "The Pipe has ended"

I think it is a buffering issue...

 

Piping ffmpeg to stream:

private static void ExtractPCMAudio(string input, int duration)
        {
            Process proc = new Process();

            proc.StartInfo.FileName = @"ffmpeg.exe";
            proc.StartInfo.Arguments = $"-t 00:3:00 -i \"{input}\" -f wav -ac 1 -acodec pcm_s16le -ar 16000 -"; //"-" Dash pipes the stream
            proc.StartInfo.UseShellExecute = false;
            proc.StartInfo.RedirectStandardInput = true;
            proc.StartInfo.RedirectStandardOutput = true;

            proc.Start();

            //Tricky, we treat the stream like a file stream, even if we aren't going to create a file
            FileStream baseStream = proc.StandardOutput.BaseStream as FileStream;

            byte[] audioBytes = null;
            int lastRead = 0;

            using (MemoryStream ms = new MemoryStream())
            {            
                byte[] buffer = new byte[4096];
                do
                {
                    lastRead = baseStream.Read(buffer, 0, buffer.Length);
                    ms.Write(buffer, 0, lastRead);
                } while (lastRead > 0);

                audioBytes = ms.ToArray();
            }
            
            proc.StandardInput.BaseStream.Close();
            
            baseStream.Close();
            FingerPrintAudio(audioBytes);
            Console.WriteLine("Done!");
            
        }

 

using byte array stream data with fpcalc:

 private static void FingerPrintAudio(byte[] audioBytes)
        {
            // Using 300 second length to get a more accurate fingerprint, but it's not required
            var fpcalc = "fpcalc.exe";
            var procStartInfo =
                new ProcessStartInfo(fpcalc, $" -raw -length 600 -json") 
                {
                    RedirectStandardOutput = true,
                    RedirectStandardInput = true,
                    RedirectStandardError  = true,
                    UseShellExecute        = false,
                    CreateNoWindow         = true,
                };

            var process = new Process {StartInfo = procStartInfo};
            process.Start();            
            
            process.StandardInput.BaseStream.Write(audioBytes, 0, audioBytes.Length); //<--This is where the piping error happens.

            string processOutput = null;
            using (var sr = new StreamWriter("Test.json"))
            {
                while ((processOutput = process.StandardOutput.ReadLine()) != null)
                {
                    sr.WriteLine(processOutput);
                    Console.WriteLine(processOutput);
                }
                sr.Close();
            }
            
        }

 

I think it is close, like really close. Piping streams or byte arrays is kind of advanced.  

[Dibbes 431]

@chef Hey mate! I've been following this thread with with a lot of interest (just didn't have anything useful to add). Just wondering, did you post a dll anywhere or is it just the Github repository?

thanks for the work, by the way!!!

[samuelqwe 313]

4 minutes ago, Dibbes said:

@chef Hey mate! I've been following this thread with with a lot of interest (just didn't have anything useful to add). Just wondering, did you post a dll anywhere or is it just the Github repository?

thanks for the work, by the way!!!

You can find a DLL on the GitHub page in the releases section:

https://github.com/chefbennyj1/Emby.IntroSkip/releases

[Dibbes 431]

1 minute ago, samuelqwe said:

You can find a DLL on the GitHub page in the releases section:

https://github.com/chefbennyj1/Emby.IntroSkip/releases

Yes, but that one is 8 days old... Chef made changes a few hours back...

[samuelqwe 313]

18 minutes ago, Dibbes said:

Yes, but that one is 8 days old... Chef made changes a few hours back...

He’s testing stuff on the side that isn’t quite ready to be released, so there isn’t currently a newer DLL available.

[Dibbes 431]

2 minutes ago, samuelqwe said:

He’s testing stuff on the side that isn’t quite ready to be released, so there isn’t currently a newer DLL available.

Hence my question if it's just the github respository

[samuelqwe 313]

11 minutes ago, Dibbes said:

Hence my question if it's just the github respository

That is publicly available, yes. That’s not to say that there isn’t a private group somewhere. @chef

[chef 3745]

1 minute ago, samuelqwe said:

That is publicly available, yes. That’s not to say that there isn’t a private group somewhere. @chef

Got it! Thanks!

[nielstenboom 5]

Hi guys,

I actually open-sourced the code for a graduation project where the goal was exactly to detect recaps, previews and opening&closing credits. You can check it out here: https://github.com/nielstenboom/recurring-content-detector

I figured maybe it can be of use! Would have to port it somehow, as it is python based.

Cheers!

[neik 837]

@chef, how is going with the development? Anything new you can share with us?  

[chef 3745]

6 hours ago, neik said:

@chef, how is going with the development? Anything new you can share with us?  

Good things are happening

[TeamB 2352]

On 1/11/2021 at 2:36 AM, nielstenboom said:

Hi guys,

I actually open-sourced the code for a graduation project where the goal was exactly to detect recaps, previews and opening&closing credits. You can check it out here: https://github.com/nielstenboom/recurring-content-detector

I figured maybe it can be of use! Would have to port it somehow, as it is python based.

Cheers!

Some cool work. What sort of performance did you experience, your approach of using frame based vectors and sequence matching is probably the most accurate way of doing this but man is it going to take a while to process a library of 6K+ episodes on a low powered NAS server

Edited January 13, 2021 by TeamB

[TeamB 2352]

20 hours ago, neik said:

@chef, how is going with the development? Anything new you can share with us?  

I think we are waiting for the next Beta of Emby that will have the chromaprint mux built into the  ffmpeg that ships with Emby. This way we can do away with fpcalc.

[nielstenboom 5]

10 hours ago, TeamB said:

Some cool work. What sort of performance did you experience, your approach of using frame based vectors and sequence matching is probably the most accurate way of doing this but man is it going to take a while to process a library of 6K+ episodes on a low powered NAS server

Yes that is indeed the tradeoff I had to make, it's not very fast.. from the top of my head I'd say about 15 mins of processing per episode on a normal laptop? Would probably be worse on a low powered NAS server haha

But I do believe the accuracy should be high for a skip feature to be usable, you don't want to skip actual content.

 

[Sammy 735]

I am no seeing a "Skip Intro" option for some shows on AndroidTV beta .70g. What's going on? I haven't updated the server past 4.5.2.0 because currently it will not expose CC's for LiveTV because the server is transcoding *.ts streams.

Did I miss something?

@chef @Luke

Edited January 24, 2021 by Sammy

[samuelqwe 313]

11 minutes ago, Sammy said:

I am no seeing a "Skip Intro" option on AndroidTV. What's going on? I haven't updated the server because currently it will not expose CC's for LiveTV because the server is transcoding *.ts streams.

Did I miss something?

@chef @Luke

Don’t think there’s really been any work at integrating this yet. The only thing we have is the plugin @chef made that can find the intro time stamps, but it doesn’t do anything with Emby directly.

So I don’t think you’ve really missed anything, because the feature hasn’t been implemented.

[Sammy 735]

Must be in the source, like from PlayOn or something..

[Sammy 735]

 

Yup. I go this particular season from PlayOn recorded from AMC..

Edited January 24, 2021 by Sammy

[chef 3745]

35 minutes ago, Sammy said:

 

Yup. I go this particular season from PlayOn recorded from AMC..

So, you're saying they implemented a skip intro in the beta server?

 

I was waiting for beta ffmpeg to be released to continue. Maybe they did it with out us?

 

Edit: nevermind

Edited January 25, 2021 by chef

[Sammy 735]

17 minutes ago, chef said:

So, you're saying they implemented a skip intro in the beta server?

 

I was waiting for beta ffmpeg to be released to continue. Maybe they did it with out us?

 

Edit: nevermind

LOL.. Yeah, "Skip Intro" was burned in by PlayOn and didn't do anything.

[neik 837]

On 1/13/2021 at 4:35 AM, TeamB said:

I think we are waiting for the next Beta of Emby that will have the chromaprint mux built into the  ffmpeg that ships with Emby. This way we can do away with fpcalc.

Out of curiosity:
Does the ffmpeg in the new release have the improvements you guys need for the next steps?

[samuelqwe 313]

45 minutes ago, neik said:

Out of curiosity:
Does the ffmpeg in the new release have the improvements you guys need for the next steps?

It does, so it should be possible to work on this further.