Jump to content

Recommended Posts

Posted (edited)
1 hour ago, jptestung said:

Hi, I tried to read a couple of posts from this topic but it's huge, and I can't find what I'm looking for. What I understand is that the dev team is working on this feature, a scheduled task will browse the media files and determine intro/end timestamps. But what I don't understand is how it will decide intro/end. Is it based on the music? Credits detection ? Is it based on an open source library or custom video/audio analysis code ?

Using chroma printing technology, you can  encode a snipit of audio from an episode (10 min). Each episode in a season is chromaprinted. We use seasons because some series change their intros per season.

Once the print is taken, we take one print from an episode, and slide it over another, until they align, and we match the print. This tells us that the audio is comparable. This is the intro for both episodes. We take the timestamp for the stream and make that data available as an endpoint in the Emby api. 

 

Each episode in a season is compared with every other episode until the algorithm is able to make a match. 

Edited by chef
  • Like 2
Deathsquirrel
Posted
2 hours ago, chef said:

Using chroma printing technology, you can  encode a snipit of audio from an episode (10 min). Each episode in a season is chromaprinted. We use seasons because some series change their intros per season.

Once the print is taken, we take one print from an episode, and slide it over another, until they align, and we match the print. This tells us that the audio is comparable. This is the intro for both episodes. We take the timestamp for the stream and make that data available as an endpoint in the Emby api. 

 

Each episode in a season is compared with every other episode until the algorithm is able to make a match. 

I'm curious how well this is working for shows like The Simpsons, which have three different intro lengths that they use throughout a season.

Posted (edited)
43 minutes ago, Deathsquirrel said:

I'm curious how well this is working for shows like The Simpsons, which have three different intro lengths that they use throughout a season.

I would also be interested in that. Because each episode is compared to every other episode in the season, it is possible, that if there are two intros that are the same, these will compare and there will be a result for both episodes. However, if there is only one episode in the season with a distinctly different intro, it is possible that only the parts of the theme which match another version of it will have results.

Does that make sense? If the Simpsons theme intro is extended for one episode only, then only the parts of the theme that is comparable will get a result. 

An intro time stamp will be generated, but in that particular instance, the tine stamp for the end of the intro would be marked at the last position in the stream where the compatible episode theme would match. 

However,  if there were two episodes with extended intros, we would get a match.

Edited by chef
  • Like 1
Posted

@TeamB

I was able to get the output stream from ffmpeg, and hold the data in a memory stream, and pipe it into fpcalc. 

But, the only thing I need to figure out now is how to stop one error I get when piping the stream.

The error is "The Pipe has ended"

I think it is a buffering issue...

 

Piping ffmpeg to stream:

private static void ExtractPCMAudio(string input, int duration)
        {
            Process proc = new Process();

            proc.StartInfo.FileName = @"ffmpeg.exe";
            proc.StartInfo.Arguments = $"-t 00:3:00 -i \"{input}\" -f wav -ac 1 -acodec pcm_s16le -ar 16000 -"; //"-" Dash pipes the stream
            proc.StartInfo.UseShellExecute = false;
            proc.StartInfo.RedirectStandardInput = true;
            proc.StartInfo.RedirectStandardOutput = true;

            proc.Start();

            //Tricky, we treat the stream like a file stream, even if we aren't going to create a file
            FileStream baseStream = proc.StandardOutput.BaseStream as FileStream;

            byte[] audioBytes = null;
            int lastRead = 0;

            using (MemoryStream ms = new MemoryStream())
            {            
                byte[] buffer = new byte[4096];
                do
                {
                    lastRead = baseStream.Read(buffer, 0, buffer.Length);
                    ms.Write(buffer, 0, lastRead);
                } while (lastRead > 0);

                audioBytes = ms.ToArray();
            }
            
            proc.StandardInput.BaseStream.Close();
            
            baseStream.Close();
            FingerPrintAudio(audioBytes);
            Console.WriteLine("Done!");
            
        }

 

using byte array stream data with fpcalc:

 private static void FingerPrintAudio(byte[] audioBytes)
        {
            // Using 300 second length to get a more accurate fingerprint, but it's not required
            var fpcalc = "fpcalc.exe";
            var procStartInfo =
                new ProcessStartInfo(fpcalc, $" -raw -length 600 -json") 
                {
                    RedirectStandardOutput = true,
                    RedirectStandardInput = true,
                    RedirectStandardError  = true,
                    UseShellExecute        = false,
                    CreateNoWindow         = true,
                };

            var process = new Process {StartInfo = procStartInfo};
            process.Start();            
            
            process.StandardInput.BaseStream.Write(audioBytes, 0, audioBytes.Length); //<--This is where the piping error happens.

            string processOutput = null;
            using (var sr = new StreamWriter("Test.json"))
            {
                while ((processOutput = process.StandardOutput.ReadLine()) != null)
                {
                    sr.WriteLine(processOutput);
                    Console.WriteLine(processOutput);
                }
                sr.Close();
            }
            
        }

 

I think it is close, like really close. Piping streams or byte arrays is kind of advanced.  

Posted

@chef Hey mate! I've been following this thread with with a lot of interest (just didn't have anything useful to add). Just wondering, did you post a dll anywhere or is it just the Github repository?

thanks for the work, by the way!!!

samuelqwe
Posted
4 minutes ago, Dibbes said:

@chef Hey mate! I've been following this thread with with a lot of interest (just didn't have anything useful to add). Just wondering, did you post a dll anywhere or is it just the Github repository?

thanks for the work, by the way!!!

You can find a DLL on the GitHub page in the releases section:

https://github.com/chefbennyj1/Emby.IntroSkip/releases

samuelqwe
Posted
18 minutes ago, Dibbes said:

Yes, but that one is 8 days old... Chef made changes a few hours back... 🙂

He’s testing stuff on the side that isn’t quite ready to be released, so there isn’t currently a newer DLL available.

Posted
2 minutes ago, samuelqwe said:

He’s testing stuff on the side that isn’t quite ready to be released, so there isn’t currently a newer DLL available.

Hence my question if it's just the github respository 🙂

samuelqwe
Posted
11 minutes ago, Dibbes said:

Hence my question if it's just the github respository 🙂

That is publicly available, yes. That’s not to say that there isn’t a private group somewhere. @chef

  • Haha 1
Posted
1 minute ago, samuelqwe said:

That is publicly available, yes. That’s not to say that there isn’t a private group somewhere. @chef

Got it! Thanks!

  • Like 1
  • 2 weeks later...
nielstenboom
Posted

Hi guys,

I actually open-sourced the code for a graduation project where the goal was exactly to detect recaps, previews and opening&closing credits. You can check it out here: https://github.com/nielstenboom/recurring-content-detector

I figured maybe it can be of use! Would have to port it somehow, as it is python based.

Cheers!

  • Like 3
  • Thanks 2
Posted

@chef, how is going with the development? Anything new you can share with us? 🙂 

Posted
6 hours ago, neik said:

@chef, how is going with the development? Anything new you can share with us? 🙂 

Good things are happening 👍😃

  • Like 5
  • Thanks 2
Posted (edited)
On 1/11/2021 at 2:36 AM, nielstenboom said:

Hi guys,

I actually open-sourced the code for a graduation project where the goal was exactly to detect recaps, previews and opening&closing credits. You can check it out here: https://github.com/nielstenboom/recurring-content-detector

I figured maybe it can be of use! Would have to port it somehow, as it is python based.

Cheers!

Some cool work. What sort of performance did you experience, your approach of using frame based vectors and sequence matching is probably the most accurate way of doing this but man is it going to take a while to process a library of 6K+ episodes on a low powered NAS server 🙂

Edited by TeamB
  • Like 1
Posted
20 hours ago, neik said:

@chef, how is going with the development? Anything new you can share with us? 🙂 

I think we are waiting for the next Beta of Emby that will have the chromaprint mux built into the  ffmpeg that ships with Emby. This way we can do away with fpcalc.

  • Like 5
nielstenboom
Posted
10 hours ago, TeamB said:

Some cool work. What sort of performance did you experience, your approach of using frame based vectors and sequence matching is probably the most accurate way of doing this but man is it going to take a while to process a library of 6K+ episodes on a low powered NAS server 🙂

Yes that is indeed the tradeoff I had to make, it's not very fast.. from the top of my head I'd say about 15 mins of processing per episode on a normal laptop? Would probably be worse on a low powered NAS server haha :D

But I do believe the accuracy should be high for a skip feature to be usable, you don't want to skip actual content.

 

  • 2 weeks later...
Posted (edited)

I am no seeing a "Skip Intro" option for some shows on AndroidTV beta .70g. What's going on? I haven't updated the server past 4.5.2.0 because currently it will not expose CC's for LiveTV because the server is transcoding *.ts streams.

Did I miss something?

@chef @Luke

Edited by Sammy
samuelqwe
Posted
11 minutes ago, Sammy said:

I am no seeing a "Skip Intro" option on AndroidTV. What's going on? I haven't updated the server because currently it will not expose CC's for LiveTV because the server is transcoding *.ts streams.

Did I miss something?

@chef @Luke

Don’t think there’s really been any work at integrating this yet. The only thing we have is the plugin @chef made that can find the intro time stamps, but it doesn’t do anything with Emby directly.

So I don’t think you’ve really missed anything, because the feature hasn’t been implemented.

Posted

Must be in the source, like from PlayOn or something..

Posted (edited)

image.thumb.png.6cdb0877c649755811aa7edcb67bd346.png 

Yup. I go this particular season from PlayOn recorded from AMC..

Edited by Sammy
Posted (edited)
35 minutes ago, Sammy said:

image.thumb.png.6cdb0877c649755811aa7edcb67bd346.png 

Yup. I go this particular season from PlayOn recorded from AMC..

So, you're saying they implemented a skip intro in the beta server?

 

I was waiting for beta ffmpeg to be released to continue. Maybe they did it with out us?

 

Edit: nevermind 😂😉

Edited by chef
Posted
17 minutes ago, chef said:

So, you're saying they implemented a skip intro in the beta server?

 

I was waiting for beta ffmpeg to be released to continue. Maybe they did it with out us?

 

Edit: nevermind 😂😉

LOL.. Yeah, "Skip Intro" was burned in by PlayOn and didn't do anything.

  • Haha 2
Posted
On 1/13/2021 at 4:35 AM, TeamB said:

I think we are waiting for the next Beta of Emby that will have the chromaprint mux built into the  ffmpeg that ships with Emby. This way we can do away with fpcalc.

Out of curiosity:
Does the ffmpeg in the new release have the improvements you guys need for the next steps?

  • Like 1
samuelqwe
Posted
45 minutes ago, neik said:

Out of curiosity:
Does the ffmpeg in the new release have the improvements you guys need for the next steps?

It does, so it should be possible to work on this further. 

  • Like 2
Guest
This topic is now closed to further replies.
×
×
  • Create New...