Completed Show Intro Skip Option

[rbjtech 5284]

9 minutes ago, chef said:

You guys know that there is API endpoints for all the sequence data right?

If you open the swagger, you can test it out.

Just wanted to make sure that was clear.

Not 100% sure what you mean by 'sequence' data ?

Are you saying there is a new API function to lookup data in the IntrodB ?  If yes, what's it called as I can't see anything new listed ?  

[Cheesegeezer 3104]

13 minutes ago, rbjtech said:

One to add - Could you change the log Class for the Chapter Insertion to use 'IntroSkip' instead of App - then we keep all the logging together 

Ahhh yeah sure, if i can remember how to do it lol  

[samuelqwe 317]

8 minutes ago, rbjtech said:

Not 100% sure what you mean by 'sequence' data ?

Are you saying there is a new API function to lookup data in the IntrodB ?  If yes, what's it called as I can't see anything new listed ?  

By sequence, he means intros.

It’s not new, but there have been API endpoints to lookup the intro data for quite a while now.

[crusher11 1101]

4 hours ago, chef said:

One of the things we as a community need to do, is audit the data.

It was something that I've always said needs to happen.

Just get a rough idea of how many episodes were right in a season, and how many needed the users attention.

For the season you mentioned it sounds like a couple of the episodes were wrong, most of the others that had acceptable data.

Overall it worked.

Is that a fair statement?

Well it failed on two episodes and there are only six episodes in the season, so that's a fairly high failure percentage but a fairly low total number of failures.

All other seasons, which have an identical intro to the problem season, are fine. So I'm not sure what trips up those two episodes. 

[rbjtech 5284]

4 minutes ago, samuelqwe said:

By sequence, he means intros.

It’s not new, but there have been API endpoints to lookup the intro data for quite a while now.

Cool ! - I wasn''t aware of this ..  Thanks for explaining @samuelqwe

{
  "IndexNumber": 2,
  "InternalId": 1082,
  "HasSequence": true,
  "TitleSequenceStart": "PT3M49S",
  "TitleSequenceEnd": "PT5M13S",
  "Duration": 20,
  "SeriesId": 1078,
  "SeasonId": 1079,
  "Confirmed": false,
  "Processed": true
}

 

[Cheesegeezer 3104]

1 hour ago, rbjtech said:

Cool ! - I wasn''t aware of this ..  Thanks for explaining @samuelqwe

{
  "IndexNumber": 2,
  "InternalId": 1082,
  "HasSequence": true,
  "TitleSequenceStart": "PT3M49S",
  "TitleSequenceEnd": "PT5M13S",
  "Duration": 20,
  "SeriesId": 1078,
  "SeasonId": 1079,
  "Confirmed": false,
  "Processed": true
}

 

This is from IntroSkip code

public class TitleSequenceResult
    {
        public int? IndexNumber { get; set; }
        public long InternalId { get; set; }
        public bool HasSequence { get; set; } = false;
        public TimeSpan TitleSequenceStart { get; set; }
        public TimeSpan TitleSequenceEnd { get; set; }
        public List<uint> Fingerprint {get; set;}
        public double Duration { get; set; }
        public long SeriesId { get; set; }
        public long SeasonId { get; set; }
        public bool Confirmed { get; set; }  
       
        public bool Processed { get; set; } = false;
    }

I'm about to test the plugin on my main library once the detection finishes.  In my test version it will also test the automatic chapter image extraction as well, so lets see how that goes.

At least the finger printing and detection will already be done.

Thanks for the roadmap list also mate!! 

[Cheesegeezer 3104]

3 hours ago, rbjtech said:

Latest updates below - Amazing work from @chef and @Cheesegeezer to get #5 out the door so quickly.  

(added Required Version Number - as development is happening fast, many testers are on older versions..)

# Item Description Stage Latest Update Version Required Needs Core Updates ? ETA 1 FingerPrint(Chroma) Creates a template based on a percentage of the Audio runtime. Ready More efficient use of dB but good enough for release when used with scheduler 2.0.2.8 No Complete 2 Detection Detects the 'repeating Audio' vs other episodes and calculates the IntroStart and IntroEnd Ready Possibly more accuracy but certainly good enough at the moment with 90-95% 2.0.2.8 No Complete 3 Schedule Due to the Detection needing the FP database, a schedule needs to be run to ensure the data is available. Ready but needs work Working now but needs manual config 2.0.2.8 No ? 4 Scaling Depending on library size, these tasks can take days, even weeks - so it's important it's broken into manageable chunks. Ready but needs work Needs more testing on low powered NAS devices 2.0.2.8 No ? 5 Manual Skip The IntroStart/IntroEnd can be added into the existing chapter points without needing Core development work Testing   2.0.2.9 No Complete 6 IntroSkip - Manual Detection of an 'IntroStart/IntroEnd (in emby Chapters?) should bring a 'Skip Intro?' button to the front of the playback screen. Design     Yes ? 7 IntroSkip - Auto Assuming accuracy is good enough, then #6 could also be automatic if given the option. Concept     Yes ? 8 Community dB A community dB of existing IntroStart/End points Concept     No ? 9 Specials Handling of Special Episodes - both In Season and Out of Season Concept     No ? 10 Meta-Data Refresh What happens when Metadata is refreshed - do we keep chapter points ? Design     No ? 11 External chapter file Write all chapters (incl new Intro) to an external XML or Text File Concept     No ?

?

Point 8 is a nice touch, but needs to be iron clad.  I had a lot of issues when writing the MBChapters plugin, where I tried use as many variables as possible to get a match and even then it would fall short on some movie titles.

Point 9 - I'll let chef decide on this.  it would need to compare against a known fingerprint and use that to detect the intro in the special.. But it is definitely not high on the priority list.

Point 10 - needs to just be an option for the user in config.... should be easy to code in.

Point 11 - External Chapter file - What is the purpose of this?  XML is nice to work with in C#, but we have a database and it would be good to keep it inside Emby so that other media platforms can't use our hard work.

Thanks for updating this, very much appreciated.  I'll look at point 10.  I will also look at creating playbackprogress markers that flag when the intro starts and ends and create some public accessors that the devs can just use.

Edited September 10, 2021 by Cheesegeezer

[chef 3810]

1 hour ago, rbjtech said:

Cool ! - I wasn''t aware of this ..  Thanks for explaining @samuelqwe

{
  "IndexNumber": 2,
  "InternalId": 1082,
  "HasSequence": true,
  "TitleSequenceStart": "PT3M49S",
  "TitleSequenceEnd": "PT5M13S",
  "Duration": 20,
  "SeriesId": 1078,
  "SeasonId": 1079,
  "Confirmed": false,
  "Processed": true
}

 

It's how the client can request information to show the skip button, and know where to skip to in the stream. 

[Cheesegeezer 3104]

@luke @ebr

can you comment please. 

[Luke 42078]

Buttons will require development on our side. There's nothing you can do to influence the UI. Also I realize it is convenient, but I don't think chapters is the appropriate storage mechanism. I can review all of this soon and look at what the best possible integration method would be.

[Cheesegeezer 3104]

1 hour ago, Luke said:

Buttons will require development on our side. There's nothing you can do to influence the UI. Also I realize it is convenient, but I don't think chapters is the appropriate storage mechanism. I can review all of this soon and look at what the best possible integration method would be.

Thanks fella, let us know what you need from our side please or if you can set a “post” method for us to dial in on. And please share your thoughts on integrating before we move to far 

[rbjtech 5284]

2 hours ago, Cheesegeezer said:

Point 11 - External Chapter file - What is the purpose of this?  XML is nice to work with in C#, but we have a database and it would be good to keep it inside Emby so that other media platforms can't use our hard work.

ok - so some ideas in the thread are to ultimately allow the Intro markers to be written back to the source file - making it more portable for the likes of VLC etc.  If this is not available 'on a plate' than that's fine we can cross it off the list - but considering all the data is available via an open API, I really don't see how the 'output' can be prevented from being used elsewhere ?

Remember Emby are very much followers here - Plex did this years ago and Emby's deep logic for Chromaprint is based on an open source on Github.   Yes a huge amount of work has gone into making it a reality and Emby want to protect that investment from other obvious competitors, but we cannot protect the 'result' - as it's out there in the open for all to see and use.

So in summary, if 'we' don't make the ability for external creation of these markers to modify the source media, somebody else most likely will ...  

[rbjtech 5284]

2 hours ago, Cheesegeezer said:

Point 9 - I'll let chef decide on this.  it would need to compare against a known fingerprint and use that to detect the intro in the special.. But it is definitely not high on the priority list.

@chef

If available, I believe we can use the 'special' metadata, that tells us which season the special is for and it could simply be included in the processing for that season.  ie Holiday specials and the end of every Season for example. 

On specials without this 'special' metadata, then they are more than likely unrelated to the 'episodes' (behind the scenes etc etc) and thus probably don't have a comparable Intro anyway.  

But agree it's a lower priority. 

[chef 3810]

1 hour ago, rbjtech said:

@chef

If available, I believe we can use the 'special' metadata, that tells us which season the special is for and it could simply be included in the processing for that season.  ie Holiday specials and the end of every Season for example. 

On specials without this 'special' metadata, then they are more than likely unrelated to the 'episodes' (behind the scenes etc etc) and thus probably don't have a comparable Intro anyway.  

But agree it's a lower priority. 

Absolutely.

 

Also, let's put the API endpoints behind admin , or user privileges.

Then they have to be logged in inorder to get access to the data. 

Then we  won't have to worry about other {nameless} projects utilizing the data... At least not right away

Ahhrmmm ....

 

[crusher11 1101]

Intro detection is still going, two days in...almost done though.

A handful of my basketball games have had intros detected, which just appear to be random snippets and have absolutely nothing in common with each other.

 

EDIT: Can we get a 'save all' button? Having to save after removing each intro, and then scroll all the way back down, is becoming time-consuming.

Edited September 11, 2021 by crusher11

[crusher11 1101]

The title sequence detection task has gone from 99% to 83%.

What?

[crusher11 1101]

What's the black circle for to the left of the episode image?

If I edit "Sequence Start" or "Sequence End", "Has Intro" should automatically change to 'true' if it's false.

[crusher11 1101]

The Alfred Hitchcock Hour is an absolute mess. Some episodes are wrongly flagged as having no intro, some are flagged with the intro in the wrong place, some are flagged with only the first half of the intro...it's a total shambles.

[rbjtech 5284]

9 minutes ago, crusher11 said:

The Alfred Hitchcock Hour is an absolute mess. Some episodes are wrongly flagged as having no intro, some are flagged with the intro in the wrong place, some are flagged with only the first half of the intro...it's a total shambles.

So there must be 'something' different about that show that is throwing the algorithm.

I think it's a good idea to have an 'ignore' setting at a show level though as you said above - for shows you 'know' will not have consistent Intro's (sports) and troublesome shows (such as maybe this Alfred Hitchcock Hour - for some unknown reason) - then you can exclude them from being looked at.

I'll add to the list as a concept that needs discussion and agreement on the best way forward.   

Edited September 11, 2021 by rbjtech

[rbjtech 5284]

13 hours ago, Luke said:

Buttons will require development on our side. There's nothing you can do to influence the UI. Also I realize it is convenient, but I don't think chapters is the appropriate storage mechanism. I can review all of this soon and look at what the best possible integration method would be.

Thanks @Luke - yes the use of chapters was always planned to be an intermediate step to getting the 'Skip Intro' button on the UI.   It also helps greatly with the testing/validation, as you can see and actually use the results.  The source reference/Intro data is in the sequence dB - that is unrelated to chapters.

Thanks !

[crusher11 1101]

53 minutes ago, rbjtech said:

So there must be 'something' different about that show that is throwing the algorithm.

 

I don't see why there would be. They all start with the same music for about 25 seconds. Alfred Hitchcock Presents is exactly the same intro and that seems mostly fine, although there do seem to be a small number of episodes with non-zero start times (haven't had time to go through and check what the story is with those, though).

I've converted my Hour and Presents episodes from the 25fps PAL speed of the original DVDs to 23.976fps to match the original presentation, so they're not a perfect match for the discs. Wouldn't imagine it would screw with the algorithm much though.

[rbjtech 5284]

8 minutes ago, crusher11 said:

I've converted my Hour and Presents episodes from the 25fps PAL speed of the original DVDs to 23.976fps to match the original presentation, so they're not a perfect match for the discs. Wouldn't imagine it would screw with the algorithm much though.

As long as audio is consistent - then it shouldn't matter no.  Video content is irrelevant, but if some are 23.976fps and some are 25fps in the same season - then audio WILL be different as it will have a different duration/runtime.  Over a 1 minute Intro though, I'm really not sure how 'different' it would be - I can't imagine the algorithm is that sensitive...

A question for @samuelqwe & @chef I think 

[crusher11 1101]

I just meant in terms of if anyone wanted to try and replicate the issue. 

[chef 3810]

@crusher11I have noticed that even minor changes in the audio may throw off the algorithm, and give different results for what we might hear as the exact same audio.

We do encode PCL audio stream at 16000, but there was a computer science article that mentioned encoding PCL audio at 11025.

Sam thought that the less clear the encoding, the better it was able to pick up bigger audio peaks... Or something like that.

@rbjtech what we could do is try encoding the audio at this lower level, and see if our outcomes have higher results.

I don't know if we would want to do second pass on processed items with a lower encoding to see if we are more successful.

 

[Micael456 45]

9 minutes ago, chef said:

I don't know if we would want to do second pass on processed items with a lower encoding to see if we are more successful.

 

@chef,

Could we make it conditional on average intro duration?

All intros in a season should be of the same length, thereabouts. Therefore, if- after the first pass- we get wildly differing lengths, or failure to find the intro, that could then trigger a lower fidelity scan to see if the results are improved.

IMO this would balance the increased processing time vs. the accuracy.

Also, as another sidenote, if we dial up the sensitivity this way there's always the (slim potential) it increases false positives, so no need to run it where it's currently working a-ok. Let's not make more problems for ourselves :D.

Edited September 11, 2021 by Micael456