Jump to content

Audio Fringerprinting (Chromaprint) Segment detection


TeamB
 Share

Recommended Posts

14 hours ago, TeamB said:

I bet there are many media projects out there that could use audio fingerprinting and it just takes someone to come up with a good solution to a problem using it.

I have no internal knowledge but I always assumed that YouTube (Google) was using this technique to identify copyrighted content being uploaded to the platform.  I think they've been able to do that for over a decade now...  I remember some travel videos I put to music way back in 2009 and (I think) they got rejected from YT due to the music I had in the background.

Link to comment
Share on other sites

4 minutes ago, ebr said:

I have no internal knowledge but I always assumed that YouTube (Google) was using this technique to identify copyrighted content being uploaded to the platform.  I think they've been able to do that for over a decade now...  I remember some travel videos I put to music way back in 2009 and (I think) they got rejected from YT due to the music I had in the background.

Youtube's Content ID uses 'finite-state transducers and Gaussian mixture models' to produce 'an inventory of music phone units similar to phonemes in speech', which is apparently an arrangement of words that makes sense.

https://ieeexplore.ieee.org/document/4217502

  • Like 2
Link to comment
Share on other sites

sydlexius
1 hour ago, roaku said:

Youtube's Content ID uses 'finite-state transducers and Gaussian mixture models' to produce 'an inventory of music phone units similar to phonemes in speech', which is apparently an arrangement of words that makes sense.

https://ieeexplore.ieee.org/document/4217502

Sounds like something @chef and the gang should get on for the next iteration of the IntroSkip plugin! 😉  Thanks for the informative URL!  Here's a link to the full-text version.

Edited by sydlexius
added link to full-text article
Link to comment
Share on other sites

Posted (edited)
9 hours ago, ebr said:

I have no internal knowledge but I always assumed that YouTube (Google) was using this technique to identify copyrighted content being uploaded to the platform.  I think they've been able to do that for over a decade now...  I remember some travel videos I put to music way back in 2009 and (I think) they got rejected from YT due to the music I had in the background.

yes, there are a lot of different fingerprinting techniques, the chromaprint one uses fast furrier transform (FFT) to analyze the frequency spread over short time periods (sub second) to produce an ID for that block of input. This is good for sweep analysts as we are using it for intro detection as it gives you a stream of ID to match against a larger input set and allows you to find the offset of the inner sequence. But there are a number of audio fingerprint approaches and a bunch of companies out there using it for various things.

9 hours ago, roaku said:

Youtube's Content ID uses 'finite-state transducers and Gaussian mixture models' to produce 'an inventory of music phone units similar to phonemes in speech', which is apparently an arrangement of words that makes sense.

https://ieeexplore.ieee.org/document/4217502

link https://www.researchgate.net/publication/224711171_Music_Identification_with_Weighted_Finite-State_Transducers

This is an interesting approach but finely tuned to "identify" an audio stream, not really useful for audio snippet offset detection, it would be great at letting you know some audio has this snippet in it but not able to tell you where in the stream it is.

It uses Machine Learning : Gaussian Mixture Models

https://scikit-learn.org/stable/modules/mixture.html

Probably not the python SciKit library 🙂 but something similar to built models. It uses what they call "music phones" or the audio broken down into small word sized snippets and fed onto the above model.

This is an unsupervised clustering system where they use the models to build paths or phones to give you threes, this then allows you to match a paths to an item, fromt he white paper:

"where each path corresponds to a unique song"
"allow us to remove redundancies in the song graph"
 
So it is a lossy search approach so that yes you can find a match but not a location.
 
 
 

 

Edited by TeamB
  • Like 1
Link to comment
Share on other sites

thornbill
On 6/9/2022 at 8:11 PM, TeamB said:

Yeah, community cross collaboration, not sure that is going to happen unfortunately.

I have been noticing this a lot over the last few years, the general communities of these media projects are not very willing to contribute to wider community projects if it is not exclusive to their current favorite project, I see this with plugins, data sources, sharing services and even on forums with people asking for help. Things feel more closed now and people holding their cards closer to their chest more often than not.

Communities are becoming more closed and even hostile towards other communities of other tools, apps and services, to test this go and post a positive affirmation about Plex or Jellyfin on this forum, yes I know this is for Emby but all the tools are doing the same thing and what one does well the others should at lease look at, you will have 4 or 5 community watchdogs on you in seconds.  This might be being a bit harsh, I know people feel loyalty towards their tool set and community they feel they belong to and just want to stand up for it but this often turns into closed minded aggression and I have seen it time and time again on this and other forums.

You are absolutely right in my opinion. I am a Jellyfin contributor and I was incredibly hesitant to post anything here because I have seen the things that are said about Jellyfin (and its developers) in this forum. Obviously I am biased in this, but the only way for a media software to really embrace the community is for it to be run by the community. Plex and Emby are businesses so naturally they are going to do anything they can to continue making a profit. If they can incorporate features and functionality "inspired" by community developed tools in their product behind a paywall, why would they not want to do so?

I doubt Plex has any concern or animosity towards either Jellyfin or Emby given their size and I haven't really seen any hostility there. There has definitely been "bad blood" between Emby and Jellyfin. You don't have to look past the first reply to your comment to see that. Again I am clearly biased, but that does seem to be largely one sided though. Emby did everything they could to shutdown the Jellyfin fork when it first started (threatening legal action at Christmas when when some unlicensed code was inadvertently included in the fork and pushing out app updates to block connections to Jellyfin servers). Maybe someday we can coexist without the hostility.

Being a community project ourselves, Jellyfin always tries to encourage and support new community projects. If anyone would be interested in a collaboration, you can find contact methods at https://jellyfin.org/contact/.

 

On 6/9/2022 at 9:13 PM, Cheesegeezer said:

Jellyfin is stuck on rev 3 of emby and nothing has moved forward.

It's been three and a half years of active development and we just had our 40th release of Jellyfin. These kind of comments are really insulting to the hundreds of contributors who have spent countless hours of their personal time to develop a project that works for them. Can we come up with a different narrative at some point?

  • Agree 2
Link to comment
Share on other sites

6 hours ago, thornbill said:

Emby did everything they could to shutdown the Jellyfin fork when it first started

I don't think that is a fair characterization.  We did nothing to "shut you down".  We simply took steps to protect our IP.  We gave our code away freely.  That's on us.

The only gripe I have with JF was the removal of code attributions I saw committed to the repo (this was a while ago and, honestly, I haven't followed it much since).  That is just wrong.

  • Confused 1
Link to comment
Share on other sites

Posted (edited)
17 hours ago, thornbill said:

Again I am clearly biased, but that does seem to be largely one sided though. Emby did everything they could to shutdown the Jellyfin fork when it first started (threatening legal action at Christmas when when some unlicensed code was inadvertently included in the fork and pushing out app updates to block connections to Jellyfin servers). Maybe someday we can coexist without the hostility.

I agree with @ebr on this one. Yes there was a bunch of stuff that could have been handled better but there was a lot of "F**K YOU" coming from the team that was trying to set up the fork.

The OS code was freely shared and I do not believe there was any issues with forking that, while I do not feel the Emby team were overly happy with the outcome they should have seen it coming months before and taken steps to clear up all the confusion around what was open and what was closed but it ended up being a big mess that a few people took personally I feel.

The point above was as I think the Jellyfin team would agree needed to be fixed when code that should not have ended up in the forked OS project, yes it was just before Christmas I think 2 days before but in situations like this it is best to let everyone know ASAP, no need to wait to get it sorted because holidays etc.

As for changing the client apps, well that was a given, seriously you though Emby would not do this, the hundreds of hours and money they spend building them to just allow them to freely work with what is effectively a competitor?

Some of the rhetoric from both sides of the fence on this has been interesting, entertaining, frustration and disappointing to watch, the hyper-bowl sprouted from both communities was like watching my kids fighting, calling each other names and poking fun at the other with snide remarks and comments.

I feel there is a place for what is effectively two different products, Emby and Jellyfin. I truly think the Emby team dropped the ball on this, they were trying to embrace OS as part of their core Open media server principle but struggled to work out how to make it all work, OS and a commercial product. I feel if they had just done some more work on how to better build this business model to embrace OS but still build a product then they would have been in a much better place now. Something like a core OS project that was just the streaming parts and all the metadata and advanced stuff being closed source plugins. That might have worked. Anyway we have what we have.

Edited by TeamB
Link to comment
Share on other sites

Cheesegeezer
12 hours ago, TeamB said:

I agree with @ebr on this one. Yes there was a bunch of stuff that could have been handled better but there was a lot of " YOU" coming from the team that was trying to set up the fork

please edit your post and change the profanity.

12 hours ago, TeamB said:

The OS code was freely shared and I do not believe there was any issues with forking that, while I do not feel the Emby team were overly happy with the outcome they should have seen it coming months before and taken steps to clear up all the confusion around what was open and what was closed but it ended up being a big mess that a few people took personally I feel.

The point above was as I think the Jellyfin team would agree needed to be fixed when code that should not have ended up in the forked OS project, yes it was just before Christmas I think 2 days before but in situations like this it is best to let everyone know ASAP, no need to wait to get it sorted because holidays etc.

As for changing the client apps, well that was a given, seriously you though Emby would not do this, the hundreds of hours and money they spend building them to just allow them to freely work with what is effectively a competitor?

Some of the rhetoric from both sides of the fence on this has been interesting, entertaining, frustration and disappointing to watch, the hyper-bowl sprouted from both communities was like watching my kids fighting, calling each other names and poking fun at the other with snide remarks and comments.

I feel there is a place for what is effectively two different products, Emby and Jellyfin. I truly think the Emby team dropped the ball on this, they were trying to embrace OS as part of their core Open media server principle but struggled to work out how to make it all work, OS and a commercial product. I feel if they had just done some more work on how to better build this business model to embrace OS but still build a product then they would have been in a much better place now. Something like a core OS project that was just the streaming parts and all the metadata and advanced stuff being closed source plugins. That might have worked. Anyway we have what we have.

Mate…. You are flogging a dead horse here. I’ll stick by my guns and just re-iterate my points… that people want to steal code and and use it as their ideas and then some… slap a price on it. 
 

I’ve seen this over the years just in this project. 
 

so you started the movement on this, and are you working on anything and have an idea on your road map. 
 

because to me… its all empty words and lots of… someone else should sort this.

I’ve tried to be level headed and reasonable with you in PM and on forum, even asked if you wanted to be part of some of my planned projects, but to be fair I’m done. I don’t know what your objective is… but it’s just hot air

 

Edited by Cheesegeezer
Link to comment
Share on other sites

2 hours ago, Cheesegeezer said:

please edit your post and change the profanity.

done, you might need to update your quote also.

2 hours ago, Cheesegeezer said:

Mate…. You are flogging a dead horse here.

yeah probably, I guess this is not the right place.

2 hours ago, Cheesegeezer said:

that people want to steal code and and use it as their ideas and then some… slap a price on it. 

I have not seen this yet, code sharing yes but not outright copyright violations. If you see this speak up as it is not health in any OS or corporate project.

2 hours ago, Cheesegeezer said:

so you started the movement on this, and are you working on anything and have an idea on your road map. 
 

because to me… its all empty words and lots of… someone else should sort this.

I wrote a web service that can host chromaprints and metadata for intros, I hosted it and it is up and running. I wrote a command line tool to use the service and detect intros. All the code is checked in.
https://github.com/faush01/ThemeService

It is a POC but works. The actual experiment was now to see if people would be interested in using it to fill in the data, build the intro DB so it becomes usable because at the moment it is empty. Without contributions it is just an empty shell. Am I going to do all this work?, nope, I dont have the tv shows, patience or time to do it all, I can help with coding and infrastructure but that is really all. So yes in essence it is just all empty words, unless people jump on board which I dont actually think is going to happen which is also ok. If it turns people dont want this then that is perfectly fine, it was fun to put together.

2 hours ago, Cheesegeezer said:

I’ve tried to be level headed and reasonable with you in PM and on forum, even asked if you wanted to be part of some of my planned projects, but to be fair I’m done. I don’t know what your objective is… but it’s just hot air

ummmm, ok then, I guess, well I am not sure how to respond to this. I just play with code and data as a hobby in my free time,. My objectives? I guess they are just to produce code, tools and things people might find useful, does not need to be a full product or addon/plugin, just code and ideas that might be useful.

11 hours ago, ebr said:

The only gripe I have with JF was the removal of code attributions I saw committed to the repo (this was a while ago and, honestly, I haven't followed it much since).  That is just wrong.

I did not know that, bit rude.

 

Link to comment
Share on other sites

thornbill
8 hours ago, TeamB said:

The point above was as I think the Jellyfin team would agree needed to be fixed when code that should not have ended up in the forked OS project, yes it was just before Christmas I think 2 days before but in situations like this it is best to let everyone know ASAP, no need to wait to get it sorted because holidays etc.

Sure letting someone know there is an issue is one thing and definitely the sooner the better for something like that. Where it becomes rather hostile is doing so immediately before a major holiday and threatening legal action if the issue was not resolved in a week.

8 hours ago, TeamB said:

As for changing the client apps, well that was a given, seriously you though Emby would not do this, the hundreds of hours and money they spend building them to just allow them to freely work with what is effectively a competitor?

Sure it would have made sense for them to disallow connections to non-Emby servers from the start, but that didn't happen until Jellyfin started gaining some traction. Even then they felt the need to hide what they were doing by obfuscating the check. It wasn't that they were blocking any server that reported to be something other than "Emby" they were checking to see if the string returned started with the letter codes for "j" "e" "l" "l" and then displaying a message saying the server needed updating. It's a very weird approach for someone who really believes they are doing the right thing.

16 hours ago, ebr said:

The only gripe I have with JF was the removal of code attributions I saw committed to the repo (this was a while ago and, honestly, I haven't followed it much since).  That is just wrong.

To the best of my knowledge the only thing that has ever been removed was the author comments that some IDEs will automatically insert at the top of files. A list of all Emby contributors (to the best that we could determine) is maintained in a CONTRIBUTORS.md file in each repository that was forked, git history is unaltered as a proper record of attribution, and all copyright notices remain intact both in LICENSE files and file header comments where present.

Link to comment
Share on other sites

Posted (edited)
1 hour ago, thornbill said:

Sure letting someone know there is an issue is one thing and definitely the sooner the better for something like that. Where it becomes rather hostile is doing so immediately before a major holiday and threatening legal action if the issue was not resolved in a week.

I dont think it was a intended to be a malicious timing thing, the notification went out then because that is when it was found, I dont think it was someone rubbing their hands together going oh I know, we will sent it just before Christmas.

1 hour ago, thornbill said:

Sure it would have made sense for them to disallow connections to non-Emby servers from the start, but that didn't happen until Jellyfin started gaining some traction.

again I dont think timing was anything other than in the next release of the clients the server check was put in. Also it might have taken a while to realize Jellyfin was intending to ship a project that was trying to stay client API compatible and use the Emby clients, once this was clear I think the clients were updated. Just my guess.

1 hour ago, thornbill said:

It wasn't that they were blocking any server that reported to be something other than "Emby" they were checking to see if the string returned started with the letter codes for "j" "e" "l" "l" and then displaying a message saying the server needed updating.

yeah that is a little weird, a simple "server not compatible" might have been better if the client was even capable of even showing or using that info. Perhaps that was the issues, adding extra message and reason handling was not possible in the time they had to get updates out the door. At least it did return an error and not just silently fail 🙂

The projects have existed now for a few years side by side and will slowly continue to diverge, clients breaking would have happened sooner or latter, probably better sooner. The same could be said for the plugins I guess.

 

Edited by TeamB
Link to comment
Share on other sites

7 hours ago, thornbill said:

the only thing that has ever been removed was the author comments that some IDEs will automatically insert at the top of files

That is attribution of the source of that code originally.  Why do you remove it?  To make it look like it didn't originate from someone else?

7 hours ago, thornbill said:

Sure it would have made sense for them to disallow connections to non-Emby servers from the start, but that didn't happen until Jellyfin started gaining some traction. Even then they felt the need to hide what they were doing by obfuscating the check. It wasn't that they were blocking any server that reported to be something other than "Emby" they were checking to see if the string returned started with the letter codes for "j" "e" "l" "l" and then displaying a message saying the server needed updating. It's a very weird approach for someone who really believes they are doing the right thing.

There isn't another server that exactly matches our API (since we wrote it).  The reason we took this approach though was because different Emby versions didn't have something consistent for us to check at the time.  We could go back and change it now I guess but we just haven't really thought about it since.

Link to comment
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now
 Share

×
×
  • Create New...