How/where to start developing a plugin?

[EZEd 53]

I want to start working on building a plugin for Emby to call either ProjectM or Milkdrop music visualizations while music is playing in Emby.  What is the minimum tool kit to get something like that done and where/how do I start?  Any info would be helpful.

Thx

Edited May 30, 2021 by EZEd

[vdatanet 1549]

I would start here:

How to build a Server Plugin · MediaBrowser/Emby Wiki (github.com)

[EZEd 53]

okay thanks - I'll take a look.  I don't want to re-invent the wheel.  Is it known whether anyone else is already working on same or similar?

[Luke 37056]

Not that I'm aware  of. Please note you can also do this with the Emby webhooks feature and then no plugin is needed. Create your own app to receive the webhooks and then process them as desired.

[EZEd 53]

Okay Luke, I'll investigate.  Does using webhooks preclude me from implementing the solution on the streaming platforms?  I was under the impression that webhooks was much like Custom CSS and only applied to the web client or Theater.  Does webhooks functionality also extend to the streamer apps (i.e.  Android TV, Roku and Apple TV)?  If so then that is definitely the avenue I'll pursue.  I have lately (last 3 years for sure) become totally reliant on streamers as my playback client of choice and am pursuing trying to get back all of the functionality that I had in Windows Media Center; only on streamers.  Music visualization native to Emby is the last piece.  I can do it now using Android TV but it requires two applications running simultaneously (Emby and ProjectM).  It works but the two applications have no knowledge of each other so I can't get currently playing song information from Emby inside of the ProjectM application while they're both running.  If there is some kind of way to call ProjectM while inside of Emby automajically and still allow "playing now" functionality with the visualization playing in the background that would be great and is all really that I'm trying to achieve.  I'll look at webhooks unless you think it is a waste of time for my use case.

[Luke 37056]

Webhooks is part of the server and notifies you about activity from all apps.

[ebr 14910]

Hi.  I'm having trouble visualizing how either a plug-in or a web hook will get you a visualization on the app side during playback.  Unless you are writing a custom app that will be brought up while you are playing music...

[Luke 37056]

Right sorry I assume he was aware that the best he can do is launch something on the server machine.

[EZEd 53]

Yes that would be my preference; that visualization would be handled by the server on the server machine (my PC in this case). But if it's easier to do, it would be okay to use a client based process. I just wasn't thinking that a client implementation would be possible. Currently Android TV is the only client that I have that will allow simultaneous use of different apps.  With AndroidTV I can start music playing in Emby then home out to the main AndroidTV page and then select ProjectM to start the visualization.  The music continues to play through this whole process.  But because ProjectM has the visual focus of the OS I don't have any feedback from Emby on what song is playing or any of the "next up" features found in "playing now" inside of Emby.  My first thought was that all of this would have to be controlled and implemented from Emby Server (on the PC).  Thus the reason I started out asking about building a plug-in.  I am not at all familiar with the mechanics of Webhooks and not sure at all of what it would or could be used for (what is it's intended purpose).  I assumed that Webhooks was a lot like Custom CSS and was only applicable to the server and it's interaction with the web client.  That nothing done with Webhooks would translate to streaming clients like (Roku or AndroidTV).  The clients that I have are AndroidTV, Roku, SamsungTV, Theater and Web App on browser.  I would like to have the functionality in all of these clients.  I thought maybe a client based solution would be limited to the functionality of the client and not be universal.  I thought maybe a server based solution could be more universal across all/more clients and that only plug-ins would make that possible.

Do you think this is possible or is this something that would have to be implemented at the core server (code) level and not even possible with a plug-in?

Again, no point in getting started if it's not even possible.

[ebr 14910]

Hi.  It is not going to be possible for you to create visualizations on the app side from the server.

[EZEd 53]

I'm not quite tracking your response.  I am not trying to "create" visualizations on the app-side. Basically I'm trying to recreate the process that is used inside of Kodi to allow it to implement visualizations (ProjectM and others) as a plug-in like implementation.  I suppose this too is analogous to how the Windows environment does it through WMC and WMP.  I assume that since the visualization code is a standalone entity/application, it is merely being "called" through API from the music playing application.  So my first thought was that I'd just need to get ExoPlayer inside of Emby to "call" ProjectM.  And ProjectM would be brought into Emby as a Plug-in.

The "App side" of the conversation only came into context because both Emby/ExoPlayer and ProjectM are separate applications inside of AndroidTV OS that can be used simultaneously (which is the closest that we have to visualizations and simultaneous music playback and how I get this done now right now),

I had not previously thought that an API or other control mechanism could be used to interconnect two applications within the AndroidTV OS.  I was asking the question if that could be done since it was not a method that I previously considered.

But back to the original track - which would be the better way to call ProjectM from Exoplayer, by using a separate plugin or using webhooks?  I assumed that webhooks was much like custom CSS and only applied to interaction of the server and the web client. Is that true?  If it is true, then webhooks is not the path that I would like to pursue as I want to be able to use the visualizations across all client types (or at least the ones that I currently possess).  Which leaves plugin as the only method that I can see to get this done.  But if you are further saying that even a plugin will not allow me to realize this functionality either then I am not going to pursue it at all.

The only issue I have then remaining is wondering literally "how is it done in Kodi?"  I assumed that a "Kodi Add-on" was analogous to an "Emby Plug-in".  Kodi does have an add-on for ProjectM.  It is at least functional on AndroidTV so I wanted to try and recreate that implementation inside of Emby.  Again, I assumed that it would require building a plug-in for Emby.  Are Emby and Kodi different enough that something like this can not be done as a plug-in?  If so, I'll drop it.

Thx

[Luke 37056]

Quote

which would be the better way to call ProjectM from Exoplayer, by using a separate plugin or using webhooks?  

Either, but you'll be displaying it in your own application. Emby Server itself does not have any UI other than the web app.

[pünktchen 1258]

1 hour ago, EZEd said:

The only issue I have then remaining is wondering literally "how is it done in Kodi?"  I assumed that a "Kodi Add-on" was analogous to an "Emby Plug-in".

To simplify it: The big difference is Kodi is server and player and provides and addon/plugin api that allows to manipulate both sides of the app. Emby only allows server side plugins.

[EZEd 53]

On 6/2/2021 at 5:21 PM, Luke said:

Either, but you'll be displaying it in your own application. Emby Server itself does not have any UI other than the web app.

So this is to say that once called by Emby server there is no way to forward the ProjectM implementation to the other clients other than the web app?

[EZEd 53]

On 6/2/2021 at 6:01 PM, pünktchen said:

To simplify it: The big difference is Kodi is server and player and provides and addon/plugin api that allows to manipulate both sides of the app. Emby only allows server side plugins.

And this substaniates my question to Luke directly above.

It would just seem that since simultaneous playback of music and photos is working (at least on AndroidTV and Roku) that there ought to be a way to make visualizations work on these two platforms as well. Where one is merely substituting the video side to direct to visualization as opposed to a photo slideshow. So would ProjectM have to be consumed as a library object for that to happen? Something like a video file for each texture map pattern produced by ProjectM?

Edited June 3, 2021 by EZEd

[Luke 37056]

Quote

there ought to be a way to make visualizations work on these two platforms as well.

It's not that easy. The user interfaces of both apps would have to be reworked to carve out space for visualizations in a place that would make sense. And then we actually have to do the work of monitoring playback and then drawing the visualizations. And both of those would be independent efforts.

[speechles 1917]

Quote

there ought to be a way to make visualizations work on these two platforms as well.

Wouldn't it be janky doing visualizations on the server and using ffmpeg to push that to an application as a video stream? In no way would it sync with music unless it is also choreographed the audio and video to stay in sync as the audioplayer and videoplayer both played simultaneously.

Also the Roku does not allow a videoplayer object and an audioplayer object to play simultaneously. Been there tried that. As soon as you instantiate the videoplayer object as the audioplayer is playing it will generate an "cannot create videoplayer object" error.

That means the Roku application would have to draw on the screen these visualizations in real time by itself. That would mean having "beat timings" sent to the Roku so it can understand how to keep up with the beat of the music. It may not be able to render graphs of the music frequencies in real time. That is asking a lot.

What would be cool would be some kind of record player visualization with a spinning record and tone arm that is on screen as the music plays. Some kind of choreographed animations to bring stuff across the screen as music plays. That is what the Roku can do very well. It cannot do normal visualizations very well at all.

Edited June 4, 2021 by speechles

[EZEd 53]

22 hours ago, Luke said:

It's not that easy. The user interfaces of both apps would have to be reworked to carve out space for visualizations in a place that would make sense. And then we actually have to do the work of monitoring playback and then drawing the visualizations. And both of those would be independent efforts.

The way that I'm using it right now on AndroidTV is you have to give ProjectM permission to monitor (so I guess that means Google is listening, recording and spying on everything you say - but I guess you have to give a little to get a little).  I suppose somehow the ambient audio is then pushed back into ProjectM which is how the pattern map is drawn and stays in beat to the music.  It doesn't matter the source.  It could be any audio program playing through AndroidTV I guess (I actually haven't tried with Pandora and Spotify) but I of course choose to use Emby.  It works okay it's just that I don't have any feedback on what song is playing because ProjectM has visual focus.  Evidently there is facility built into ProjectM AndroidTV version because in the setup there are settings available for song information and other items...but I suppose you have use some form of software API to pass that data to ProjectM.  I was thinking maybe a plug-in to Emby could do that.

[pünktchen 1258]

31 minutes ago, EZEd said:

but I suppose you have use some form of software API to pass that data to ProjectM.  I was thinking maybe a plug-in to Emby could do that

If it has some kind of http api then you could use Emby's SessionInfo endpoint in a server plugin to get the track info and send it to ProjectM. Maybe you should ask them on Discord what's possible.

[EZEd 53]

Okay kool. I'll start there.

[ebr 14910]

15 hours ago, EZEd said:

but I suppose you have use some form of software API to pass that data to ProjectM.  I was thinking maybe a plug-in to Emby could do that.

Before you get too deep into that, try side-loading the Emby mobile app on the device and see if you get song info then...

[EZEd 53]

okay ebr will do

[EZEd 53]

Okay ebr, I tried the mobile app.  This was tried on a Jetstream AGT418. Will have to get the AndroidTV OS version later (I believe its 9 but not sure).  1) Started music playing in Emby Mobile App.  2) Homed out to Android TV home screen 3) Selected and started ProjectM.  4) Made sure that "show artist info" was selected in ProjectM.  Result: No change to screen presentation.  So the artist and song info still did not show up (only the name of the visualization, as usual).  One thing that did work that didn't with the Android TV app is that the play/pause button inside of ProjectM now controlled the song play/pause of Emby.  It would play and pause the song and probably would have skipped to next and back a song if I had had more than one queued (I didn't so will check that tomorrow).  It didn't have that functionality with just the Emby Android TV version.  But still no song or artist info in the mobile app.  Is there some other setting to make or something else to try, either within the Emby apps or ProjectM to get it to display the desired info?

I have been reading out on the web (mostly Github) about those that have tried or are trying to get visualizations working with Exoplayer.  There are some things out there but I can't tell if anyone has been successful with my specific planned use case yet.  I'll keep looking and let you know if I find anything.