New Server Plugin: Emby Data Explorer

[softworkz 3335]

This is another brand-new plugin which is available in the catalog now:

Emby Data Explorer

Important Note

Make sure to clear your browser cache after installation and restarting the server!

History

• In 2016 I was working on Metadata Providers and wanted to see and better understand which metadata is coming from which provider and which information was actually taken by Emby Server.
• In turn, I had created the MetadataViewer Plugin.
• Later, it became outdated and I was no longer around to update, so eventually it had to be pulled due to incompatibilities after the server had evolved.
• Meanwhile, there had also been an idea for a "Backstage View" plugin with the purpose to provide some low-level insights into Emby item data, but it never took off.
• This month, I came back to some work on metadata providers and I had the same problems again, which ended up in creating this plugin.
• It's a fusion of the MetadataViewer plugin and the Backstage View concept - with a name that leaves room for more...

Beta

• For now, it's available for the beta server only
• It's been done in two and a half days, so please report bugs in case you find any

How it Works

The plugin adds a new entry to the Item Context Menu:

 

This will open a dialog for data display:

 

This dialog is the one and only UI that the plugin provides. Everything is happening there.

I will describe individual features in subsequent topics.

ENJOY!

Edited November 27, 2022 by softworkz

[softworkz 3335]

Exploring Metadata Sources

Exploring the flow of metadata in Emby is one of the core features of the plugin. It provides answers to questions like:

• Where does my metadata come from?
• Which data does each metadata provider supply exactly?
• The data of which providers is actually chosen and used by Emby and why?
• How does Emby merge the data together from multiple providers?
• What influence does the selected metadata language have on the results?
• Which are the best providers for my content (in a specific library), which ones do I really need and which can I remove?
• How should in order the providers in the library settings to get best results?

Basics

General

• A metadata provider is always specific to a certain kind of item in Emby (e.g. movie, series, season, episode, person, music artist, song, etc)
• Metadata plugins typically include a smaller or larger set of providers, each for a specific kind of item

Provider types

There are three types of metadata providers in Emby:
(we're ignoring image providers here)

• Remote Metadata Providers
  Those are querying a remote service to identify and retrieve metadata information about items in Emby
  • Input: ItemLookupInfo
    A set of data which contains everything that Emby knows about and item and what might be helpful to acquire matching data. This includes things like item title, year, index numbers like season/episode numbers, etc.
  • Output: A metadata result information which includes an artificial "dummy item" to which the metadata results are applied
• Local Metadata Providers
  These are typically for reading data that is stored in the file system alongside the actual media files (e.g. Nfo files)
  • Input: ItemInfo and local path
    ItemInfo contains some basic information about an item. Also the path is important for those providers to know where to look for relevant files
  • Output: A metadata result information which includes an artificial "dummy item" to which the metadata results are applied
• Custom Metadata Providers
  These are general--purpose providers and can be used in many different ways. They are different from other providers as they get the actual item as input which they can modify directly. The Ffprobe  is a provider of this kind
  • Input: MetadataRsult with the actual item, possibly containing the resulting and merged data from other providers which have run before
  • Output: The same MetadataResult, possibly with changes made to the item directly
  • Variations
    Two variation exist for those kinds of providers:
    • Pre-Refresh: These are run before all others on metadata retrieval
    • Normal: Are run at the end after all other providers, which allows them to make changes to the metadata that came from other providers

Provider Configuration

• Remote and Local providers can be configured in the options for each library
  • Providers can be enabled or disabled
  • The order of provider invocation can be configured
• Custom providers cannot be configured as these are meant for specific tasks and they are expected to determine on their own when they should become active or not

Execution Order

It is important to understand the order of provider execution, because it depends on this order, which data Emby is finally using from those providers.
There is a fixed order by provider type which cannot be configured. It is fixed and stands above everything else. This is the order when doing a full metadata refresh or on initial metadata retrieval for an item.

1. Custom Pre-Refresh providers
2. Remote Metadata providers
   In the order which is configured in library options
3. Local Metadata providers
   In the order which is configured in library options
4. Custom Providers
   The non-pre-refresh ones

Metadata Retrieval Language

Remote metadata providers often can provide data in multiple languages. Emby needs to decide in which language it should request metadata for an item. 
This is determined over multiple levels:

1. Check whether "PreferredMedataLanguage" is set on the item directly (by the user)
   if not, then
2. Go up in the hierarchy, parent-by-parent and check whether PreferredMedataLanguage is set on any of them
   if not, then
3. Look at the library options and see whether PreferredMedataLanguage is configured for the library
   if not, then
4. Use the PreferredMedataLanguage from the global Emby Server configuration which users are choosing when setting up Emby Server

 

Sections

With that knowledge in mind, we can look at the individual section that are shown

Options Section

This section allows to control what is being shown and how it's presented:

• Request Language
  The language for which to retrieve metadata from remote providers
• Metadata Providers
  Allows to choose the set of metadata providers to be shown as columns
• Show all Fields
  Normally, only a common set of properties is shown to allow for a better overview
  Also, rows without any values are hidden by default
  Select this to show everything
• Show Full Screen
  Does what it says

 

Lookup Info Section

 

This includes (roughly) all the data that is used for querying remote providers.
The "Requested Language" corresponds to the selection made in the Options Section.
The "Configured Language" row is indicating which language would normally be queried for the item when you do a metadata refresh. It also tells why, or rather where the configured language is taken from.

What's stull planned is providing more control over this, to allow querying the providers in the same way as when the item would have been newly added and no provider ids are known yet.

 

Item/Result Properties Section

 

This section displays all the data in a single tabular view:

First Column
Shows the current values of the item in the Emby database.

Subsequent Columns
Each of those columns shows the results of a single metadata provider.

The columns are in the configured execution order from left to right!
This means that Emby would run the providers in the same order in case of a full metadata refresh.

But take also note of the Is Enabled row. Emby will only run the providers which are enabled for the library.

Understanding the Data Merging

Basically (= besides some exceptions) for remote providers, Emby is taking the first data that is available. So you can go row by row from left to right to see why a certain value is taken by Emby and not one of the values in a different column.

Controlling Column Display

To simplify the view, you can control the columns to be shown with the dropdown selection in the Options Section:

 

• All
  Shows all providers which are applicable to the item
  (excluded are custom providers which do not provide a result)
• Only Remote Providers
  Often you'll be interested in the remote providers' results only. This setting will hide all other providers
• Only Enabled Providers
  Shows only those providers as columns that Emby would use for retrieving metadata with the current configuration
• Only Enabled Remote Providers
  Shows only remote providers which are enabled
• None
  Use this, when you are not interested in metadata at all and you want to view the item's data only

 

Edited November 27, 2022 by softworkz

[softworkz 3335]

Data Display

General

• Empty Cells
  When cells are empty, it means that there's no value - or that the value is identical to the default value for an item of that type
  Numbers equal to zero are hidden as well
• Word Wrapping
  When many columns are shown, there's often not much space to display longer values and text. To make this more efficient and avoid rows getting overly tall, there's a word-wrapping by-letter. It's not nice for reading, but it saves space and allows to compare texts more quickly and easily
• Data Types
  • Booleans
    are shown as checkmark/cross symbols. This makes visual comparison much easier than having TRUE/FALSE text everywhere
  • Dates
    Are shown in ISO format: yyyy-mm-dd

Item Links

Every value that represents an Emby item is represented as link which allows to open that item in the data explorer.
Currently there is no way back. The only option to return to a previous item is to go to the Relations Section and find a link that leads to the previous item.

 

Local File and Folder Links

Values that are indicating a patch in the local file system are also shown as links:

Unfortunately, browsers do not allow anymore to open file:/// links to open from web pages.

But there's a solution. 

It works on Windows only with Chrome (probably Edge as well) after installing the following Chrome extension:

https://chrome.google.com/webstore/detail/local-explorer-file-manag/eokekhgpaakbkfkmjjcbffibkencdfkl

Once everything is set up and working, you can even DISABLE the extension, so it won't be able to read the content of the sites you are visiting. The Data Explorer is already taking care of preparing the links as needed.

You need to keep the native installation part of the extension, though.

File Links

For file links, there's also a small "folder symbol" shown. This allows you to open the containing folder in Windows Explorer rather than opening a file directly.

 

 

Edited November 27, 2022 by softworkz

[softworkz 3335]

features

[softworkz 3335]

Planned Features

(as time will allow)

• Allow querying the providers in the same way as if the item would have been newly added and no provider ids are known yet
• Show and compare media sources
• Provide back/forward navigation
• Provide link buttons to show items in the Emby UI in a new window
• Keep row headings sticky on top

 

Edited November 27, 2022 by softworkz

[rbjtech 4257]

Very nice - thanks for sharing @softworkz - I'll take a look later on today.

 

[rbjtech 4257]

Hmm - Plugin installed and emby rebooted, plugin is loading (in the logs) but no extra item in the context menu - tried multiple libraries.

Latest beta - 4.8.0.19

Would a log help ?

Edited November 27, 2022 by rbjtech

[softworkz 3335]

Press F12, go to the network tab, disable caching and press F5

[rbjtech 4257]

Rookie mistake - yep, clearing the browser cache did it - thanks !

[softworkz 3335]

Just now, rbjtech said:

Rookie mistake

Nah, I'll need to find a better way to inject the menu. Of course it should work without needing to do that. 
PS: you can enable the cache again, it's not changing over time.

[rbjtech 4257]

6 minutes ago, softworkz said:

Nah, I'll need to find a better way to inject the menu. Of course it should work without needing to do that. 
PS: you can enable the cache again, it's not changing over time.

Browser Cache was just cleared; I didn't disable it.

Of interest and for another thread, how to get the function on the context menu via a plugin is of great interest ! @chef @Cheesegeezer

Edited November 27, 2022 by rbjtech

[softworkz 3335]

1 minute ago, rbjtech said:

Browser Cache was just cleared; I didn't disable it.

Of interest and for another thread, how to get the function on the context menu via a plugin is of great interest ! @chef @Cheesegeezer

I'm not sure whether we want to make this generally available. I started an internal discussion about it, but I can't make any promises..

[rbjtech 4257]

Some immediate feedback - looks fantastic - all the info you can ever want..

but - multi-versions of the same ItemId do not show the details of the various versions.  ie it needs to show all MediaSourceID's from the ItemId ?

I think that would be useful.

 

[softworkz 3335]

4 minutes ago, rbjtech said:

but - multi-versions of the same ItemId do not show the details of the various versions.  ie it needs to show all MediaSourceID's from the ItemId ?

LOL, I had written the first lines for this and then I decided to make a cut and get an initial version out.
Of course this can be added!

BTW, when you install the "Local Explorer" extension for Chrome or Edge, you can click on the path links and open the items locally. Folders open in Windows explorer and files have that folder symbol at the right, which opens the folder where the file is located.

[rbjtech 4257]

haha sorry, I didn't mean to sound ungrateful - but I use multi-versions a lot, so it was the very first thing I looked for .. 

[softworkz 3335]

As I already have what I need, I wanted to check out first, what is being asked for. But this is surely a hot candidate.
Multiple sources could be nicely shown in a multi-column table for comparison.

[Cheesegeezer 3086]

Nice one @softworkz!!!

this will be a very useful and powerful tool i can see, especially for a lot of the power users, 

thanks for sharing bud

[Cheesegeezer 3086]

just giving it a run now. took me a while to find it, i was looking in the Metadata section but it's up in the General lol 

[ebr 14910]

Apologize for not installing to look myself, but this is restricted to Admin users, right?

[softworkz 3335]

19 minutes ago, ebr said:

Apologize for not installing to look myself, but this is restricted to Admin users, right?

Good point. Update is out.

It's not yet available for the release channel anyway.

[rbjtech 4257]

@softworkz

Appears to be broken in the latest .26 beta FYI

Thanks.

 

[softworkz 3335]

Yea I knew it will break. A lot of my plugins are broken now.

I'm still waiting for @Luketo apply the GenericUI fixes I have submitted, so I can update all the plugins.

Edited March 25, 2023 by softworkz

[rbjtech 4257]

This now appears to be fixed - After a Plugin update to v4.8.0.26

Thanks ! @Luke@softworkz@Happy2Play

[Multiverse 4]

I've tried to install it on 4.8.0.51, but after the restart threre is no plugin inside Emby. But it is inside my plugin folder!

BTW: I'm running Emby Server on Debian Bookworm (Stable).

Edited October 9, 2023 by Multiverse

[Luke 37056]

11 minutes ago, Multiverse said:

I've tried to install it on 4.8.0.51, but after the restart threre is no plugin inside Emby. But it is inside my plugin folder!

BTW: I'm running Emby Server on Debian Bookworm (Stable).

HI, what folder did you put it in?