Automated Tagging

[bakes82 90]

4 minutes ago, roaku said:

Sorry if I sound frustrated, but I've asked for changes in this space months ago and was told by one powerful Emby overlord that everything is the way it is on purpose and isn't going to change. There's only going to be future development to *further* entrench this item structure.

Now I'm being told by another powerful Emby overlord that it's all just dust in the wind. How can we hope to develop against it?

From a DB side having multi unique keys for the same item is kinda dumb, the media source? or stream? is the uniquest as its per file, but they would need to make some junction tables and refactor some tables and move them around, Im sure there is a performance gain also if they did it as you drop alot of duplicate data from the db and the size should shrink.  I know in my library I have a bunch of duplicate files for DV/4k/1080p.  But Im also sure its a decent amount of work to refactor it for a small performance gain.

[roaku 795]

18 minutes ago, bakes82 said:

I can tell you if you put 4k in a new library and use the auto merge plugin and dont share the 4k library this approach works.  @ebrdo you guys plan to drop the multi unique ids because from a DB point it seems odd to repeat the data multi times instead of just having a junction table between the meta data ref/source

This is because with this route you're utilizing *folder* based permissions *instead* of Metadata based like tags or rating.

[bakes82 90]

3 minutes ago, roaku said:

This is because with this route you're utilizing *folder* based permissions *instead* of Metadata based like tags or rating.

Yes I know.  But guess what it already works, and odds of it not working probably not going to disappear.  The tag system seems like it hasnt been fully thought out and implemented.

[roaku 795]

9 minutes ago, bakes82 said:

Yes I know.  But guess what it already works, and odds of it not working probably not going to disappear.  The tag system seems like it hasnt been fully thought out and implemented.

No, it's just the Metadata Manager that needs further development to expose the child items for updating (and Luke has said that's coming). The metadata updates already work if you know the item id.The whitelisting/blacklisting already works.

And "we decided we didn't like the tagging system so we quit" is possibly the worst outcome for this feature request.

[roaku 795]

32 minutes ago, bakes82 said:

From a DB side having multi unique keys for the same item is kinda dumb, the media source? or stream? is the uniquest as its per file, but they would need to make some junction tables and refactor some tables and move them around, Im sure there is a performance gain also if they did it as you drop alot of duplicate data from the db and the size should shrink.  I know in my library I have a bunch of duplicate files for DV/4k/1080p.  But Im also sure its a decent amount of work to refactor it for a small performance gain.

I'm not worried about the DB schema here. That concrete got poured a long time ago.

My concern is about how the APIs expose the data. Right now, with multi-version items, the 'primary' item is favored to the point of making it nearly impossible to access the other items through /User/Items without further querying on something brittle like file 'path' that can be found in the aggregated MediaSources (I haven't actually tried this, and I'm not sure how much of it would be available over REST anyway).

But, you also *need* those other items to do certain things correctly, like semantic tagging or rating blacklisting.

I'd like for the 'primary' item to be promoted a bit so it can act like a parent and provide at least the IDs of its item children so we can act on them as needed. It's still a bit more querying in some cases, but at least it would be reliable and consistently available across APIs.

Right now, the userless /Items endpoint and InternalItemsQuery is absolutely necessary for full interaction with a library that has grouped items. They better not 'fix the glitch'.

[bakes82 90]

We should care about DB schema, its not implemented right and has issues.  Why would you ever store tags in a CSV column lol, now you cant get stats w/out querying the media items table with all the extra data and then split the column, you now have that same data in thousands of columns taking up space.  Its just horrible.  When it comes to updating and editing you now need to make sure you dont wipe out the whole data set adding/removing 1 value.  Why do I want a shitty ass DB thats dog slow because I want to use tags or some other feature that hasnt been fully thought out?  This may be fine on a smaller setups but when you have tens of thousands of movies and hundred of thousand episodes, having dupe data in the DB slows down performance and UI as it tries to pull stuff out.  It wouldnt be an issue if it was like mysql or something where you could put that on a dedicated system but its sqlite and its built in.

[roaku 795]

I said I don't care about it 'here'. That means in the context of delivering the tagging features requested in this thread.

Edited January 24, 2021 by roaku

[bakes82 90]

48 minutes ago, roaku said:

I said I don't care about it 'here'. That means in the context of delivering the tagging features requested in this thread.

Well good news, neither one of us are building the plugin   So if chef wants to keep going moar power lol.

[chef 3746]

Its alive! It will tag whatever "resolution" is in the rule... if resolution exists. It is fast too.

            foreach (var rule in rules)
            {
                Log.Info(rule.Profile.Type);
                Log.Info(rule.Profile.Container);
               
                var internalItemQuery = new InternalItemsQuery();
                if (!string.IsNullOrEmpty(rule.Profile.AudioCodec)) internalItemQuery.AudioCodecs     = new[] {rule.Profile.AudioCodec};
                if (!string.IsNullOrEmpty(rule.Profile.Container))  internalItemQuery.Containers      = new[] {rule.Profile.Container};
                if (!string.IsNullOrEmpty(rule.Profile.Rating))     internalItemQuery.OfficialRatings = new[] {rule.Profile.Rating};
                if (!string.IsNullOrEmpty(rule.Profile.VideoCodec)) internalItemQuery.VideoCodecs     = new[] {rule.Profile.VideoCodec};
                
                internalItemQuery.IncludeItemTypes = new[] { rule.Profile.Type };
                internalItemQuery.Recursive = true;               
                                
                var itemQuery = LibraryManager.GetItemsResult(internalItemQuery);

                Log.Info($"Query has {itemQuery.TotalRecordCount} items.");

                Parallel.ForEach(itemQuery.Items,new ParallelOptions() { MaxDegreeOfParallelism = 4}, (item, state) =>
                {
                    if (cancellationToken.IsCancellationRequested)
                    {
                        state.Break();
                    }

                    if (string.IsNullOrEmpty(rule.Profile.Resolution)) return; //Parallel fors return, they don't continue.
                    var videoStreams = item.GetMediaStreams();
                    foreach (var stream in videoStreams)
                    {
                        if (string.IsNullOrEmpty(stream.DisplayTitle)) continue;
                        if (stream.DisplayTitle.ToLowerInvariant().Contains(rule.Profile.Resolution))
                        {
                            //Tag this item.InternalId - it matches the "resolution"
                            // It should also match all other rule conditons because of the query
                        }
                    }
                    //Here is rule results when the rule hasn't got a resolution.
                });

            }

 

It looks like we can target HDR and SDR the same way in the future, if we want, although it is pretty much the same as 1080p and 4k... from what I can see.

 

I shall continue LOL!

 

Note: Year hasn't been added yet, because Year is an "int", and is a reference type, and always has a value. Year will be 0 if left empty in the config.

This means the "internalItemQuery" has to have a condition, or else the query will try to return everything with Year 0. Should be straight forward.

Edited January 25, 2021 by chef

[chef 3746]

@ebr there is something here I'm not sure about.

expectation:

This code would add tags to a baseItem

                            var myNewTags = new[] {"newTag1","newTag2"};

                            var tags = item.Tags.ToList(); //Arrays in c# have a set length - create a list
                            
                            tags.AddRange(myNewTags); //My new tags are now added to my tag list
                            
                            item.Tags = tags.ToArray(); //put the whole tag list back on the baseItem as string array
                             
                            LibraryManager.UpdateItem(item, item.Parent, ItemUpdateType.MetadataEdit); //Update the baseItem with the LibraryManager

 

even after a library scan, the tags are not shown on the item in the metedata editor.

did I miss a step somewhere?  

Thank you for your time.

 

Edited January 25, 2021 by chef

[bakes82 90]

3 hours ago, chef said:

@ebr there is something here I'm not sure about.

expectation:

This code would add tags to a baseItem

                            var myNewTags = new[] {"newTag1","newTag2"};

                            var tags = item.Tags.ToList(); //Arrays in c# have a set length - create a list
                            
                            tags.AddRange(myNewTags); //My new tags are now added to my tag list
                            
                            item.Tags = tags.ToArray(); //put the whole tag list back on the baseItem as string array
                             
                            LibraryManager.UpdateItem(item, item.Parent, ItemUpdateType.MetadataEdit); //Update the baseItem with the LibraryManager

 

even after a library scan, the tags are not shown on the item in the metedata editor.

did I miss a step somewhere?  

Thank you for your time.

 

If you have SQLDBBrowser you could check the library.db and look at the media items table and the tags column.  Im pretty sure thats where it should be going.  On the UI side do you need to do a refresh after clearing cache?

 

[chef 3746]

@bakes82Now I'm really confused. 

I have two versions of the this movie. One is 4k and the other is 1080p.

 

However in the metadata manager only one version shows up...

 

Edited January 26, 2021 by chef

[bakes82 90]

3 minutes ago, chef said:

@bakes82Now I'm really confused. 

I have two versions of the this movie. One is 4k and the other is 1080p.

 

However in the metadata manager only one version shows up...

 

correct, thats what happens when you merge them   the manage ui screen doesnt show the nested ones, just whatever one shows in the list first.  Why Im saying the tagging stuff isnt fully implemented yet   They assure us it read the nested values though, but could change at any time I guess.

[chef 3746]

3 minutes ago, bakes82 said:

correct, thats what happens when you merge them   the manage ui screen doesnt show the nested ones, just whatever one shows in the list first.  Why Im saying the tagging stuff isnt fully implemented yet   They assure us it read the nested values though, but could change at any time I guess.

Ah, okay. I see. I had better tread lightly while testing bulk, auto tagging then, because there are some instances where they are hidden. 

Even a select-box in the metadata manager would be good. Or I suppose adding it to the folder in the left hand items list would also be good.

Might be just me, but that seems like it should be a priority implementation.

 

[roaku 795]

23 minutes ago, chef said:

@bakes82Now I'm really confused. 

I have two versions of the this movie. One is 4k and the other is 1080p.

 

However in the metadata manager only one version shows up...

 

lol that's what I've been explaining for the last week

[chef 3746]

22 minutes ago, roaku said:

lol that's what I've been explaining for the last week

LOL oh gawd..  im sorry man. 

[bakes82 90]

3 minutes ago, chef said:

LOL oh gawd..  im sorry man. 

I figured you would get caught up eventually.  Looks like todays the day, now go back like to page 4 and read forward again

[chef 3746]

On 1/23/2021 at 10:14 AM, roaku said:

It works exactly how I've been describing.

In fact, blacklisting and whitelisting will not work properly *unless* you bypass the version groupings to tag the separated base items, as shown in the link I included.

If you only tag the primary item in a group with a blacklist ban, the alternate versions will still show up in the User's ui. This is considered correct behavior, with the announced fix being to enhance the Metadata Manager to support tagging alternate versions.

Right now, you can work around it by  modifying each movie file's nfo file and rescanning or separating each version into its own movie, modifying the Metadata as needed, then regrouping.

And here's confirmation that all movies can be retrieved independently from their grouping, both through the REST API and native C# querying.

 

I've seen how Movie Version B still appears in the UI if only Movie Version A gets a blacklist tag or ratings block in my own library.

I've worked around this defined and documented behavior in my own library.

What, precisely, are you saying isn't going to 'work like I want'?

 

Now, if we put tag based blacklisting/whitelisting and any other system features that behave like it aside, then yes, generic tags only *need* to be applied to the primary item...right now...but the Metadata Manager is eventually going to be updated to support multi-version tagging and other metadata. This auto tagging plugin will need to be updated then, anyway. There's no downside to supporting version tagging now, and it avoids the whitelist/blacklisting problem that exists right now.

 

This! 

 

And I was able to get My 4k movie blocked in a users library while still allowing 1080p.

So... Onwards!  

I have to write code to remove the tag values, and then we should try it to make sure it doesn't do anything ... Unexpected.

[bakes82 90]

1 hour ago, chef said:

This! 

 

And I was able to get My 4k movie blocked in a users library while still allowing 1080p.

So... Onwards!  

I have to write code to remove the tag values, and then we should try it to make sure it doesn't do anything ... Unexpected.

While youre in there can you change the "name" of the values in the drop down?  When using non Emby format and the auto grouping from multiple libraries/paths it gets wonky.

[chef 3746]

8 hours ago, bakes82 said:

While youre in there can you change the "name" of the values in the drop down?  When using non Emby format and the auto grouping from multiple libraries/paths it gets wonky.

Which drop down name? For resolution? 

[chef 3746]

working like a charm

 

@bakes82

I think this is what you mean

 

[bakes82 90]

 

Quote

 

@bakes82

I think this is what you mean

 

Nope, but IDT what I want can be done.  Its in the MediaSources.

You see this drop down:

I know if I follow the "Emby" format you can make the names say something.  But I run Plex/Emby/Jelly and well I'm not going to rename/move 20k movies and 200k episodes.  I use the auto group plugin so I have a Library called 1080p, 4k, 4k Remux, 4k DV.  So I would want to change the names to be like some custom format using the "parent" name AKA library name.

EX:

var test = items.Items.First().GetMediaSources(false, false,null).First().Name;

But I dont think you can save back mediasources its a auto generated.  I think @roaku said we cant change it above ;(

Edited January 26, 2021 by bakes82

[rbjtech 4287]

13 hours ago, chef said:

And I was able to get My 4k movie blocked in a users library while still allowing 1080p.

Ah really !?   That's great - because the only way to do this currently is to put the 4K media in a separate LIBRARY and permissioned it at that level - which frankly is a PITA keeping them on separate share / directory. 

[bakes82 90]

Just now, rbjtech said:

Ah really !?   That's great - because the only way to do this currently is to put the 4K media in a separate LIBRARY and permissioned it at that level - which frankly is a PITA keeping them on separate share / directory. 

How is it any diff than adding a tag for every user?

[rbjtech 4287]

Not with you 100% ? - currently a tag is applied to a media ID, not an instance ID - so by tagging a movie - you currently tag both.  If @chef is saying that you can individually tag a movie instance - then this solves the problem and you can simply 'deny' 4K to a user.