Completed Show Intro Skip Option

[Sammy 790]

@rbjtech @chef

It seems that every time I check it is running again as the percent complete actually goes down.

 

Memory Usage:

[rbjtech 5284]

Thanks Sammy - yea I think there are some issues with large data sets - yours is @ 8Gb Mem (same as mine with presumably the default 4 detection processes running), I doubled mine to 8 and it took 16Gb Mem .. Looking at perfmon, it's spending all it's time reading the db and little else.

I'm going to start again (just shutdown and rename the FP db!) with a more controlled data set - maybe 10 shows - and see what the performance is like then - as with the current rate of progress, this task will take until 2022 ! 

@chef - Any thoughts here ?  What size of data sets did you test on ?

Edited September 1, 2021 by rbjtech

[Sammy 790]

@rbjtech  @chef

size of titlesequence.db currently

[Sammy 790]

I think Detect Title Sequence gets started again on schedule before it ever finishes so I'm going to delete the schedule and see what happens.

[chef 3810]

Yes, the db file has an index, but I also see it growing exponentially.

The db file size seems to keep it's size even if items are removed.

Then if items are added, the size grows.

Apparently this is a thing with databases. 

The table data shouldn't ever be bigger then the library.db.

It should actually be smaller, because we don't have as many columns as the library.db but we have the same amount of rows.

I have to learn what is happening. 

 

Maybe we need to VACCUME the database? 

 

Edited September 1, 2021 by chef

[rbjtech 5284]

The table data shouldn't ever be bigger then the library.db. - the problem column is the FingerPrint data - it's a huge text field - this is the issue.  The number of rows isn't the issue (I had 30K and could query in fractions of a second without the FP Data)  - it's think it's the file size/memory usage.  My titlesequence.db is now 2.2 Gb in size ..  If I remove the FP data, it's 2Mb - and everything is super responsive again.

I'm using symlinks to setup virtual libraries - testing on a set of 5 shows, then 10 etc but I *think* not trying to FP 'everything' and then Detect 'everything' is the key here to keeping the dB a manageable and thus workable size.  Maybe even keeping a 'working' db for the FP only and a 'detect' dB for the actual results would also work - I think.

I'll continue to experiment but please let me know how I can help !

[rbjtech 5284]

16 minutes ago, chef said:

Maybe we need to VACCUME the database? 

This will remove the entries marked for deletion - and yes, this is how I removed the data in the 'FingerPrint' column (replaced all values with NULL and then vacumed).  If you don't do this, the file size stays exactly the same even if the values are NULL.    But for the way we are doing it now, you NEED all that data - so you can't remove it until after you have Detected the sequence on it - and only then if the Show is finished, otherwise you should probably keep it for when you add episodes, you'll need it again no ? 

[chef 3810]

37 minutes ago, rbjtech said:

This will remove the entries marked for deletion - and yes, this is how I removed the data in the 'FingerPrint' column (replaced all values with NULL and then vacumed).  If you don't do this, the file size stays exactly the same even if the values are NULL.    But for the way we are doing it now, you NEED all that data - so you can't remove it until after you have Detected the sequence on it - and only then if the Show is finished, otherwise you should probably keep it for when you add episodes, you'll need it again no ? 

I've added the vacuum pragma to the db file.

I see what you mean, the fingerprint arrays are huge, and we would need them to continue comparing in the future.

What should we do? Let the file be huge?

As for grabbing items from the database faster, I have created a query that will request only items from a specific season.

That will speed things up a lot. Now we don't have to request the entire database every time, just what we need.

 

 

 

 

Edited September 1, 2021 by chef

[rbjtech 5284]

3 minutes ago, chef said:

I see what you mean, the fingerprint arrays are huge, and we would need them to continue comparing in the future.

What should we do? Let the file be huge?

My view is once the show has 'finished' and the Detection has been completed - then remove the FP Data - it is not of use anymore and will not be re-used ?   This needs a little though if you are trying to backfill a 'finished' show (for example) but if you have a 100% completion rate (ie no missing episodes) than I see zero reason for keeping it post Detection.

so

1. FP all data for Show A

2. Detect IntroSequence for Show A

3. Remove FP Data for Show A

4. Vacuum dB after X number of Shows

5. Re-start the process for Show B ..

  

9 minutes ago, chef said:

That will speed things up a lot. Now we don't have to request the entire database all every time, just what we need.

ok - this is cool - Currently I have to wait nearly 70 seconds to view any IntroSkip dB query (I just get a spinner for that time) - and I *think* this is also causing a major memory issue for the Detect processing as presumably, it has to do the same query ?

[neik 873]

I would like to give this a try but have one open question before starting:

Can I select which library or show the plugin fingerprints and find intros for or does the plugin run through everything it can find?

[rbjtech 5284]

5 minutes ago, neik said:

I would like to give this a try but have one open question before starting:

Can I select which library or show the plugin fingerprints and find intros for or does the plugin run through everything it can find?

All TV Shows - hence the issues ..   If you'd like to try - I would highly recommend another Emby instance running on a different Port with a small test TV library.  You can use symbolic links to create a virtual library in the OS and point your new Emby instance to that - or just copy the files if you only want to test one or two shows ..  

The plugin does not touch the Emby data or dB's, but it will hammer the resources if used on a shared system ..  

Edited September 1, 2021 by rbjtech

[chef 3810]

9 minutes ago, rbjtech said:

 

1. FP all data for Show A

2. Detect IntroSequence for Show A

3. Remove FP Data for Show A

4. Vacuum dB after X number of Shows

5. Re-start the process for Show B ..

 

Can we do it for each season? The reason I ask is because we need Show A's fingerprint to scan Show B.

If we scan an entire season and get all the proper sequences, then we can delete the FP for each episode in that season.

If the season is continuing we would have to save the FP data, but everything else could go.

 

[samuelqwe 317]

3 minutes ago, chef said:

Can we do it for each season?

Sure, why not? It would take less space overall and it’s unlikely you’ll need the data again, unless an episode is unexpectedly added.

[rbjtech 5284]

7 minutes ago, chef said:

Can we do it for each season? The reason I ask is because we need Show A's fingerprint to scan Show B.

If we scan an entire season and get all the proper sequences, then we can delete the FP for each episode in that season.

If the season is continuing we would have to save the FP data, but everything else could go.

 

Yes by 'Show' I mean 'Entire Series / All Seasons' (The term Emby uses) but I guess there is no reason why you could not do it per Season - again, as long as the Season is Complete.  But how would you know until you got to the end of it ?  

A single 'show' (and all it's seasons + episodes) is not a problem - I'm testing on 5 'shows' now (~150 episodes) and have zero issues. 

Edited September 1, 2021 by rbjtech

[chef 3810]

Ok cool, 

 

This code will do it. Just have to find where in the API Emby marks a series as 'continuing'?

var processedEpisodeResults = titleSequences.Where(s => s.SeasonId == seasonQuery.Items[seasonIndex].InternalId);
if (processedEpisodeResults.Count() == episodeQuery.TotalRecordCount) 
{
    if (processedEpisodeResults.All(result => result.HasSequence || result.Confirmed)) //Look for both confirmed or found sequences
    {
        Log.Info($"{series.Name} - {seasonQuery.Items[seasonIndex].Name} has complete title sequence profile.");
        foreach(var result in processedEpisodeResults)
        {
              result.Fingerprint.Clear();                 //Empty fingerprint List
              repo.Delete(result.InternalId.ToString());  //Remove the db entry
              repo.SaveResult(result, cancellationToken); //Add the entry back with the empty fingerprint data
         }
         continue; //move on
    }
}

 

 

@Luke Does the "IsCompleteMedia" flag the item when the series is no longer in production (the series is complete)?   or does it mean that the series DTO has a complete set of metadata? Thank you sir.

[Luke 42077]

No that's about if it's a full video file or not. It would be false if it were a recording in progress.

[rbjtech 5284]

1 hour ago, chef said:

 

@Luke Does the "IsCompleteMedia" flag the item when the series is no longer in production (the series is complete)?   or does it mean that the series DTO has a complete set of metadata? Thank you sir.

Perhaps this is deduced from the 'mediaItems' table EndDate having a value ?

edit - yea, that's confirmed - a 'Ended' shows has a value here (assume a date), while a continuing show has NULL.  While this is true - the 'Status' shows this - 1 = Continuing, 2 = Ended

Edited September 1, 2021 by rbjtech

[chef 3810]

1 minute ago, rbjtech said:

Perhaps this is deduced from the 'mediaItems' table EndDate having a value ?

Ah okay, I'll look for something like EndDate. 

[rbjtech 5284]

1 hour ago, rbjtech said:

Perhaps this is deduced from the 'mediaItems' table EndDate having a value ?

edit - yea, that's confirmed - a 'Ended' shows has a value here (assume a date), while a continuing show has NULL.

Ah sorry no it's in mediaitems - Status=1 (Continuing), Status=2 (Ended).  EndDate is stored in Metadata but doesn't appear to be used.  

there is also an 'IsMissing' Boolean - I'm just testing to see if this flags missing episodes from the entire Show - if yes, then this could also be used to maintain the FP data - maybe.. ?

[Cheesegeezer 3104]

@chefi think if you search for a metadata check in VS for InternalQueriesItem in Emby core, you should see a list of all properties that can be questioned.  I’m pretty sure that bool isContinuing or something like that is available for item.

Im not on laptop at the moment but will have a check when i can

[Cheesegeezer 3104]

It’s actually this you need @chef

InternalItemsQuery query = new InternalItemsQuery(user)

 

[chef 3810]

I can find Status in the Swagger by adding "Status" to the Filters,

 

 

 

55 minutes ago, Cheesegeezer said:

It’s actually this you need @chef

InternalItemsQuery query = new InternalItemsQuery(user)

 

oh... Thank you Cheese,

 

  I do this to get all the series:

                var seriesQuery = LibraryManager.QueryItems(new InternalItemsQuery()
                {
                    Recursive = true,
                    IncludeItemTypes = new[] { "Series" },
                    User = UserManager.Users.FirstOrDefault(user => user.Policy.IsAdministrator),
                    
                });
                
                foreach(var series in seriesQuery.Items) //This is actually a Parallel loop in the plugin
                {
                    ...
                }

 

 

...

 

This doesn't feel right... Is this what I have to do?

                            var s = LibraryManager.GetItemsResult(new InternalItemsQuery()
                            {
                                SeriesStatuses = new MediaBrowser.Model.Entities.SeriesStatus[] { MediaBrowser.Model.Entities.SeriesStatus.Ended }
                            });

 

How would I check  if the series in my loop has 'Ended'?

 

[Cheesegeezer 3104]

@chef here is the list of queries available... looks like IsAiring or HasAired might be the ones.  I'll answer your question in the next reply.

namespace MediaBrowser.Controller.Entities
{
    public class InternalItemsQuery
    {
        public InternalItemsQuery();
        public InternalItemsQuery(User user);

        public int? MinParentalRating { get; set; }
        public int? MaxParentalRating { get; set; }
        public bool? HasDeadParentId { get; set; }
        public bool? IsVirtualItem { get; set; }
        public bool? IsInProgress { get; set; }
        public long OwnerId { get; set; }
        public long[] ListIds { get; set; }
        public long[] CollectionIds { get; set; }
        public bool IsCollectionItemsQuery { get; set; }
        public long[] ParentIds { get; set; }
        public long[] ExcludeOwnerIds { get; set; }
        public string ParentType { get; set; }
        public long[] AncestorIds { get; set; }
        public long[] TopParentIds { get; set; }
        public BaseItem Parent { set; }
        public SeriesStatus[] SeriesStatuses { get; set; }
        public string ExternalId { get; set; }
        public long[] AppearsInItemIds { get; set; }
        public int? ContainsIndexNumber { get; set; }
        public int? IndexNumber { get; set; }
        public bool? HasIndexNumber { get; set; }
        public int? ParentIndexNumberNotEquals { get; set; }
        public DateTimeOffset? MinStartDate { get; set; }
        public DateTimeOffset? MaxStartDate { get; set; }
        public DateTimeOffset? MinEndDate { get; set; }
        public DateTimeOffset? MaxEndDate { get; set; }
        public bool? IsAiring { get; set; }
        public bool? HasAired { get; set; }
        public bool? IsHD { get; set; }
        public bool? Is4K { get; set; }
        public bool? IsMovie { get; set; }
        public bool? IsSports { get; set; }
        public bool? IsKids { get; set; }
        public bool? IsNews { get; set; }
        public bool? IsSeries { get; set; }
        public int? MinPlayers { get; set; }
        public int? MaxPlayers { get; set; }
        public double? MinCriticRating { get; set; }
        public double? MinCommunityRating { get; set; }
        public int? SortParentIndexNumber { get; set; }
        public int? ParentIndexNumber { get; set; }
        public long[] AlbumIds { get; set; }
        public long[] ArtistIds { get; set; }
        public long[] AlbumArtistIds { get; set; }
        public long[] FirstAlbumArtistIds { get; set; }
        public string HasNoExternalSubtitleTrackWithLanguage { get; set; }
        public string HasNoSubtitleTrackWithLanguage { get; set; }
        public string SeriesTimerId { get; set; }
        public string OwnerWithPresentationUniqueKey { get; set; }
        public long OwnerWithPresentationUniqueKeyFromItemId { get; set; }
        public bool? IsDeadPerson { get; set; }
        public bool? IsDeadItemLink { get; set; }
        public int? MaxListOrder { get; set; }
        public string HasNoInternalSubtitleTrackWithLanguage { get; set; }
        public Type FixedType { get; set; }
        public long[] HasSameGenresFromItems { get; set; }
        public string NotSyncedToTargetId { get; set; }
        public bool IsNextUpQuery { get; set; }
        public bool RecursiveTaggedItems { get; }
        public int? MinIndexNumber { get; set; }
        public int? MinSortIndexNumber { get; set; }
        public bool HasLiveTvContent { get; }
        public string WithImportedCollectionName { get; set; }
        public bool EnableGroupByMetadataKey { get; set; }
        public string HasNoAudioTrackWithLanguage { get; set; }
        public int MinSimilarityScore { get; set; }
        public DtoOptions DtoOptions { get; set; }
        public long[] AllAlbumArtistIds { get; set; }
        public long[] ContributingArtistIds { get; set; }
        public long[] ExcludeArtistIds { get; set; }
        public string AncestorWithPresentationUniqueKey { get; set; }
        public string PresentationUniqueKeyNotEquals { get; set; }
        public string SeriesPresentationUniqueKey { get; set; }
        public bool? GroupByPresentationUniqueKey { get; set; }
        public bool GroupBySeriesPresentationUniqueKey { get; set; }
        public bool GroupProgramsBySeries { get; set; }
        public bool GroupByAlbumId { get; set; }
        public bool EnableTotalRecordCount { get; set; }
        public Dictionary<string, string> ExcludeProviderIds { get; set; }
        public bool? HasChapterImages { get; set; }
        public (string, SortOrder)[] OrderBy { get; set; }
        public long? MinRunTimeTicks { get; set; }
        public DateTimeOffset? MinDateCreated { get; set; }
        public DateTimeOffset? MinDateLastSaved { get; set; }
        public DateTimeOffset? MinDateLastSavedForUser { get; set; }
        public DateTimeOffset? MaxDateLastSaved { get; set; }
        public DateTimeOffset? MaxPremiereDate { get; set; }
        public DateTimeOffset? MinPremiereDate { get; set; }
        public string[] SubtitleCodecs { get; set; }
        public string[] VideoCodecs { get; set; }
        public string[] ExcludeInheritedTags { get; set; }
        public string[] IncludeInheritedTags { get; set; }
        public string[] Genres { get; set; }
        public bool? HasOwnerId { get; set; }
        public bool? HasExtraType { get; set; }
        public Dictionary<string, string> AnyProviderIdEquals { get; set; }
        public bool? IsSpecialSeason { get; set; }
        public bool? IsUnaired { get; set; }
        public string[] ExcludeTags { get; set; }
        public bool? IsHiddenFromResume { get; set; }
        public string NameStartsWith { get; set; }
        public string NameLessThan { get; set; }
        public string SearchTerm { get; set; }
        public string MinSortName { get; set; }
        public string PresentationUniqueKey { get; set; }
        public string Path { get; set; }
        public string PathNotStartsWith { get; set; }
        public string Name { get; set; }
        public string NameStartsWithOrGreater { get; set; }
        public long[] PersonIds { get; set; }
        public string[] ExcludeItemTypes { get; set; }
        public string[] MediaTypes { get; set; }
        public bool Recursive { get; set; }
        public bool EnableAutoSort { get; set; }
        public int? StartIndex { get; set; }
        public int? Limit { get; set; }
        public User User { get; set; }
        public BaseItem SimilarTo { get; set; }
        public bool IncludeHiddenUserViews { get; set; }
        public bool ForceOriginalFolders { get; set; }
        public string[] IncludeItemTypes { get; set; }
        public bool? IsFolder { get; set; }
        public bool? IsLiked { get; set; }
        public bool? IsPlayed { get; set; }
        public bool? IsResumable { get; set; }
        public bool AllowOwnedItems { get; set; }
        public bool AllowGlobalLists { get; set; }
        public bool TagNamesCached { get; set; }
        public ItemLinkType[] AllowedGlobalItemLinkTypes { get; set; }
        public bool? IsRecordingsFolder { get; set; }
        public bool? IsFavorite { get; set; }
        public Dictionary<string, string> WithImportedCollectionMatchingProviderId { get; set; }
        public long[] ItemIds { get; set; }
        public long[] ExcludeItemIds { get; set; }
        public long[] TagIds { get; set; }
        public ImageType[] ImageTypes { get; set; }
        public UnratedItem[] BlockUnratedItems { get; set; }
        public ExtraType[] ExtraTypes { get; set; }
        public int[] Years { get; set; }
        public string[] Tags { get; set; }
        public bool? HasChildIsMovie { get; set; }
        public bool? HasChildIsKids { get; set; }
        public long[] GenreIds { get; set; }
        public bool? HasChildIsNews { get; set; }
        public string[] OfficialRatings { get; set; }
        public string[] Containers { get; set; }
        public string[] HasAnyProviderId { get; set; }
        public string[] MissingAnyProviderId { get; set; }
        public string[] MissingAllProviderIds { get; set; }
        public string[] AudioCodecs { get; set; }
        public string[] AudioLanguages { get; set; }
        public string[] SubtitleLanguages { get; set; }
        public bool? HasChildIsSports { get; set; }
        public long[] ListItemIds { get; set; }
        public long[] StudioIds { get; set; }
        public bool? HasParentalRating { get; set; }
        public long AdjacentTo { get; set; }
        public PersonType[] PersonTypes { get; set; }
        public int? MinWidth { get; set; }
        public int? MinHeight { get; set; }
        public int? MaxWidth { get; set; }
        public int? MaxHeight { get; set; }
        public ItemLinkType? GroupItemsIntoItemLinkType { get; set; }
        public bool? Is3D { get; set; }
        public bool? HasProductionYear { get; set; }
        public bool? IsLocked { get; set; }
        public bool? HasOfficialRating { get; set; }
        public bool? HasPath { get; set; }
        public bool? HasPrefix { get; set; }
        public bool? HasContainer { get; set; }
        public bool? HasThemeSong { get; set; }
        public bool? HasThemeVideo { get; set; }
        public bool? HasSubtitles { get; set; }
        public bool? HasExtra { get; set; }
        public bool? HasTrailer { get; set; }
        public bool? HasOverview { get; set; }
        public bool IncludeLiveTVView { get; set; }

        public void SetUser(User user);

 

[Cheesegeezer 3104]

1 hour ago, chef said:

I can find Status in the Swagger by adding "Status" to the Filters,

This doesn't feel right... Is this what I have to do?

                            var s = LibraryManager.GetItemsResult(new InternalItemsQuery()
                            {
                                SeriesStatuses = new MediaBrowser.Model.Entities.SeriesStatus[] { MediaBrowser.Model.Entities.SeriesStatus.Ended }
                            });

 

How would I check  if the series in my loop has 'Ended'?

 

 

You'll have to excuse my termanology, i'm a bit crap with it.

So create a Method, you can pass the BaseItem if it helps with your code but definitiely pass the User to the Method Args

private readonly ILibraryManager _libraryManager;
//you need to initial the above in your constructor

private List<YourClass> GetCurrentlyAiringShows (Baseitem item, User user)
{
		List<YourClass> airedShows = new List<YourClass>();
  		
  		InternalItemsQuery query = new InternalItemsQuery(user);       
        	query.IsAiring = true;
  		//add more query lines for what you want
  		InternalItemsQuery results = query;  //condsolidate all the queries to one field
  		
  		//get the query results, selecting only the ones that meet your query criteria and add the itemid's to your list.
  		airedShows.AddRange(_libraryManager.GetInternalItemIds(results).Select(delegate (long i)
        	{
        		YourClass class = new YourClass();
          		class.ItemId = i
                	return class;
        	}
                                                                               
            //for checking what is returned
            foreach (var doofa in airedShows)
            {
              _logger.Info("blah blah = {0}", doofa.ItemId)
            }
                                                                                
            return airedShows;                                                                
}  

 

Edited September 2, 2021 by Cheesegeezer

[chef 3810]

1 hour ago, Cheesegeezer said:

 

You'll have to excuse my termanology, i'm a bit crap with it.

So create a Method, you can pass the BaseItem if it helps with your code but definitiely pass the User to the Method Args

private readonly ILibraryManager _libraryManager;
//you need to initial the above in your constructor

private List<YourClass> GetCurrentlyAiringShows (Baseitem item, User user)
{
		List<YourClass> airedShows = new List<YourClass>();
  		
  		InternalItemsQuery query = new InternalItemsQuery(user);       
        	query.IsAiring = true;
  		//add more query lines for what you want
  		InternalItemsQuery results = query;  //condsolidate all the queries to one field
  		
  		//get the query results, selecting only the ones that meet your query criteria and add the itemid's to your list.
  		airedShows.AddRange(_libraryManager.GetInternalItemIds(results).Select(delegate (long i)
        	{
        		YourClass class = new YourClass();
          		class.ItemId = i
                	return class;
        	}
                                                                               
            //for checking what is returned
            foreach (var doofa in airedShows)
            {
              _logger.Info("blah blah = {0}", doofa.ItemId)
            }
                                                                                
            return airedShows;                                                                
}  

 

So I do have to make a new query for series that are ended and check to see if it is in the list.

 

Thank you sir.

Edited September 2, 2021 by chef