Completed Show Intro Skip Option

[chef 3749]

I know why Agent Cater won't get sequence data for the first half of season 1. The title sequence for those episodes is under ten seconds long. We ignore possible sequences under that duration.  It's actually fine. 

[Micael456 43]

1 hour ago, chef said:

We save the TMDB ID.

An audio fingerprint encoding of just the title sequence at PCM_S16le are all pretty close to the same, or useable for matching. 

We could host those fingerprint arrays on an online resource.

 

I wonder if we could have a Google sheets file for each series, sidestep the hosting issues ;).

 

For the record, I still agree with @rbjtech that the data will be riddled with duplicates and balloon in size, but if that's Google's problem...

[chef 3749]

1 minute ago, Micael456 said:

I wonder if we could have a Google sheets file for each series, sidestep the hosting issues ;).

 

For the record, I still agree with @rbjtech that the data will be riddled with duplicates and balloon in size, but if that's Google's problem...

Right, but we can check each submission. If we have a series already, we just discard it, or maybe compare the Arrays. 

And we could...to best handle possible hacks.... record server IDs with the request so we know where the submit came from... if it where a hack. 

[Micael456 43]

30 minutes ago, chef said:

Right, but we can check each submission. If we have a series already, we just discard it, or maybe compare the Arrays. 

And we could...to best handle possible hacks.... record server IDs with the request so we know where the submit came from... if it where a hack. 

You'd have to have a hash for each full audio track, and compare that, rather than just the series. So episode one of a show might have X number of various hashes depending on the source, quality, etc. Wouldn't just be the fingerprint of the audio section... though if we compared the hashes of the audio track that *would* cut down on the fingerprint time.

 

I guess the question is would that actually be quicker than just re-fingerprinting?

 

Additionally to your point about hacking and bad data, if going down that (optional°) route we could occasionally tell the server to fingerprint anyway even if one exists for that series+audio hash, to trigger a sanity check.

 

°It would have to be an option though, let people keep it local or attempt to use community for speeds' sake.

[STR8 69]

I'm at 94.6% with the scan now so I will restart in a couple of hours to let it finish then I will restart the server.

Just a reminder I never installed the latest revision of the plugin.

[chef 3749]

24 minutes ago, STR8 said:

I'm at 94.6% with the scan now so I will restart in a couple of hours to let it finish then I will restart the server.

Just a reminder I never installed the latest revision of the plugin.

Cool! That's great! Thank you! Hopefully there will be some positive results that the guys upstairs can see. 

[chef 3749]

Should there be a confirm action in the UI?

Where you could, if you wanted, confirm that the plugin has correctly identified a seasons worth of sequences?

It wouldn't change anything more then, when the task runs, it would take into account confirmed seasons as 'skippable' earlier in the library query.

That is if the the amount of episodes in the query are equal to what is in the database (so as not to miss new additions to the library).

 

Might not be worth having as an option... 

 

Edited August 22, 2021 by chef

[samuelqwe 315]

4 hours ago, chef said:

Yes, that looks to be the correct folder.  A restart of the Emby service should load the plugin. 

@samuelqwe does that folder look right to you?

That looks like the correct folder.

[Sammy 739]

On 8/21/2021 at 5:27 AM, rbjtech said:

So based on my system, which has ~ 30,000 'Joe Average' episodes (No Anime) - the runtime breakdown via a quick export and pivot table is below - from that I can see 4 key groupings :-

< 19 mins

20-39 mins

40-55 mins

> 56 mins

So based on the above - I believe (for me) a 40 minute cut off is too soon to 'return 20' and you would scanning a lot of 40-45 min shows with unnecessary scan time - imho.

It would be good to get others runtime averages stats - as things like Anime etc would (I guess) totally change the view here.

 

 

 

 

 

I finally got a chance to create the pivot table and cluster bar graph. Here are my results. I have no anime either and have basically two clusters.

[rbjtech 4337]

7 minutes ago, Sammy said:

I finally got a chance to create the pivot table and cluster bar graph. Here are my results. I have no anime either and have basically two clusters.

Cool - That's useful Info - Your results are very similar to mine  

[Sammy 739]

One or more errors occurred. (Attempted to divide by zero.)
at System.Threading.Tasks.TaskReplicator.Run[TState](ReplicatableUserAction`1 action, ParallelOptions options, Boolean stopOnFirstFailure)
at System.Threading.Tasks.Parallel.ForWorker[TLocal](Int32 fromInclusive, Int32 toExclusive, ParallelOptions parallelOptions, Action`1 body, Action`2 bodyWithState, Func`4 bodyWithLocal, Func`1 localInit, Action`1 localFinally)
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw(Exception source)
at System.Threading.Tasks.Parallel.ThrowSingleCancellationExceptionOrOtherException(ICollection exceptions, CancellationToken cancelToken, Exception otherException)
at System.Threading.Tasks.Parallel.ForWorker[TLocal](Int32 fromInclusive, Int32 toExclusive, ParallelOptions parallelOptions, Action`1 body, Action`2 bodyWithState, Func`4 bodyWithLocal, Func`1 localInit, Action`1 localFinally)
at System.Threading.Tasks.Parallel.ForEachWorker[TSource,TLocal](TSource[] array, ParallelOptions parallelOptions, Action`1 body, Action`2 bodyWithState, Action`3 bodyWithStateAndIndex, Func`4 bodyWithStateAndLocal, Func`5 bodyWithEverything, Func`1 localInit, Action`1 localFinally)
at System.Threading.Tasks.Parallel.ForEachWorker[TSource,TLocal](IEnumerable`1 source, ParallelOptions parallelOptions, Action`1 body, Action`2 bodyWithState, Action`3 bodyWithStateAndIndex, Func`4 bodyWithStateAndLocal, Func`5 bodyWithEverything, Func`1 localInit, Action`1 localFinally)
at System.Threading.Tasks.Parallel.ForEach[TSource](IEnumerable`1 source, ParallelOptions parallelOptions, Action`2 body)
at IntroSkip.AudioFingerprinting.AudioFingerprintScheduledTask.Execute(CancellationToken cancellationToken, IProgress`1 progress)
at Emby.Server.Implementations.ScheduledTasks.ScheduledTaskWorker.ExecuteInternal(TaskOptions options)

[chef 3749]

10 minutes ago, Sammy said:

One or more errors occurred. (Attempted to divide by zero.)
at System.Threading.Tasks.TaskReplicator.Run[TState](ReplicatableUserAction`1 action, ParallelOptions options, Boolean stopOnFirstFailure)
at System.Threading.Tasks.Parallel.ForWorker[TLocal](Int32 fromInclusive, Int32 toExclusive, ParallelOptions parallelOptions, Action`1 body, Action`2 bodyWithState, Func`4 bodyWithLocal, Func`1 localInit, Action`1 localFinally)
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw(Exception source)
at System.Threading.Tasks.Parallel.ThrowSingleCancellationExceptionOrOtherException(ICollection exceptions, CancellationToken cancelToken, Exception otherException)
at System.Threading.Tasks.Parallel.ForWorker[TLocal](Int32 fromInclusive, Int32 toExclusive, ParallelOptions parallelOptions, Action`1 body, Action`2 bodyWithState, Func`4 bodyWithLocal, Func`1 localInit, Action`1 localFinally)
at System.Threading.Tasks.Parallel.ForEachWorker[TSource,TLocal](TSource[] array, ParallelOptions parallelOptions, Action`1 body, Action`2 bodyWithState, Action`3 bodyWithStateAndIndex, Func`4 bodyWithStateAndLocal, Func`5 bodyWithEverything, Func`1 localInit, Action`1 localFinally)
at System.Threading.Tasks.Parallel.ForEachWorker[TSource,TLocal](IEnumerable`1 source, ParallelOptions parallelOptions, Action`1 body, Action`2 bodyWithState, Action`3 bodyWithStateAndIndex, Func`4 bodyWithStateAndLocal, Func`5 bodyWithEverything, Func`1 localInit, Action`1 localFinally)
at System.Threading.Tasks.Parallel.ForEach[TSource](IEnumerable`1 source, ParallelOptions parallelOptions, Action`2 body)
at IntroSkip.AudioFingerprinting.AudioFingerprintScheduledTask.Execute(CancellationToken cancellationToken, IProgress`1 progress)
at Emby.Server.Implementations.ScheduledTasks.ScheduledTaskWorker.ExecuteInternal(TaskOptions options)

Okay, I know where that is. I'll grab that exception and use a default if zero is returned. It is the new duration code not finding runtimes in the metadata. 

If no runtimes are found for an episode  we remove a value from the total count (number of episodes) of the season. 

No runtimes on any episodes would result in a total of zero. 

Computers do not like dividing by zero , at all! Infact that's how (in the future) we will defeat our AI overlords! 

Thanks !

Edited August 23, 2021 by chef

[chef 3749]

• Fixed episode runtime average returns 0 error

There is some bad news...

 In order to handle the new editable table in the UI, I had to edit the database.

If you want to load this plugin please do the following, and I apologize a head of time.

(It's going to remove all the found sequences... I know... not good.) 

1. Shut down the server

2. Add the new plugin

3. Find: "Emby-Server/programdata/data/titlesequences.db" 

4. Delete it!

5. Run the server again.

 

Again I apologize, I tried to migrate the database,  but it is currently beyond my understanding. I just haven't mastered SQLite yet. I'm working  on how to properly change a data base structure.

This should be the only time we would have to delete the db file.

Very sorry.

IntroSkip_v2.zip

[softworkz 3349]

On 8/14/2021 at 6:43 PM, samuelqwe said:

@Luke @softworkz

It appears that some platforms have Chromaprint within their FFmpeg builds and others don’t.

Anything that can be done here so that all platforms have Chromaprint?

In the upcoming beta, Windows-ffmpeg versions will have Chromaprint included.

[softworkz 3349]

6 hours ago, chef said:

If you want to load this plugin please do the following, and I apologize a head of time.

(It's going to remove all the found sequences... I know... not good.) 

1. Shut down the server

2. Add the new plugin

3. Find: "Emby-Server/programdata/data/titlesequences.db" 

4. Delete it!

Why not simply rename the db file and save everybody from doing this step?  

[rbjtech 4337]

6 hours ago, chef said:

• Fixed episode runtime average returns 0 error

Again I apologize, I tried to migrate the database,  but it is currently beyond my understanding. I just haven't mastered SQLite yet. I'm working  on how to properly change a data base structure.

This should be the only time we would have to delete the db file.

Very sorry.

No need to apologise @chef - people know this is a Alpha/early Beta.   We are very grateful for the effort you are putting in to make Intro Skip a reality.

It shouldn't be run on any 'live' system as you have previously explained.  If it is, then it's just to gather success statistics, there is no practical use of the data at this stage, so it will not impact anybody's system (other than tying up CPU I guess).

[chef 3749]

3 hours ago, softworkz said:

Why not simply rename the db file and save everybody from doing this step?  

Okay.

[chef 3749]

4 hours ago, softworkz said:

In the upcoming beta, Windows-ffmpeg versions will have Chromaprint included.

Thank you. That's awesome.

[Sammy 739]

1/0 = ∞ , infinity, which a computer cannot handle under normal circumstances.

[Sammy 739]

I  tried the latest introskip plugin after shutting down the Emby server and renaming the *.db to *_old.db. it runs for a micro second then quits with nothing processed and since it doesn't fail no log is produced.

[chef 3749]

25 minutes ago, Sammy said:

I  tried the latest introskip plugin after shutting down the Emby server and renaming the *.db to *_old.db. it runs for a micro second then quits with nothing processed and since it doesn't fail no log is produced.

Sorry Sammy, you mean the server runs and crashes? Okay! A log was produced, do you think you can find it logs directory and I can see it?

I'm going to do the same and see if I can break it too. 

[Sammy 739]

My server is running just fine but the plugin doesn't run at all. Log attached.

embyserver_23082021.txt

[chef 3749]

I can't seem to break it. 

The log says we loaded okay. If the database didn't load properly we would see  "Error initializing title sequence database" in the logs.

 

was the server completely shut down when you renamed the db file?

When you say doesn't run at all, you mean the tasks don't run?

Did you run finger printing task first?

Try this: IntroSkip_v2.zip

Edited August 23, 2021 by chef

[chef 3749]

@Sammy I found it!

For some reason the DB didn't add the new column properly. 

You'll have to make sure you're completely shut down before renaming the DB file. Sorry for the hassle.

Emby is hanging on to the old DB file here:

2021-08-23 09:08:04.116 Error Server: Error processing request
	*** Error Report ***
	Version: 4.7.0.7
	Command line: C:\Users\tbeck\AppData\Roaming\Emby-Server\system\EmbyServer.dll
	Operating system: Microsoft Windows 10.0.19043
	Framework: .NET Core 3.1.17
	OS/Process: x64/x64
	Runtime: C:/Users/tbeck/AppData/Roaming/Emby-Server/system/System.Private.CoreLib.dll
	Processor count: 16
	Data path: C:\Users\tbeck\AppData\Roaming\Emby-Server\programdata
	Application path: C:\Users\tbeck\AppData\Roaming\Emby-Server\system
	SQLitePCL.pretty.SQLiteException: Error: no such column: Confirmed - SELECT ResultId, TitleSequenceStart, TitleSequenceEnd, HasSequence, Fingerprint, Duration, SeriesId, SeasonId, IndexNumber, Confirmed from TitleSequenceResults ORDER BY SeasonId desc
	SQLitePCL.pretty.SQLiteException: Exception of type 'SQLitePCL.pretty.SQLiteException' was thrown.
	   at SQLitePCL.pretty.SQLiteDatabaseConnection.PrepareStatement(String sql, String& tail)
	   at SQLitePCL.pretty.DatabaseConnection.PrepareStatement(IDatabaseConnection This, String sql)
	   at Emby.AutoOrganize.Data.SqliteTitleSequenceRepository.GetResults(TitleSequenceResultQuery query)
	   at IntroSkip.Api.TitleSequenceService.Get(SeasonTitleSequenceRequest request)
	   at Emby.Server.Implementations.Services.ServiceController.Execute(HttpListenerHost appHost, Object requestDto, IRequest req)
	   at Emby.Server.Implementations.Services.ServiceHandler.ProcessRequestAsync(HttpListenerHost appHost, IRequest httpReq, IResponse httpRes, RestPath restPath, String responseContentType, CancellationToken cancellationToken)
	   at Emby.Server.Implementations.HttpServer.HttpListenerHost.RequestHandler(IRequest httpReq, ReadOnlyMemory`1 urlString, ReadOnlyMemory`1 localPath, CancellationToken cancellationToken)
	Source: SQLitePCL.pretty
	TargetSite: SQLitePCL.pretty.IStatement PrepareStatement(System.String, System.String ByRef)

 

I had to add "Confirmed"  as a new database column because we had to keep track of what was edited in the UI  table, otherwise the task could remove and reprocess edited items. 

 

Softworx was right, I should have tried to drop the table and rename it. It would have caused less headaches. Apologies.

 

Edited August 23, 2021 by chef

[Sammy 739]

55 minutes ago, chef said:

@Sammy I found it!

For some reason the DB didn't add the new column properly. 

You'll have to make sure you're completely shut down before renaming the DB file. Sorry for the hassle.

Emby is hanging on to the old DB file here:

2021-08-23 09:08:04.116 Error Server: Error processing request
	*** Error Report ***
	Version: 4.7.0.7
	Command line: C:\Users\tbeck\AppData\Roaming\Emby-Server\system\EmbyServer.dll
	Operating system: Microsoft Windows 10.0.19043
	Framework: .NET Core 3.1.17
	OS/Process: x64/x64
	Runtime: C:/Users/tbeck/AppData/Roaming/Emby-Server/system/System.Private.CoreLib.dll
	Processor count: 16
	Data path: C:\Users\tbeck\AppData\Roaming\Emby-Server\programdata
	Application path: C:\Users\tbeck\AppData\Roaming\Emby-Server\system
	SQLitePCL.pretty.SQLiteException: Error: no such column: Confirmed - SELECT ResultId, TitleSequenceStart, TitleSequenceEnd, HasSequence, Fingerprint, Duration, SeriesId, SeasonId, IndexNumber, Confirmed from TitleSequenceResults ORDER BY SeasonId desc
	SQLitePCL.pretty.SQLiteException: Exception of type 'SQLitePCL.pretty.SQLiteException' was thrown.
	   at SQLitePCL.pretty.SQLiteDatabaseConnection.PrepareStatement(String sql, String& tail)
	   at SQLitePCL.pretty.DatabaseConnection.PrepareStatement(IDatabaseConnection This, String sql)
	   at Emby.AutoOrganize.Data.SqliteTitleSequenceRepository.GetResults(TitleSequenceResultQuery query)
	   at IntroSkip.Api.TitleSequenceService.Get(SeasonTitleSequenceRequest request)
	   at Emby.Server.Implementations.Services.ServiceController.Execute(HttpListenerHost appHost, Object requestDto, IRequest req)
	   at Emby.Server.Implementations.Services.ServiceHandler.ProcessRequestAsync(HttpListenerHost appHost, IRequest httpReq, IResponse httpRes, RestPath restPath, String responseContentType, CancellationToken cancellationToken)
	   at Emby.Server.Implementations.HttpServer.HttpListenerHost.RequestHandler(IRequest httpReq, ReadOnlyMemory`1 urlString, ReadOnlyMemory`1 localPath, CancellationToken cancellationToken)
	Source: SQLitePCL.pretty
	TargetSite: SQLitePCL.pretty.IStatement PrepareStatement(System.String, System.String ByRef)

 

I had to add "Confirmed"  as a new database column because we had to keep track of what was edited in the UI  table, otherwise the task could remove and reprocess edited items. 

 

Softworx was right, I should have tried to drop the table and rename it. It would have caused less headaches. Apologies.

 

So I don't need to rename or delete the *.db with the latest version just posted?