New Statistics plugin

[hackthis02 93]

@Happy2Play 'Zero days' was the last one to work correctly. Not the one it's currently working on. I tried printing out the name of the item before I started the work but that's how we got the error log from @magelo "2022-09-06 10:14:30.550 Debug Statistics: CalculateMovieQualities 'movie name'". Unless that wasn't the log file but his attempt to anonymize it. But his error did go one level deeper showing me where in the function it was braking. Which I patched to no avail.

[magelo 0]

On 9/6/2022 at 8:11 PM, Happy2Play said:

@magelo Does updating to 3.0.2.5 correct the issue?  As mine worked on .3 failed on .4 and works on .5.

Hi, I was not fast enough to reboot Emby Server after 3.0.2.5 was installed (I'm on vacation)... and version 3.0.3.1 was already available. Version 3.0.3.1 is not failing anymore with Emby Server 4.8.0.8. But can I conclude the plugin is working properly?

. 

[magelo 0]

On 9/7/2022 at 10:08 AM, hackthis02 said:

@Happy2Play 'Zero days' was the last one to work correctly. Not the one it's currently working on. I tried printing out the name of the item before I started the work but that's how we got the error log from @magelo "2022-09-06 10:14:30.550 Debug Statistics: CalculateMovieQualities 'movie name'". Unless that wasn't the log file but his attempt to anonymize it. But his error did go one level deeper showing me where in the function it was braking. Which I patched to no avail.

@hackthis02, it's from my log file, but I removed the movie name, and modified some path which were revealing the account name used tu run Emby Server. Like I replied to @Happy2Play I can't confirm if 3.0.2.5 was working, but 3.0.3.1 is not failing anymore. See my post to @Happy2Play I still have strange entries in media qualities.

[hackthis02 93]

33 minutes ago, magelo said:

Hi, I was not fast enough to reboot Emby Server after 3.0.2.5 was installed (I'm on vacation)... and version 3.0.3.1 was already available. Version 3.0.3.1 is not failing anymore with Emby Server 4.8.0.8. But can I conclude the plugin is working properly?

. 

Since you're running the beta and it's not failing, I'm going to consider it working. Once 4.8 is official, if it's still happening, I'll look into a fix.

[magelo 0]

4 hours ago, hackthis02 said:

Since you're running the beta and it's not failing, I'm going to consider it working. Once 4.8 is official, if it's still happening, I'll look into a fix.

OK! Fine. It's better than having a failed plugin task every night.

[Happy2Play 8282]

Honestly do not think it is a 4.8 beta server has any difference here as I have a test system with 4.7.6.0 and 4.8.0.8 pointing to the same media that both fail on all versions beyond 3.0.0.0.  But as mentioned in other topics the media is mostly dummy media.

My primary stable 4.7.6.0 system does not fail, nor does my primary beta 4.8.0.8 system. 

Only variable I can see is real vs dummy media with my limited testing.  But will keep testing.

Edited September 8, 2022 by Happy2Play

[hackthis02 93]

11 minutes ago, Happy2Play said:

Honestly do not think it is a 4.8 beta server issues as I have a test system with 4.7.6.0 and 4.8.0.8 pointing to the same media that both fail on all versions beyond 3.0.0.0.  But as mentioned in other topics the media is mostly dummy media.

My primary stable 4.7.6.0 system does not fail, nor does my primary beta 4.8.0.8 system. 

Only variable I can see is real vs dummy media with my limited testing.  But will keep testing.

Yes, please keep testing. I've been unable to reproduce any of these issue. I would welcome any new data that might point to the underlining issue. How did you make a server with dummy media? I could try that. 

Edited September 8, 2022 by hackthis02

[Happy2Play 8282]

1 minute ago, hackthis02 said:

Yes, please keep testing. I've been unable to reproduce any of these issue. I would welcome any new data that might point to this. How did you make a server with dummy media? I could try that. 

Originally testing empty files with extensions mp4/mkv, or stub files extension .disc along with empty strm files.  So all codecs would technically be null.

Working statistics

Failing statistics

Quality goes crazy (snippet) over 4 thousand lines primarily (1   0)

Media Qualities
Movies	Episodes
1	0
1	0
1	0
1	0
1	0
1	0
1	0
1	0
1	0
1	0
1	0
1	0
1	0
1	0
1	0
1	0
1	0
1	0
1	0
1	0
1080p	40	0
4K	10	0
1	0
1	0
1	0
480p	13	0
SD	9	0
1	0
1	0
1	0

 

[Happy2Play 8282]

Also note on a clean run of statistics with no previous statistics.xml.  So to me suggests my theory on null info on these quality/codec checks.

[hackthis02 93]

1 minute ago, Happy2Play said:

Also note on a clean run of statistics with no previous statistics.xml.  So to me suggests my theory on null info on these quality/codec checks.

It may be finishing but that could because I wrap everything in giant null checks. Is there anything in your log?

[Happy2Play 8282]

1 minute ago, hackthis02 said:

It may be finishing but that could because I wrap everything in giant null checks. Is there anything in your log?

Not in reference to this.  Just the statistics task error.

2022-09-08 17:18:36.995 Error TaskManager: Error
	*** Error Report ***
	Version: 4.7.6.0
	Command line: C:\Users\Media\Desktop\Stable\system\EmbyServer.dll
	Operating system: Microsoft Windows 10.0.25193
	Framework: .NET 6.0.7
	OS/Process: x64/x64
	Runtime: C:/Users/Media/Desktop/Stable/system/System.Private.CoreLib.dll
	Processor count: 8
	Data path: C:\Users\Media\Desktop\Stable\programdata
	Application path: C:\Users\Media\Desktop\Stable\system
	System.NullReferenceException: System.NullReferenceException: Object reference not set to an instance of an object.
	   at Statistics.Helpers.Calculator.CalculateMovieCodecs()
	   at Statistics.ScheduledTasks.CalculateStatsTask.MediaBrowser.Model.Tasks.IScheduledTask.Execute(CancellationToken cancellationToken, IProgress`1 progress)
	   at Emby.Server.Implementations.ScheduledTasks.ScheduledTaskWorker.ExecuteInternal(TaskOptions options)
	Source: statistics
	TargetSite: statistics.Models.Configuration.ValueGroup CalculateMovieCodecs()
	
2022-09-08 17:18:36.995 Debug TaskManager: Calculate statistics for all users Failed after 0 minute(s) and 3 seconds

 

[Happy2Play 8282]

Testing on a smaller subset of media on a different portable I am not having any issues so far.

So this now suggest something else.  As I have empty mp4/mkv/strm with actual media.

[Happy2Play 8282]

@hackthis02 This appear to an issue with unidentified items/Shows.  Only found this in TV so far adding libraries to new portable found two libraries that caused the issue.  So emptied those libraries and readded media until error happened.

The hard part finding the Show/s causing the issue but have cleared the error on both my stable and beta once identify the problem media.  First check Folder tab and Shows tab item count but if you have any nested folder structure this will be deceiving.

One example but found 3 Shows that were causing the error.

With this folder it fails without it scan finishes.  Fixing the Series so it is Identified corrects the issue also.

Not sure if you can catch this but gives you a specific cause.

Edited September 9, 2022 by Happy2Play

[Happy2Play 8282]

Also to my knowledge I don't believe Movies can have this condition as they are always identified whether correct or not.

[Karpana 9]

Jumping in here at Happy2play's recommendation, as I'm still having this issue. 

I recognize my prod system is the unsupported emby beta 4.8.0.8. 

And from the recent posts, it seems to be an issue with specific media. 

I also understand that the media that is being shown in the debug logs is the last successfully completed media. 

How does identify the offending media ,which is apparently a TV episode.

Is there a Linux command I can run to get the files in the same order as the plugin? Or maybe a SQL query I can run?  

I'd really like to resolve this as I'm getting frustrated with nightly failure messages, but am happy to get my hands dirty to figure out which media is the problem and help the community by finding a way to identify those media that are gonna cause a break....

 

Thank you in advance. 

[Happy2Play 8282]

Not a dev so don't know if they can catch these episodes that have no SeriesName and exclude them or log them as the problem

With some google foo and trial and error this should tell you the problem items.  As it provided the specific Series Items that broke my test setups.

@Karpana Shut down Emby and made a copy of library.db to explore and see things without breaking the working db.

select Path, Name, SeriesName, ProviderIds from MediaItems where type=8 and SeriesName is null

 

[Karpana 9]

Thanks @Happy2PlayI'll give that a go. 

[magelo 0]

On 9/8/2022 at 2:00 PM, magelo said:

Hi, I was not fast enough to reboot Emby Server after 3.0.2.5 was installed (I'm on vacation)... and version 3.0.3.1 was already available. Version 3.0.3.1 is not failing anymore with Emby Server 4.8.0.8. But can I conclude the plugin is working properly?

. 

With version 3.0.3.2

I will check later if the sum is correct.

Edited September 11, 2022 by magelo

[Karpana 9]

On 9/9/2022 at 8:04 PM, Happy2Play said:

Not a dev so don't know if they can catch these episodes that have no SeriesName and exclude them or log them as the problem

With some google foo and trial and error this should tell you the problem items.  As it provided the specific Series Items that broke my test setups.

@Karpana Shut down Emby and made a copy of library.db to explore and see things without breaking the working db.

select Path, Name, SeriesName, ProviderIds from MediaItems where type=8 and SeriesName is null

 

@Happy2Play
I want to thank you for your patience with me working through this problem.
I am happy to report that your little block of SQL was exactly what I needed to locate 1 "rogue" media file that was sitting in the root directory of my TV shows folder.  Apparently, this resulted in Statistics going ... haywire and failing at 88%.

once I deleted that rogue media file (and the 2 associated metadata files), refreshed the TV shows library, statistics was able to complete to success.

Cheers!!!
Karpana

[Happy2Play 8282]

@hackthis02is there a way to catch these rogue items and log or ignore?  As these rogue unidentified items appear to be the issue only visible via folder view. 

[hackthis02 93]

1 hour ago, Happy2Play said:

@hackthis02is there a way to catch these rogue items and log or ignore?  As these rogue unidentified items appear to be the issue only visible via folder view. 

I'm on out of town for the next week and a half. While I can't promise anything. If I get some free time, I'll review your findings and see what I can do.

[dethknite 33]

Was researching an issue that caused Emby to essentially stop responding to web requests. I do not think it is the Statistics plugin, although I did see this error within about a 1 - 2 minute timespan during when it last occurred. (attached)

Looks like the Statistics plugin is not catching an exception if the directory/file DNE (in this case the directory does not exist on disk in the cache).

emby.log

[Happy2Play 8282]

3 hours ago, dethknite said:

Was researching an issue that caused Emby to essentially stop responding to web requests. I do not think it is the Statistics plugin, although I did see this error within about a 1 - 2 minute timespan during when it last occurred. (attached)

Looks like the Statistics plugin is not catching an exception if the directory/file DNE (in this case the directory does not exist on disk in the cache).

emby.log 2.97 kB · 3 downloads

Yes as this plugin is reading cache info that Emby will clean up after 30 days causes this issue.  If you were to refresh search for missing on your TV library these go way correct?

[dethknite 33]

4 hours ago, Happy2Play said:

Yes as this plugin is reading cache info that Emby will clean up after 30 days causes this issue.  If you were to refresh search for missing on your TV library these go way correct?

That is correct... I did refresh after I saw it... I just wanted to mention in case it needed a try catch. Wasn't sure if it was causing stability issues and I am still looking further as the issue with the non-responsive server hasn't happened again after that last time (2 times total over a 4 hour-ish window). Thanks for the reply.

[hackthis02 93]

@Happy2Play Finally getting to read your posts. Just to make sure I understand correctly, the issue is that Emby has one or more shows that are not identified in the DB. So, they don't have any of the normal accompanying metadata but are still included in the list of entries?