New Statistics plugin

[crossfire13148 29]

On 11/22/2024 at 5:23 PM, ChaoOos said:

Hi, you had time to look into this "bug"?

yeah once in a while, it ends up "skipping" things
it may be thetvdb hickup more though.
like then you "auto organize" if the library is scanning, apparently it seems AO "cannot find source for the media"
thinking tvdb only allows one connection per ip?

BUT i am at the point where i am ready to dump this plug in.
A ) Broken STILL, and we have pointed out this specials counting for a VERY long time.
B ) no way to exclude what libraries we want Included/excluded
by this i mean i have 2 Tv Series Folders, and it only scans 1 of them, not the other.
C ) no way to include/exclude what users are scanned
because of these, my scan takes over 4 hours....and all because i want to see a list of what episodes of a tv show i am missing

this may be a great thing for people that need all the data this looks for, but for just a home server, it is too much

and while i do program, my programming is more for electronics and scripting then it is for user software, so i am not 100% ready to start one from scratch

Edited February 5, 2025 by crossfire13148
to fix typos

[hackthis02 129]

3 hours ago, crossfire13148 said:

yeah once in a while, it ends up "skipping" things
it may be thetvdb hickup more though.
like then you "auto organize" if the library is scanning, apparently it seems AO "cannot find source for the media"
thinking tvdb only allows one connection per ip?

BUT i am at the point where i am ready to dump this plug in.
A ) Broken STILL, and we have pointed out this specials counting for a VERY long time.
B ) no way to exclude what libraries we want Included/excluded
by this i mean i have 2 Tv Series Folders, and it only scans 1 of them, not the other.
C ) no way to include/exclude what users are scanned
because of these, my scan takes over 4 hours....and all because i want to see a list of what episodes of a tv show i am missing

this may be a great thing for people that need all the data this looks for, but for just a home server, it is too much

and while i do program, my programming is more for electronics and scripting then it is for user software, so i am not 100% ready to start one from scratch

I'm sorry for the long downtime. I'm back looking at the code again.

A] I'm reviewing everyone's comments and how to report the episode progress. I should have something new in the next build.

B] I don't offer options to include/exclude libraries because as I'm aware Emby doesn't expose those to me. I make a call and get all movies/TV Shows in one big chunk. if you're not seeing everything check that all libraries are setup correctly? Maybe install another plug-in like Reports and see if it can see everything? On my side, if a developer sees this and wants to point me in the direction to API calls to get this kind of data please do so.

C] I'm also working on some performance updates to help with the speed of the plug-in. That being said, that's more on the users side then mine. Factors like library size, number of users, and system performance effect the plug-in greatly. When scanning the same library my production server takes around 5 minutes, where my dev machine takes around 30-40 seconds.

Edited February 5, 2025 by hackthis02

[adrianwi 279]

Good luck.

I'd take a look at https://github.com/CyferShepard/Jellystat, which now works with emby and is heading in the right direction as a replacement for PlexPy/Tautulli 

[hackthis02 129]

@DarWunand @crossfire13148
per your suggestions, the current build is taking specials out of the equation. So as you can see from this example. The show is registering 121, I've collected 120 and 3 specials. I have watched 120 of my collected episodes and even though I've only watched 1 of the 3 specials, it's still reporting 100%



Also two things I found.
1. TVDB has errors. I saw some cases of season finales being split into a two parter when it wasn't. Also some episodes with more then one entry. 
     You can see from the image above, Psych only has 120 episodes but TVDB had the season 7 finale as a two parter.
2. If you don't want a special to show up in a season folder or added to the total number of episodes, remove season & episode from the special's metadata.

[DarWun 349]

4 hours ago, hackthis02 said:

@DarWunand @crossfire13148
per your suggestions, the current build is taking specials out of the equation. So as you can see from this example. The show is registering 121, I've collected 120 and 3 specials. I have watched 120 of my collected episodes and even though I've only watched 1 of the 3 specials, it's still reporting 100%



Also two things I found.
1. TVDB has errors. I saw some cases of season finales being split into a two parter when it wasn't. Also some episodes with more then one entry. 
     You can see from the image above, Psych only has 120 episodes but TVDB had the season 7 finale as a two parter.
2. If you don't want a special to show up in a season folder or added to the total number of episodes, remove season & episode from the special's metadata.

Thanks @hackthis02! The plugin just updated about 30minutes ago. I'll spend some time checking things out and report back if I see anything "wonky".

With respect to your observation that TVDB has errors, I fully concur. But the musical episode at the end of Season 7 of Psych being counted as two episodes is technically not one of them. I don't remember the full details. But what I do recall is that the musical episode had been planned for a long time. Originally it was supposed to be part of Season 6. But it got pushed to Season 7. Chronologically it should have aired in the middle of Season 7. But it didn't air until several months after S07E14 due to scheduling issues at the USA Network. Even though it aired as a single two hour event in December 2013 and was marketed as a Special, it was counted as two episodes in Season 7 to complete the 16 episode order USA Network had placed for the season.

Edited February 10, 2025 by DarWun

[hackthis02 129]

19 minutes ago, DarWun said:

Thanks @hackthis02! The plugin just updated about 30minutes ago. I'll spend some time checking things out and report back if I see anything "wonky".

With respect to your observation that TVDB has errors, I fully concur. But the musical episode at the end of Season 7 of Psych being counted as two episodes is technically not one of them. I don't remember the full details. But what I do recall is that the musical episode had been planned for a long time. Originally it was supposed to be part of Season 6. But it got pushed to Season 7. Chronologically it should have aired in the middle of Season 7. But it didn't air until several months after S07E14 due to scheduling issues at the USA Network. Even though it aired as a single two hour event in December 2013 and was marketed as a Special, it was counted as two episodes in Season 7 to complete the 16 episode order USA Network had placed for the season.

You are correct about that. Seems my copy is a double episode. Can confirm, renaming my files and scanning reported the correct number of episodes. 

Edited February 10, 2025 by hackthis02

[DarWun 349]

1 hour ago, hackthis02 said:

You are correct about that. Seems my copy is a double episode. Can confirm, renaming my files and scanning reported the correct number of episodes. 

Sorry to be the bearer of "bad news" @hackthis02. But after updating to version 3.3.2.0 of the Statistics plugin, running the plugin results in the following error:

System.ArgumentException: System.ArgumentException: An item with the same key has already been added. Key: 362696

So far I haven't been able to figure out what series that key is referring to. If I can, I'll see if deleting it corrects the issue. But I thought I'd report the error for you to mull over while I do my additional troubleshooting.

I've attached the full log from when the error occurred.

Edited to add:  I've determined that the key is for The Witcher series. I'll do some testing to see if I can figure out why that series is causing an issue. I have a suspicion that it is due to the fact that I have the library split across two locations (collected episodes from Seasons 1 to 3 are in one location...there is a folder for the upcoming episodes in unaired Season 4 and beyond in another location).
 

embyserver.txt

Edited February 11, 2025 by DarWun

[DarWun 349]

10 minutes ago, DarWun said:

Sorry to be the bearer of "bad news" @hackthis02. But after updating to version 3.3.2.0 of the Statistics plugin, running the plugin results in the following error:

System.ArgumentException: System.ArgumentException: An item with the same key has already been added. Key: 362696

So far I haven't been able to figure out what series that key is referring to. If I can, I'll see if deleting it corrects the issue. But I thought I'd report the error for you to mull over while I do my additional troubleshooting.

I've attached the full log from when the error occurred.

Edited to add:  I've determined that the key is for The Witcher series. I'll do some testing to see if I can figure out why that series is causing an issue. I have a suspicion that it is due to the fact that I have the library split across two locations (collected episodes from Seasons 1 to 3 are in one location...there is a folder for the upcoming in Seasons 4 and beyond in another location).
 

embyserver.txt 91.01 kB · 1 download

@hackthis02I can confirm that the issue seems to be a result of The Witcher series being split across two folder locations in my library.

My usage scenario is that I usually record episodes, and then delete them as I watch them. For continuing series that I have bought the DVDs/Blurays for and ripped for archival purposes, I place  the ripped episodes in a folder under TV (Aired). I then have a second folder for the series under the folder TV (Unaired) where I place recorded episodes as future seasons air. I don't archive the recorded episodes. Version 3.3.2.0 of the plugin doesn't seem to like that structure.

I deleted the folder under TV (Unaired) and the plugin ran successfully. So not a big deal. The easy fix is to restructure my folders so that each series only has episodes stored in a single location. 

 

[hackthis02 129]

1 hour ago, DarWun said:

@hackthis02I can confirm that the issue seems to be a result of The Witcher series being split across two folder locations in my library.

My usage scenario is that I usually record episodes, and then delete them as I watch them. For continuing series that I have bought the DVDs/Blurays for and ripped for archival purposes, I place  the ripped episodes in a folder under TV (Aired). I then have a second folder for the series under the folder TV (Unaired) where I place recorded episodes as future seasons air. I don't archive the recorded episodes. Version 3.3.2.0 of the plugin doesn't seem to like that structure.

I deleted the folder under TV (Unaired) and the plugin ran successfully. So not a big deal. The easy fix is to restructure my folders so that each series only has episodes stored in a single location. 

 

I know I'm assuming the library is in the structure recommended here.
https://github.com/MediaBrowser/Emby/wiki/TV-Library
which is series/season/episode.  That being said, I'll look at your log and see if I can add some safeguards for libraries that don't match that structure.

 Also looking are your log you're missing a few TVDB files. 

Edited February 11, 2025 by hackthis02

[DarWun 349]

23 minutes ago, hackthis02 said:

 Also looking are your log you're missing a few TVDB files. 

My library does follow that structure. The variance is that I have two folder locations for the same series to track recorded episodes that I delete upon watching versus stored episodes that I've archived from media rips. The Statistics plugin didn't have an issue with this in earlier iterations. But with the lates update it does. I expect that I am a fringe case with respect to this library configuration. So I wouldn't expect you to waste any time trying to figure this out.

Thanks. The missing TVDB logs is something I've been puzzled by for awhile. If it wasn't for the warnings from the Statistics plug-in, I would have no idea the files were missing. The metadata and images for the series identified as having missing files all show up fine in Emby. So I never bothered to pursue it.

Edited February 11, 2025 by DarWun

[hackthis02 129]

1 hour ago, DarWun said:

Sorry to be the bearer of "bad news" @hackthis02. But after updating to version 3.3.2.0 of the Statistics plugin, running the plugin results in the following error:

System.ArgumentException: System.ArgumentException: An item with the same key has already been added. Key: 362696

So far I haven't been able to figure out what series that key is referring to. If I can, I'll see if deleting it corrects the issue. But I thought I'd report the error for you to mull over while I do my additional troubleshooting.

I've attached the full log from when the error occurred.

Edited to add:  I've determined that the key is for The Witcher series. I'll do some testing to see if I can figure out why that series is causing an issue. I have a suspicion that it is due to the fact that I have the library split across two locations (collected episodes from Seasons 1 to 3 are in one location...there is a folder for the upcoming episodes in unaired Season 4 and beyond in another location).
 

embyserver.txt 91.01 kB · 1 download

You're log doesn't point to where it broke. Could you re-run it with debug logging on?

[hackthis02 129]

2 minutes ago, DarWun said:

My library does follow that structure. The variance is that I have two folder locations for the same series to track recorded episodes that I delete upon watching versus stored episodes that I've archived from media rips. The Statistics plugin didn't have an issue with this in earlier iterations. But with the lates update it does. I expect that I am a fringe case with respect to this library configuration. So I wouldn't expect you to waste any time trying to figure this out.

Thanks. The missing TVDB logs is something I've been puzzled by for awhile. If it wasn't for the warnings from the Statistics plug-in, I would have no idea the files were missing. The metadata and images for the series identified as having missing files all show up fine in Emby. So I never bothered to pursue it.

Yeah, I still want to take a look at it since, while you might be a fringe case. It might also mean others cases outside the normal one location per library will break too.

[DarWun 349]

28 minutes ago, hackthis02 said:

You're log doesn't point to where it broke. Could you re-run it with debug logging on?

Log attached with debug logging enabled.

Images to better explain (hopefully) how my library is configured for The Witcher.

Archived episodes from bluray rips stored at this location:

Folder where recordings will be stored when future seasons air:

With this configuration, the Statistics plugin scan fails. If I move the Season 4 and 5 folders to the same folder as the archived episodes the scan completes without an issue.

 

 

embyserver (1).txt

Edited February 11, 2025 by DarWun

[crossfire13148 29]

yup, scan lasted 7 seconds, then died with:
"System.ArgumentException: System.ArgumentException: An item with the same key has already been added. Key: 83306"
and FULL of the same 
2025-02-10 20:44:52.986 Error Statistics: Could not find a part of the path '/var/lib/emby/cache/tvdb/<Number Here>/episodes-official.json'.
even though it was working 12 hours ago when it scanned, and i use the "Auto Organize" to set said directories the way emby wants them.

[DarWun 349]

3 minutes ago, crossfire13148 said:

yup, scan lasted 7 seconds, then died with:
"System.ArgumentException: System.ArgumentException: An item with the same key has already been added. Key: 83306"
and FULL of the same 
2025-02-10 20:44:52.986 Error Statistics: Could not find a part of the path '/var/lib/emby/cache/tvdb/<Number Here>/episodes-official.json'.
even though it was working 12 hours ago when it scanned, and i use the "Auto Organize" to set said directories the way emby wants them.

Do you by any chance have your library split between two locations on your server for Superjail (Series ID 83306 according to TVDB)? If so, that would be similar to what I am seeing. If not...well then I have no clue.

[hackthis02 129]

4 minutes ago, DarWun said:

Do you by any chance have your library split between two locations on your server for Superjail (Series ID 83306 according to TVDB)? If so, that would be similar to what I am seeing. If not...well then I have no clue.

I believe @crossfire13148stated early that he had a split library similar to yours.

[DarWun 349]

11 minutes ago, hackthis02 said:

I believe @crossfire13148stated early that he had a split library similar to yours.

I missed that...whoops. I'm not as much of a fringe case as I assumed I was ;-). At least not in this case...

Edited February 11, 2025 by DarWun

[crossfire13148 29]

5 minutes ago, DarWun said:

Do you by any chance have your library split between two locations on your server for Superjail (Series ID 83306 according to TVDB)? If so, that would be similar to what I am seeing. If not...well then I have no clue.

i had to actually look for it... yes they are split, and they should not have been.... dont know why it is either...
welp time to move them to 1 folder.
either way, tried to roll back to 3.2.0.0 which worked the was it was intended and used to work, but of course, plug in auto updates to the newest

side note: in EMBY there IS and option to combine shows. expecting them to be in different folders. this plug in never had an issue before with that. now it does.
AND seeing the "counting specials" was fixed before, next update broken again.

total episodes - total specials = total number of REAL episodes.

[crossfire13148 29]

4 minutes ago, hackthis02 said:

I believe @crossfire13148stated early that he had a split library similar to yours.

i have 2 Series Libraries, not SHOWS split (at least not till this show... and still dont know why auto organize did that, instead of putting in the same folder it had already put them in)

[DarkStar1977 119]

@hackthis02the new version fails to calculate statistics:

And yes I rebooted the server after upgrade and before running the calculate statistics task.

Log attached.

embyserver.txt

[hackthis02 129]

Alright for the time being I'm going to revert my optimizations. I'll leave in the new changes to show progress page though.

[robiahuca 1]

4 hours ago, DarkStar1977 said:

@hackthis02the new version fails to calculate statistics:

And yes I rebooted the server after upgrade and before running the calculate statistics task.

Log attached.

embyserver.txt 759.01 kB · 0 downloads

Same error here:

An item with the same key has already been added. Key: 422373 at System.Collections.Generic.Dictionary`2.TryInsert(TKey key, TValue value, InsertionBehavior behavior) at System.Linq.Enumerable.ToDictionary[TSource,TKey,TElement](List`1 source, Func`2 keySelector, Func`2 elementSelector, IEqualityComparer`1 comparer) at System.Linq.Enumerable.ToDictionary[TSource,TKey,TElement](IEnumerable`1 source, Func`2 keySelector, Func`2 elementSelector) at Statistics.Helpers.Calculator..ctor(User user, IUserManager userManager, ILibraryManager libraryManager, IUserDataManager userDataManager, IFileSystem fileSystem, ILogger logger, UpdateModel tvdbData) at Statistics.ScheduledTasks.CalculateStatsTask.<>c__DisplayClass19_1.b__0() at System.Threading.ExecutionContext.RunFromThreadPoolDispatchLoop(Thread threadPoolThread, ExecutionContext executionContext, ContextCallback callback, Object state) --- End of stack trace from previous location --- at System.Threading.Tasks.Task.ExecuteWithThreadLocal(Task& currentTaskSlot, Thread threadPoolThread) --- End of stack trace from previous location --- at Statistics.ScheduledTasks.CalculateStatsTask.MediaBrowser.Model.Tasks.IScheduledTask.Execute(CancellationToken cancellationToken, IProgress`1 progress) at Emby.Server.Implementations.ScheduledTasks.ScheduledTaskWorker.ExecuteInternal(TaskOptions options)

[hackthis02 129]

1 minute ago, robiahuca said:

Same error here:

An item with the same key has already been added. Key: 422373 at System.Collections.Generic.Dictionary`2.TryInsert(TKey key, TValue value, InsertionBehavior behavior) at System.Linq.Enumerable.ToDictionary[TSource,TKey,TElement](List`1 source, Func`2 keySelector, Func`2 elementSelector, IEqualityComparer`1 comparer) at System.Linq.Enumerable.ToDictionary[TSource,TKey,TElement](IEnumerable`1 source, Func`2 keySelector, Func`2 elementSelector) at Statistics.Helpers.Calculator..ctor(User user, IUserManager userManager, ILibraryManager libraryManager, IUserDataManager userDataManager, IFileSystem fileSystem, ILogger logger, UpdateModel tvdbData) at Statistics.ScheduledTasks.CalculateStatsTask.<>c__DisplayClass19_1.b__0() at System.Threading.ExecutionContext.RunFromThreadPoolDispatchLoop(Thread threadPoolThread, ExecutionContext executionContext, ContextCallback callback, Object state) --- End of stack trace from previous location --- at System.Threading.Tasks.Task.ExecuteWithThreadLocal(Task& currentTaskSlot, Thread threadPoolThread) --- End of stack trace from previous location --- at Statistics.ScheduledTasks.CalculateStatsTask.MediaBrowser.Model.Tasks.IScheduledTask.Execute(CancellationToken cancellationToken, IProgress`1 progress) at Emby.Server.Implementations.ScheduledTasks.ScheduledTaskWorker.ExecuteInternal(TaskOptions options)

Tell me if this issue persistent in 3.3.2.1

[magelo 0]

Hi! v3.3.2.0 was not able to complete collecting data. 3.3.2.1 is not failing anymore.

[DarWun 349]

@hackthis02After updating to v3.3.2.1 of the plugin, scans complete for series split across folder locations. Everything else in terms of how specials are reported seems good too. I'll report back if I notice anything odd.