Developing a standalone embystat server

[StevenDTX 2]

Seems indeed like there are shows in more then one library, but @@rdmiller13 mentioned he doesn't have any duplicate libraries.

Someone else posted this issue on github as well and I created a test version for it posted here: https://github.com/mregni/EmbyStat/issues/1007#issuecomment-561301424

 

Feel free to test it out and report back to me if this version fixed your problem. Be aware, this is a Windows x64 build only.

 

Indeed, I have duplicate shows in two libraries.  I keep my 4k content in separate TV-UHD and Movies-UHD libraries.

 

Unfortunately, I am running on Docker, so I cant test this version.

[anthonws 40]

So, I'm using the latest version, 0.2.0-beta.14, but I can't see any history of what users saw. Is it still under development?

 

Are the core capabilities right now the stats about the available media?

 

Thanks!

 

anthonws.

[jaycedk 382]

Yes that is under reconstruction

[Diedrich 355]

reggi should put up one of those"Under Construction" images from the 90's. Bonus points for it being one of those gifs where the worker is digging.

[jaycedk 382]

he he yes, that would answer a lot of forum questions

[WilhelmStroker 96]

just updated, and the scan fails with the following error:

 

2019-12-05 20:34:23.1002 [INFO] MEDIASYNC-JOB Found 2141 missing episodes for show Top of the Pops (UK) 
2019-12-05 20:34:23.1145 [INFO] MEDIASYNC-JOB Processed (83/94) Top of the Pops (UK) 
2019-12-05 20:34:23.1951 [ERROR] Error while running job LiteDB.LiteException: Cannot insert duplicate key in unique index '_id'. The duplicate value is '453207'.
   at LiteDB.IndexService.AddNode(CollectionIndex index, BsonValue key, Byte level, IndexNode last)
   at LiteDB.IndexService.AddNode(CollectionIndex index, BsonValue key, IndexNode last)
   at LiteDB.LiteEngine.InsertDocument(CollectionPage col, BsonDocument doc, BsonType autoId)
   at LiteDB.LiteEngine.<>c__DisplayClass22_0.<Insert>b__0(CollectionPage col)
   at LiteDB.LiteEngine.Transaction[T](String collection, Boolean addIfNotExists, Func`2 action)
   at LiteDB.LiteEngine.InsertBulk(String collection, IEnumerable`1 docs, Int32 batchSize, BsonType autoId)
   at LiteDB.LiteCollection`1.InsertBulk(IEnumerable`1 docs, Int32 batchSize)
   at EmbyStat.Repositories.ShowRepository.<>c__DisplayClass5_0.<UpsertShow>b__0() in d:\a\1\s\EmbyStat.Repositories\ShowRepository.cs:line 91
   at EmbyStat.Repositories.BaseRepository.ExecuteQuery(Action query) in d:\a\1\s\EmbyStat.Repositories\BaseRepository.cs:line 29
   at EmbyStat.Jobs.Jobs.Sync.MediaSyncJob.PerformShowSyncAsync(IReadOnlyList`1 showsThatNeedAnUpdate, Library library, DateTime updateStartTime) in d:\a\1\s\EmbyStat.Jobs\Jobs\Sync\MediaSyncJob.cs:line 264
   at EmbyStat.Jobs.Jobs.Sync.MediaSyncJob.ProcessShowsAsync(IEnumerable`1 libraries, CancellationToken cancellationToken) in d:\a\1\s\EmbyStat.Jobs\Jobs\Sync\MediaSyncJob.cs:line 202
   at EmbyStat.Jobs.Jobs.Sync.MediaSyncJob.RunJobAsync() in d:\a\1\s\EmbyStat.Jobs\Jobs\Sync\MediaSyncJob.cs:line 85
   at EmbyStat.Jobs.BaseJob.Execute() in d:\a\1\s\EmbyStat.Jobs\BaseJob.cs:line 54
2019-12-05 20:34:23.2327 [ERROR] MEDIASYNC-JOB Job failed, check logs for more info. 
2019-12-05 20:34:23.4448 [WARN] Failed to process the job '016a7f2a-15f7-403b-a860-cd394def78f8': an exception occurred. Retry attempt 1 of 2 will be performed in 00:00:39. LiteDB.LiteException: Cannot insert duplicate key in unique index '_id'. The duplicate value is '453207'.
   at LiteDB.IndexService.AddNode(CollectionIndex index, BsonValue key, Byte level, IndexNode last)
   at LiteDB.IndexService.AddNode(CollectionIndex index, BsonValue key, IndexNode last)
   at LiteDB.LiteEngine.InsertDocument(CollectionPage col, BsonDocument doc, BsonType autoId)
   at LiteDB.LiteEngine.<>c__DisplayClass22_0.<Insert>b__0(CollectionPage col)
   at LiteDB.LiteEngine.Transaction[T](String collection, Boolean addIfNotExists, Func`2 action)
   at LiteDB.LiteEngine.InsertBulk(String collection, IEnumerable`1 docs, Int32 batchSize, BsonType autoId)
   at LiteDB.LiteCollection`1.InsertBulk(IEnumerable`1 docs, Int32 batchSize)
   at EmbyStat.Repositories.ShowRepository.<>c__DisplayClass5_0.<UpsertShow>b__0() in d:\a\1\s\EmbyStat.Repositories\ShowRepository.cs:line 91
   at EmbyStat.Repositories.BaseRepository.ExecuteQuery(Action query) in d:\a\1\s\EmbyStat.Repositories\BaseRepository.cs:line 29
   at EmbyStat.Jobs.Jobs.Sync.MediaSyncJob.PerformShowSyncAsync(IReadOnlyList`1 showsThatNeedAnUpdate, Library library, DateTime updateStartTime) in d:\a\1\s\EmbyStat.Jobs\Jobs\Sync\MediaSyncJob.cs:line 264
   at EmbyStat.Jobs.Jobs.Sync.MediaSyncJob.ProcessShowsAsync(IEnumerable`1 libraries, CancellationToken cancellationToken) in d:\a\1\s\EmbyStat.Jobs\Jobs\Sync\MediaSyncJob.cs:line 202
   at EmbyStat.Jobs.Jobs.Sync.MediaSyncJob.RunJobAsync() in d:\a\1\s\EmbyStat.Jobs\Jobs\Sync\MediaSyncJob.cs:line 85
   at EmbyStat.Jobs.BaseJob.Execute() in d:\a\1\s\EmbyStat.Jobs\BaseJob.cs:line 54
   at EmbyStat.Jobs.BaseJob.Execute() in d:\a\1\s\EmbyStat.Jobs\BaseJob.cs:line 66
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)

[darkassassin07 423]

Getting a similar error here repeated for several different ids. The media sync claims to have failed, but the majority of the statistics did show up.

 

I can see things like movie/epp counts, runtimes, ratings. Pretty much everything on the tv shows/movies pages filled in fine. The users page however only displays the usernames and last active times. Everything else on that page from user profile images to watched history is broken.

[Happy2Play 8282]

Anyone else seeing Memory cycling without any jobs going?  Restarting EmbyStat produces the same results.

 

 

Windows 10 v0.2.0-beta.14

Emby 4.4.0.1 beta

Edited December 6, 2019 by Happy2Play

[rdmiller13 8]

 Server 2016 v0.2.0-beta.14

 

Static here while it's idle.

[reggi 409]

@@Diedrich

 

I just created an issue for it Good idea to add some construction messages! https://github.com/mregni/EmbyStat/issues/1021

 

For the memory, I don't like that it's moving like that at all Maybe it's the LiteDB V4 that is doing something strange. Anyway, it still needs a lot of development and research to get the memory lower because even 355Mb for EmbyStat is a lot I guess.

Maybe it's HangFire that is using a lot of memory as well.

Anyway, LiteDB V5 is in beta, when it's released I'll upgrade EmbyStat and hope for the best some page exceptions and memory leaks are gone.

 

Watch history is broken indeed. That would be the next step to implement but have to fix the duplicate key exception first.

[Happy2Play 8282]

@@reggi

 

Note on a clean/new install (EmbyStat-win10-x64-v0.2.0-beta.14.zip), I am not currently seeing the memory issue.  Currently running at about 345MB after Media synchronisation.

 

So something may have happened in the update, but to be expected while testing.

Edited December 9, 2019 by Happy2Play

[theusedversion 35]

I'm so mad at myself.  I forgot to turn auto updates off when I finally got this to work.  Tried to get 13 to work several times but never could because it always failed in the media synchronization at a certain point.  So went back to 12 and got it to work.  Used it and loved it.  Then it updated to 14 and it's not working again and 12 doesn't seem to work anymore for some reason. I get a red warning sign at the end of the wizard that says "API Key is valid".  Guess I will just have to hope that one of the future versions work for me.

[darkassassin07 423]

It seems the latest version no longer lets you change the sort order of the 'overview' tab on the shows page.

 

You use to be able to click on 'percentage', or 'seasons', or 'title', etc... But now it's sorted by title with no way to change it :/

[lexisdude 133]

Ok , I installed this today (Win10 v0.2.0-beta.14); and it is currently in the process of "2019-12-13 12:03:10.2891 [iNFO] MEDIASYNC-JOB Calculating movie statistics " . I kicked this off about 4 hours ago, how long exactly can I expect the first/initial run to go for? I am watching the usage in task manager and its still utilizing memory , with .1 disk usage so I am guessing it's still working on things. When this was a plugin this process only took about 15 minutes tops.

After four hours I checked in the directory containing the data.db file - and it appears to be writing it at about 2KBps .. lol.. This is going to take forever with my library. Is this normal behavior?
 

Edited December 13, 2019 by lexisdude

[reggi 409]

@@darkassassin07, that's true. I had to disable it at the moment because the table was acting all weird. It's on my backlog to enable it again for sure.

 

@@lexisdude, 2 hours is way to long indeed. I have a little over 1100 movies and calculating statistics takes 15 seconds. Can you send me your log file (via PM) so I can have a look if something is failing?

[lexisdude 133]

reggi

The process never finished. I killed the server about 20 hrs after running it. On next boot up it did have some data in there, but as I said in my previous post - it was writing at about 2Kbps. The DB file is up to about 77Megs now, lol

And just to verify whether it was a cyclic disk issue/OS problem, I relocated the entire emby/mediabrowser setup to a different drive; kicked off the process again - and found the exact same problem. I did not remove my TV shows as part of the scan process; but that is what my server is heavily based upon. 

Edited December 15, 2019 by lexisdude

[Ametz 9]

Ok i have used your "Statistic" plugin until it stopped working now (Failed when sync).

And i used the "Largest movie" aspect of the mod a lot (so i can make changes to shrinking my library size accordingly)

 

But because the plugin no longer works (Failed when run) i installed this standalone instead (EmbyStat 020).

But as far as i can see, this function with "Largest movie" is missing?     

 

So now i have no way to fix movies like this in my library, to more normal 4 GB (running out of disc space so...):

Largest Movie

51,1 Gb

X-Movie name-X

[rdmiller13 8]

I love that feature too - as well as largest collection - been converting all my episodes to at least HEVC (two machines going pretty 24/7 for 3 months now) - recovered 13tb so far doing that...but as far as the largest movie - nice too, but I can scan my movies folder in Windows for gigantic etc an tell it <10gb, 8gb, 7gb, on down to 4gb...minus ones like Avengers or LOTR that are large for other reasons   Then convert the ones on that search list.

[reggi 409]

I have to re-add does features as well in the next few releases. I will first implement a huge table with all movies like you have on the show page. That way I can easily create the missing statistics as well.

[AciDCooL 7]

Heya,

 

Haven't touched EmbyStat for a while, but I have a problem with Media synchronisation right of the bat.

 

attached my logs.

2019-12-18 (1).log

Edited December 18, 2019 by AciDCooL

[justray2k 1]

I have to re-add does features as well in the next few releases

First things first, you do a great job!

I am(or was) a statistics plugin user within Emby the last two years. As it isn't running anymore i'm trying embystat instead and missing MY main feature.

 

Bringing back the statistics with quality/resolution of every movie/series would be cool I hope that's on your list;-)

Edited December 18, 2019 by justray2k

[nonyhaha 3]

Hello! 

 

Any new development in the error from litedb: 

2019-12-19 08:50:26.3252 [ERROR] Error while running job LiteDB.LiteException: Cannot insert duplicate key in unique index '_id'. The duplicate value is '52970'. ?

I do not have duplicates, and I do not have no season episodes  

 

Thank you! 

[Alex3111 7]

Can someone point me how to install this, as I was use to the plug-in, but it is no longer updating the users.

 

System:  Netgear ReadyNAS 314

OS:  Linux ReadyNAS 4.4.184.x86_64.1 #1 SMP Thu Sep 5 02:25:58 UTC 2019 x86_64 GNU/Linux (Debian 8.3.0-6)

EmbyServer:  Version 4.4.0.2 beta

 

I can access my ReadyNAS via PuTTY, and I have the v0.2.0-beta.14 file downloaded and unzipped into a folder within my NAS.

 

What's next?  How do I get this thing running?  If someone could shine some light on at least following an install, I would greatly appreciate it.

TIA

[reggi 409]

@@AciDCooL, looks like something has changed on the web socket API that's why you have so many errors. As for the sync not running, you have (as many users) the same "Cannot insert duplicate key in unique index '_id'" exception which I'm working on. (@@nonyhaha, this also is your problem as I read your post).

I have a fix, but is seems that the problem shifts then to a new problem, episodes linked to a show that really belong to a different show. So have to fix that first, I think it's something with the new TVDB API but I'm not sure.

 

@@justray2k, does statistics are indeed on the roadmap but feel free to update/create new feature requests on https://embystat.featureupvote.com/

 

@@Alex3111, I'm not a ReadyNAS user myself but it looks like ReadyNAS is based on Debian, so you have to download the Debain version and unzip it to a certain folder. Then start running the EmbyStat process. I presume you want this to run as a service or something so best thing is to google on how to setup a service in Debian (as far as I can tell). If the application is running open your browser and navigate to <IP-NAS>:6555 and the EmbyStat wizard will open. If you still have issues let me know.

[Sammy 737]

I just started using EmbyStat. Thank you for the work!!! I was a previous Statistics Plugin User and then I had a system crash about the same time I was going to make the switch a month ago now. So I was delayed.

 

The Emby Stat Executable is in a portable location in the download folder where it downloaded. Should I move it and the folder contents to another location? How to force it to load on re-boot of the PC? Can it be put into the system tray much like embytray is?

 

Thanks!