Developing a standalone embystat server

[Shidapu 14]

Love the work put into this. Thanks to this plugin i found tons of episodes i was missing.
But i have one issue right now, and thats when its idling, its using 5-20% cpu. Not sure why. So i have disabled it in services until i use it.

[reggi 409]

Hey guys,

 

After another long time of silence I have a new beta version for you all beta.17 (I skipped 16 by accident).

 

I updated to LiteDB v5 and I hope that will fix the page fault exceptions.

Also I updated to Angular 9 which should give a smaller package size and faster rendering, still have to tweak the config for that.

I also added an installer on the release page for windows, the installer can install the software as a program or as a service. If you find any problems with the installer let me know!

 

I tried to create a .deb file for ubuntu but it kept failing somehow. Need some more testing for that to work sadly.

 

If you install this version make sure you DON'T update but install a fresh instance. The new LiteDB version needs a new database file to work.

 

 

Happy testing!! And let's hope I get some more time to work on the project and get some kind of stable version.

[jaycedk 371]

Hi reggi.

Just installed the windows version not as a service.

Getting this error running sync job.

it fails after 1900 out off 4200 movies.

 

2020-04-07 19:42:43.2241 [ERROR] MEDIASYNC-JOB    Job failed, check logs for more info.
2020-04-07 19:42:43.4015 [ERROR] Failed to process the job 'ba0869e7-2145-46ce-abd3-fa7d6ff78553': an exception occurred. System.NullReferenceException: Object reference not set to an instance of an object.
   at EmbyStat.Clients.Base.Http.BaseHttpClient.GetMovies(String parentId, Int32 startIndex, Int32 limit)
   at EmbyStat.Jobs.Jobs.Sync.MediaSyncJob.PerformMovieSyncAsync(String parentId, Int32 startIndex, Int32 limit)
   at EmbyStat.Jobs.Jobs.Sync.MediaSyncJob.PerformMovieSyncAsync(String parentId, Int32 startIndex, Int32 limit)
   at EmbyStat.Jobs.Jobs.Sync.MediaSyncJob.PerformMovieSyncAsync(String parentId, Int32 startIndex, Int32 limit)
   at EmbyStat.Jobs.Jobs.Sync.MediaSyncJob.PerformMovieSyncAsync(String parentId, Int32 startIndex, Int32 limit)
   at EmbyStat.Jobs.Jobs.Sync.MediaSyncJob.ProcessMoviesAsync(IReadOnlyList`1 libraries, CancellationToken cancellationToken)
   at EmbyStat.Jobs.Jobs.Sync.MediaSyncJob.RunJobAsync()
   at EmbyStat.Jobs.BaseJob.Execute()
   at EmbyStat.Jobs.BaseJob.Execute()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)

Edited April 7, 2020 by jaycedk

[jaycedk 371]

hmm ok disabling short movie calculations and all is fine.

That leads me to think, that 1 of my movies has a 0 length, or is damaged.

[WilhelmStroker 96]

- some of the setup for the service only references Jellyfin, not Emby

- I'm seeing a total missing episode number of 13243 which i don't think is correct

- Getting an unhandled backend exception, and this error in the logs

 

2020-04-07 19:51:42.7558 [ERROR] An unhandled error occurred. LiteDB.LiteException: Document size exceed 2037500 limit

   at LiteDB.Engine.DataService.Insert(BsonDocument doc)

   at LiteDB.Engine.LiteEngine.InsertDocument(Snapshot snapshot, BsonDocument doc, BsonAutoId autoId, IndexService indexer, DataService data)

   at LiteDB.Engine.LiteEngine.<>c__DisplayClass7_0.<Insert>b__0(TransactionService transaction)

   at LiteDB.Engine.LiteEngine.AutoTransaction[T](Func`2 fn)

   at LiteDB.SharedEngine.Insert(String collection, IEnumerable`1 docs, BsonAutoId autoId)

   at LiteDB.LiteCollection`1.Insert(T entity)

   at EmbyStat.Repositories.StatisticsRepository.<>c__DisplayClass2_0.<AddStatistic>b__0()

   at EmbyStat.Repositories.BaseRepository.ExecuteQuery(Action query)

   at EmbyStat.Repositories.StatisticsRepository.AddStatistic(String json, DateTime calculationDateTime, StatisticType type, IReadOnlyList`1 collectionIds)

   at EmbyStat.Services.ShowService.CalculateCollectedRows(List`1 libraryIds)

   at EmbyStat.Services.ShowService.GetCollectedRows(List`1 libraryIds, Int32 page)

   at EmbyStat.Controllers.Show.ShowController.GetCollectedRows(List`1 libraryIds, Int32 page)

   at lambda_method(Closure , Object , Object[] )

   at Microsoft.Extensions.Internal.ObjectMethodExecutor.Execute(Object target, Object[] parameters)

   at Microsoft.AspNetCore.Mvc.Infrastructure.ActionMethodExecutor.SyncActionResultExecutor.Execute(IActionResultTypeMapper mapper, ObjectMethodExecutor executor, Object controller, Object[] arguments)

   at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.InvokeActionMethodAsync()

   at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.Next(State& next, Scope& scope, Object& state, Boolean& isCompleted)

   at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.InvokeNextActionFilterAsync()

--- End of stack trace from previous location where exception was thrown ---

   at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.Rethrow(ActionExecutedContextSealed context)

   at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.Next(State& next, Scope& scope, Object& state, Boolean& isCompleted)

   at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.InvokeInnerFilterAsync()

--- End of stack trace from previous location where exception was thrown ---

   at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.<InvokeNextExceptionFilterAsync>g__Awaited|25_0(ResourceInvoker invoker, Task lastTask, State next, Scope scope, Object state, Boolean isCompleted)

[reggi 409]

First of all, thanks for still wanting to test the application so fast Really appreciate it!!

 

 

@@jaycedk, that is some strange behavior really because from your log it seems that you have some items that go 4 levels deep (so Boxset in Boxset in Boxset in ...)

Why the GetMovies() is failing is unknown but probably because it needs data that is not returned by Emby. Strange part is that it worked after a second run But I don't think that disabling the "to short" option fixed something because that calculation is only done after the movie sync.

 

@@WilhelmStroker, I will check the setup again for strange stuff. It works with Emby as well! But the code I used for the installer comes from the Jellyfin repo so there are some leftovers probably!

As for the missing episodes and exception, it looks like the tvdb sync did not work at all. Did you get some warning or errors during the media sync job (shows part)?

[WilhelmStroker 96]

Yup it definitely works well with Emby but there's definitely Jellyfin leftovers. Only a minor issue anyway

 

I just ran another media sync and it looks like it falls over on Bates Motel. I only have season 3, 4 and 5 on Emby.

 

2020-04-08 07:01:03.5786 [INFO] MEDIASYNC-JOB Found 20 missing episodes for show Bates Motel 
2020-04-08 07:01:03.7420 [INFO] MEDIASYNC-JOB Processed (9/98) Bates Motel 
2020-04-08 07:01:03.8254 [ERROR] Error while running job LiteDB.LiteException: Cannot insert duplicate key in unique index '_id'. The duplicate value is '"11000"'.
   at LiteDB.Engine.IndexService.AddNode(CollectionIndex index, BsonValue key, PageAddress dataBlock, Byte level, IndexNode last)
   at LiteDB.Engine.IndexService.AddNode(CollectionIndex index, BsonValue key, PageAddress dataBlock, IndexNode last)
   at LiteDB.Engine.LiteEngine.InsertDocument(Snapshot snapshot, BsonDocument doc, BsonAutoId autoId, IndexService indexer, DataService data)
   at LiteDB.Engine.LiteEngine.<>c__DisplayClass7_0.<Insert>b__0(TransactionService transaction)
   at LiteDB.Engine.LiteEngine.AutoTransaction[T](Func`2 fn)
   at LiteDB.SharedEngine.Insert(String collection, IEnumerable`1 docs, BsonAutoId autoId)
   at EmbyStat.Repositories.ShowRepository.<>c__DisplayClass6_0.<InsertShow>b__0()
   at EmbyStat.Repositories.BaseRepository.ExecuteQuery(Action query)
   at EmbyStat.Jobs.Jobs.Sync.MediaSyncJob.PerformShowSyncAsync(IReadOnlyList`1 showsThatNeedAnUpdate, Library library, DateTime updateStartTime)
   at EmbyStat.Jobs.Jobs.Sync.MediaSyncJob.ProcessShowsAsync(IEnumerable`1 libraries, CancellationToken cancellationToken)
   at EmbyStat.Jobs.Jobs.Sync.MediaSyncJob.RunJobAsync()
   at EmbyStat.Jobs.BaseJob.Execute()

[jaycedk 371]

 

@jaycedk, that is some strange behavior really because from your log it seems that you have some items that go 4 levels deep (so Boxset in Boxset in Boxset in ...)

Why the GetMovies() is failing is unknown but probably because it needs data that is not returned by Emby. Strange part is that it worked after a second run   But I don't think that disabling the "to short" option fixed something because that calculation is only done after the movie sync.

Hi Reggi.

Is there any thing I can do to identify where the problem is.

Tbh I have no clue as to where to start looking.

[kesm 25]

Hi, 

Just to be sure that I am not missing something, currently there is no way to view what episodes are missing?
 

 

I have this view and the number of missing episodes but I can't have a list with the list of series and episodes that are missing?
 

[thegrunge 29]

Emby stats Does not work well for me on windows. It works but does not update the content read by the members

[reggi 409]

@@jaycedk, I'm going to prepare a new build for you with some extra logging. i'll send a PM tomorrow when it's ready

 

@@thegrunge thanks for testing embystat. At this moment the user stat pages are broken (added a construction gif on the page)

Only the static pages should work a bit.

 

@@kesm, yes there is but it's still a bit hidden in the UI. In the screenshot below you can see where you need to go to see the missing episodes. 

[thegrunge 29]

@reggi Ok thanks!

[kesm 25]

 

@@kesm, yes there is but it's still a bit hidden in the UI. In the screenshot below you can see where you need to go to see the missing episodes. 

Sorry I totally missed the overview tab.

 

- Could be great to have the possibility to sort by percentage to easily find missing episodes.

- I have a lot of missing episodes that aren't really ones as episodes aren't aired yet. Like Dirty John Season 2. Can you ignore episodes without airing date to determine the number of missing episodes?

[darkassassin07 417]

Installed the new version (.18) using the win 10 .exe installer without the install as service option by stopping the nssm service, renaming the install folder to /embystat-old, installed the new version pointing both install and data paths to the same /embystat folder then restarted the nssm service.

 

After going through the setup wizard supplying api key and all that, I clicked on 'start synchronization' and went to the jobs page. There nothing appeared to be happening (no progress bar on the media sync job, nothing in the log pane) I manually started the sync and away it went.

 

Processed all 1503 movies and 213 shows successfully "[iNFO] MEDIASYNC-JOB Job finished after 19 minutes."

 

Then the sync immediately restarted. (I'm guessing because of my manual start+auto start from the setup) I let that run, but this time it failed with the error:

2020-04-08 17:05:04.6263 [ERROR] Error while running job LiteDB.LiteException: Cannot insert duplicate key in unique index '_id'. The duplicate value is '"123394"'.
   at LiteDB.Engine.IndexService.AddNode(CollectionIndex index, BsonValue key, PageAddress dataBlock, Byte level, IndexNode last)
   at LiteDB.Engine.IndexService.AddNode(CollectionIndex index, BsonValue key, PageAddress dataBlock, IndexNode last)
   at LiteDB.Engine.LiteEngine.InsertDocument(Snapshot snapshot, BsonDocument doc, BsonAutoId autoId, IndexService indexer, DataService data)
   at LiteDB.Engine.LiteEngine.<>c__DisplayClass7_0.<Insert>b__0(TransactionService transaction)
   at LiteDB.Engine.LiteEngine.AutoTransaction[T](Func`2 fn)
   at LiteDB.SharedEngine.Insert(String collection, IEnumerable`1 docs, BsonAutoId autoId)
   at EmbyStat.Repositories.ShowRepository.<>c__DisplayClass6_0.<InsertShow>b__0()
   at EmbyStat.Repositories.BaseRepository.ExecuteQuery(Action query)
   at EmbyStat.Jobs.Jobs.Sync.MediaSyncJob.PerformShowSyncAsync(IReadOnlyList`1 showsThatNeedAnUpdate, Library library, DateTime updateStartTime)
   at EmbyStat.Jobs.Jobs.Sync.MediaSyncJob.ProcessShowsAsync(IEnumerable`1 libraries, CancellationToken cancellationToken)
   at EmbyStat.Jobs.Jobs.Sync.MediaSyncJob.RunJobAsync()
   at EmbyStat.Jobs.BaseJob.Execute()

 

Then it re-ran twice automatically: "[WARN] Failed to process the job '93e5e598-7c91-45de-8943-2e4768a17a48': an exception occurred. Retry attempt 1 of 2 will be performed in 00:00:27."

both times failing with the same error including the same duplicate key value until it gave up.

anon-2020-04-08.log

[Happy2Play 8215]

Getting a different error.

2020-04-08 19:11:03.2754 [INFO] MEDIASYNC-JOB	Processed 400 / 606 movies 
2020-04-08 19:11:03.6882 [ERROR] MEDIASYNC-JOB	Movie error: Object reference not set to an instance of an object. 
2020-04-08 19:11:03.6882 [ERROR] MEDIASYNC-JOB	Movie error: Object reference not set to an instance of an object. 
2020-04-08 19:11:03.6882 [ERROR] Error while running job System.NullReferenceException: Object reference not set to an instance of an object.
   at EmbyStat.Clients.Base.Http.BaseHttpClient.GetMovies(String parentId, Int32 startIndex, Int32 limit)
   at EmbyStat.Jobs.Jobs.Sync.MediaSyncJob.PerformMovieSyncAsync(String parentId, Int32 startIndex, Int32 limit)
   at EmbyStat.Jobs.Jobs.Sync.MediaSyncJob.PerformMovieSyncAsync(String parentId, Int32 startIndex, Int32 limit)
   at EmbyStat.Jobs.Jobs.Sync.MediaSyncJob.PerformMovieSyncAsync(String parentId, Int32 startIndex, Int32 limit)
   at EmbyStat.Jobs.Jobs.Sync.MediaSyncJob.PerformMovieSyncAsync(String parentId, Int32 startIndex, Int32 limit)
   at EmbyStat.Jobs.Jobs.Sync.MediaSyncJob.ProcessMoviesAsync(IReadOnlyList`1 libraries, CancellationToken cancellationToken)
   at EmbyStat.Jobs.Jobs.Sync.MediaSyncJob.RunJobAsync()
   at EmbyStat.Jobs.BaseJob.Execute()
2020-04-08 19:11:03.7146 [ERROR] MEDIASYNC-JOB	Job failed, check logs for more info. 
2020-04-08 19:11:03.8453 [WARN] Failed to process the job '22729a84-350d-4803-b955-cda3427c3bbb': an exception occurred. Retry attempt 1 of 2 will be performed in 00:00:42. System.NullReferenceException: Object reference not set to an instance of an object.
   at EmbyStat.Clients.Base.Http.BaseHttpClient.GetMovies(String parentId, Int32 startIndex, Int32 limit)
   at EmbyStat.Jobs.Jobs.Sync.MediaSyncJob.PerformMovieSyncAsync(String parentId, Int32 startIndex, Int32 limit)
   at EmbyStat.Jobs.Jobs.Sync.MediaSyncJob.PerformMovieSyncAsync(String parentId, Int32 startIndex, Int32 limit)
   at EmbyStat.Jobs.Jobs.Sync.MediaSyncJob.PerformMovieSyncAsync(String parentId, Int32 startIndex, Int32 limit)
   at EmbyStat.Jobs.Jobs.Sync.MediaSyncJob.PerformMovieSyncAsync(String parentId, Int32 startIndex, Int32 limit)
   at EmbyStat.Jobs.Jobs.Sync.MediaSyncJob.ProcessMoviesAsync(IReadOnlyList`1 libraries, CancellationToken cancellationToken)
   at EmbyStat.Jobs.Jobs.Sync.MediaSyncJob.RunJobAsync()
   at EmbyStat.Jobs.BaseJob.Execute()
   at EmbyStat.Jobs.BaseJob.Execute()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)

But note I am on Beta Server 4.5.0.5 if it makes a difference.

[Taurec 60]

Hi,

what do I have to set that I will be forwarded to the film if I click on a double ID.

A new TAB (Firefox) opens but I only get to the login page of Emby.

 

cu Taurec

[AMMJ93 0]

2020-04-10 11:39:58.8970 [ERROR] MEDIASYNC-JOB	Movie error: Object reference not set to an instance of an object. 
2020-04-10 11:39:58.8970 [ERROR] MEDIASYNC-JOB	Movie error: Object reference not set to an instance of an object. 
2020-04-10 11:39:58.8970 [ERROR] Error while running job System.NullReferenceException: Object reference not set to an instance of an object.
   at EmbyStat.Clients.Base.Converters.MovieConverter.<>c.<ConvertToMovie>b__0_14(BaseItemPerson y)
   at System.Linq.Enumerable.WhereSelectArrayIterator`2.ToArray()
   at EmbyStat.Clients.Base.Converters.MovieConverter.ConvertToMovie(BaseItemDto x, String collectionId)
   at System.Linq.Enumerable.SelectArrayIterator`2.ToList()
   at EmbyStat.Jobs.Jobs.Sync.MediaSyncJob.PerformMovieSyncAsync(String parentId, Int32 startIndex, Int32 limit)
   at EmbyStat.Jobs.Jobs.Sync.MediaSyncJob.PerformMovieSyncAsync(String parentId, Int32 startIndex, Int32 limit)
   at EmbyStat.Jobs.Jobs.Sync.MediaSyncJob.PerformMovieSyncAsync(String parentId, Int32 startIndex, Int32 limit)
   at EmbyStat.Jobs.Jobs.Sync.MediaSyncJob.PerformMovieSyncAsync(String parentId, Int32 startIndex, Int32 limit)
   at EmbyStat.Jobs.Jobs.Sync.MediaSyncJob.ProcessMoviesAsync(IReadOnlyList`1 libraries, CancellationToken cancellationToken)
   at EmbyStat.Jobs.Jobs.Sync.MediaSyncJob.RunJobAsync()
   at EmbyStat.Jobs.BaseJob.Execute()

I get this error when syncing movies, EmbyStat version: 0.2.0-beta.18

[darkassassin07 417]

Hi,

what do I have to set that I will be forwarded to the film if I click on a double ID.

A new TAB (Firefox) opens but I only get to the login page of Emby.

 

cu Taurec

Do you have firefox set to 'remember me' for your emby user?

If not, the movie page tries to open but the browser session isnt authenticated so emby kicks you to the login page.

 

 

@@Luke would it be possible to have emby return to the page you tried to access once login had been completed instead of just heading to the now-authed users home page?

Edited April 10, 2020 by darkassassin07

[reggi 409]

I'm working on all the null reference exception in the movie and show converters. So hopefully that will be fixed in the next release

As for the duplicate key exception, it looks like not everything is getting removed from the database on a new sync. i'll check why that's happening as well. I created a github issue for it https://github.com/mregni/EmbyStat/issues/1156

 

Thanks everyone for testing It's very nice to see a lot of people willing to put time in testing the application.

[vaise 304]

I have recently got emby stat working again due to the database issue on my docker container.

I have a question on the size calculation.

My TV shows library says on the embystat that is takes up 4.94TB, but my TV Shows folder says it is using 9.5TB,

I cant imagine there should be a 50% difference ?

edit - correction, I have 4k included, now it is worse off, as TV shows on its own states 4.22TB userd, but the TC shows folder of content as I say is 9.5TB - so very close to 50% difference.

There is no way emby does not include all my folders of TV shows.

further edit - movies is also off - file size of that folder is 3.5TB, embystat says 2.45TB

 

***** ignore for now - I think the media sync stopped (its first), I have kicked it off again and will check results later.

 

Confirmed - missing about 1000 movies compared to embys 'reports'.

Logs have the duplicate key stuff but not sure if that relates to all the missing content.

Edited April 12, 2020 by vaise

[pju 10]

Just backing post #739 - seeing exactly the same.

- Running on Windows 10, latest patches.

- Starting with a Emby Server 4.3.1.0 with no libraries,

- Defined libraries and running a Library Scan.

- Installed EmbyStat 0.2.0-beta.18

- First "Media synchronisation" ran succesfully

- Next "Media synchronisation" failed with "LiteDB.LiteException: Cannot insert duplicate key in unique index '_id'...duplicate value is '"104"'."

 

Have repeated all the above twice - same thing happening and same id "104" being reported.

anon.2020-04-11 02.log

[embyzone 3]

I cant wait to have this working!

Sofar I havent managed to do a full scan without it crashing, but I think this also depends on how large your media collection is.

I will keep updating the docker and try it out! - Keep up the good work!

Cheers!

[Diedrich 355]

I cant wait to have this working!

Sofar I havent managed to do a full scan without it crashing, but I think this also depends on how large your media collection is.

I will keep updating the docker and try it out! - Keep up the good work!

Cheers!

The legacy statistics plugin has been revived by a community member for server compatibility. Here is the link:

https://emby.media/community/index.php?/topic/65998-New-Statistics-plugin&do=findComment&comment=858143

[reggi 409]

@@vaise and everyone else with the duplicate key exception.

I'm currently building a new beta version (beta.19) so will be online in a few hours (depends on my builds, did some changes to them so maybe needs some more tweaking)

 

I added some more validation and fallback stuff so the duplicate exception but also null reference exceptions, ... are handled properly. I also added some more debug logging but more on that later when I have the build ready

[adrianwi 237]

I'm struggling to get v18 to complete a media scan - is this related to the duplicate key exception?

 

I've attached a log, which is after it reported as failed and I started to try again.

 

Thanks

2020-04-15.log

Edited April 15, 2020 by adrianwi