Developing a standalone embystat server

[bluezog 7]

I've notived everytime I run the media synchronization task I get 'MEDIASYNC-JOB => Can't seem to find Prison Break on TVDB, skipping show for now' in the logs.

 

The media is sorted:

/Tv Shows/Prison Break/Season X/Prison Break SXXEXX <epp name>.extention

 

The tvdb link is:

https://www.thetvdb.com/series/prison-break

 

Emby found it and grabbed the metadata just fine, but embystat wont for whatever reason.

 

 

 

 

@@reggi, I'm also still seeing the incorrect total playback time for tv epps. Just curious if you've tracked down whats up with that.

 

I had this issue too, look at the Prison Break metadata at the TheTVDB Id: and make sure it is - 360115, mine was something different even though Emby had identified it correctly.

Hope that helps.

[darkassassin07 428]

@@bluezog

That seems to have done the job. Thanks mate

[reggi 409]

Looks like for some reason I get wrong TVDB id's back from Emby or in my database. Have to look into that!

As for the wrong episode time, I reopened the issue https://github.com/mregni/EmbyStat/issues/812 (like you requested some time ago I just saw)

 

I haven't had a lot of time lately to work on the project. Hopefully I will increase speed again soon!

 

The memory usage seems to be harder to fix as well. I did some refactoring that helped a bit on my system. But I still end up with > 500Mb unmanaged memory for some reason.

[darkassassin07 428]

Looks like for some reason I get wrong TVDB id's back from Emby or in my database. Have to look into that!

I think this is a problem on embys side, but a really minor one. Emby had the wrong tvdb id stored for prison break 75340 but the actual id is 360115.

 

If you google: tvdb "75340"

You get lots of links to the tvdb all about prison break, but they all lead to a "This record has either been deleted or has never existed" page.

However emby retrieved that id, its collecting from an outdated source.

[Happy2Play 8296]

So that would mean the series id changed for some reason at TVDB.  And I would assume the server log would have reflected the same issue for that id every time it was queried on their api.  I don't remember the series but when they initially pushed v3 I had multiple series with invalid tvdb ids.  So all the existing metadata was correct except for the obsolete id.

 

Changes like this will alway require manual intervention as all existing ids continue to get used.

Edited September 18, 2019 by Happy2Play

[Pog22 50]

Yeah just checked mine was the same

[reggi 409]

Hey guys,

 

After a long time I have a new beta ready

 

Changes:

- Displaying proper total runtime for shows #812

- Update logo's and primary color #886

- Added missing translation keys #903

- Improved Docker support, config folder should be mountable from now on #906

- Added reverse proxy headers for Nginx setup #915

- Create multi-architecture Docker images #925

 

I created 2 multi architecture images, so from now on you can use new Docker tags to run your container:

uping/embystat:nightly

uping/embystat:beta

Docker will pick the correct container for your OS.

 

Can someone running EmbyStat with Nginx test if the reverse proxy configuration?

 

It also possible that the memory usage is dropped a bit, but I'm still looking into the unmanaged memory problem.

Edited September 20, 2019 by reggi

[darkassassin07 428]

I have my embystat behind nginx:

I installed the new version by stopping the nssm service, deleting everything in the embystat install folder and unzipping the new version into that folder. I then started the service again, and connected via nginx.

That brought me to the setup wizard which I filled in all the way up to the last choice: 'finish' or 'start synchronization'.

I pressed start which brought me to the jobs page automatically queuing up the media sync and small sync. Both failed with the error 'job not running because wizard is not finished'. Refreshing or returning to any other page just brings me back to the start of the setup wizard.

 

I then connected directly to the embystat server (localhost), finished the wizard and pressed start sync. This time it completed just fine and browsing+starting jobs through nginx now works as well.

 

 

Plus to boot, the tv epps runtime seems to be fixed. Im getting an average of 32ish min per episode now.

 

 

I can give you a log if you like, but there isn't really anything interesting in it.

[reggi 409]

@@darkassassin07 that is some super weird behavior

 

Can you maybe point me to some tutorial how to setup nginx or share your setup config? Then I can try setting it up myself and see what happens.

As for as I can tell now is that if you work with Nginx the system has no rights to write to disk (wizard state is saved in the usersettings.json file on disk)

[darkassassin07 428]

Here is a trimmed down version of my config (I removed all the non-relevant stuff for other services and the client auth via ssl and/or auth_basic)

My nginx proxy is running on a rpi 3b that's on the same LAN as the embystat server which is running on win 10.

 

You will have to update the ssl_certificate, ssl_certificate_key, server_name and proxy_pass directives to match your setup. (you could also switch to http and remove the ssl stuff too)

user www-data;
worker_processes 4;
pid /run/nginx.pid;
include /etc/nginx/modules-enabled/*.conf;
 
events {
    worker_connections 1024;
}
 
http {
    sendfile on;
    tcp_nopush on;
    tcp_nodelay on;
    server_tokens off;
    
    map_hash_bucket_size 64;
    types_hash_max_size 2048;
    client_header_buffer_size 1k;
    client_body_buffer_size 8k;
    client_max_body_size 5m;
    
    include /etc/nginx/mime.types;
    default_type application/octet-stream;
    server_names_hash_bucket_size 128;
 
    ssl on;
    ssl_buffer_size 4k;
    ssl_protocols TLSv1.2;
    ssl_ciphers "ECDHE-RSA-AES256-GCM-SHA384:ECDHE-RSA-AES128-GCM-SHA256:DHE-RSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-SHA:DHE-RSA-AES128-SHA";
    ssl_prefer_server_ciphers on;
    ssl_session_cache shared:SSL:10m;
 
    ssl_certificate /home/pi/SSL/cert.pem;
    ssl_certificate_key /home/pi/SSL/key.key;
 
    add_header X-Xss-Protection "1; mode=block" always;
    add_header X-Content-Type-Options "nosniff" always;
    add_header 'Referrer-Policy' 'no-referrer' always;
    add_header Strict-Transport-Security "max-age=31536000" always;
    add_header X-Frame-Options "SAMEORIGIN" always;
    add_header 'Feature-Policy' "" always;
    
    gzip on;
    gzip_disable "msie6";
    gzip_comp_level 6;
    gzip_min_length 1100;
    gzip_buffers 16 8k;
    gzip_proxied any;
    gzip_types text/plain text/css text/js text/xml text/javascript application/javascript application/x-javascript application/json application/xml application/rss+xml image/svg+xml;
 
    proxy_http_version 1.1;
    proxy_hide_header X-Powered-By;
    proxy_set_header HOST $host;
    proxy_set_header X-Real-IP $remote_addr;
    proxy_set_header Range $http_range;
    proxy_set_header If-Range $http_if_range;
    proxy_set_header X-Forwarded-Proto $scheme;
    proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
    proxy_set_header X-Forwarded-Host $host:443;
 
    proxy_max_temp_file_size 0;
    proxy_buffering off;
 
    client_header_timeout 10s;
    client_body_timeout   30s;
    keepalive_timeout     60s;
    send_timeout          30s;
    proxy_read_timeout    30s;
    proxy_send_timeout    30s;
    keepalive_requests    10;
 
 
    #------------------------------------------------------------------------------------------------
    server {
        server_name embystat.mydomain.com;
        listen 443 ssl http2;

 
        location / {
            proxy_pass http://192.168.0.106:6555;
        }
    }
    
} 

/edit

 

 

I'm not sure how I missed this, but I am also getting this error in nginxs error log:

2019/09/19 14:07:51 [error] 3645#3645: *168458 access forbidden by rule, client: 192.168.0.106, server: embystat.mydomain.com, request: "PUT /api/settings HTTP/2.0", host: "embystat.mydomain.com", referrer: "https://embystat.mydomain.com/wizard"

This is because I have:

limit_except GET HEAD POST {
    deny all;
}

In my full config.

Commenting this out, or adding PUT to the list fixes my proxy errors.

 

I feel like an idiot.... I've been staring at this all day....

 

 

 

TLDR:

Embystat works fine behind nginx in my testing. The problems I was having are my own making :/

Edited September 21, 2019 by darkassassin07

[Shenniko 5]

Hi,

 

Apologies if this has been asked before.. Are there plans for a Linux version of this (not using docker)?

 

Thanks

 

Shenn

[WilhelmStroker 96]

There already is:

 

EmbyStat-debian-x64-v0.2.0-beta.11.zip

90.3 MB

EmbyStat-fedora-x64-v0.2.0-beta.11.zip66.3 MB

EmbyStat-linux-arm-x64-v0.2.0-beta.11.zip63.8 MB

EmbyStat-ubuntu-x64-v0.2.0-beta.11.zip66.3 MB

EmbyStat-win10-x64-v0.2.0-beta.11.zip68.2 MB

 

[reggi 409]

@@Shenniko, you can download the unix versions from the GitHub Release page like @@WilhelmStroker mentioned. I had a working systemctl script but seem too have lost it.

If you create a working one you can always send it to me so I can add it to the project as a template.

 

@@darkassassin07, thanks for the edit! Nice to hear that EmbyStat is working behind Nginx

Edited September 23, 2019 by reggi

[bluezog 7]

Hi all,

 

I am looking for some help to figure out why Embystat will not complete a Media synchronisation job, it stops with an error after about 40 sec and tells me to look at the logs. I am pretty sure its not Embystat and something in my Emby data somewhere but I have no idea what it could be. 

 

I have attached the log from today.

 

I am using the latest version of Embystat (EmbyStat-win10-x64-v0.2.0-beta.11) from a clean start.

 

Any help would be appreciated,

 

Thanks 

2019-09-23.log

[reggi 409]

Hey @@bluezog, thanks for testing the application! 

I had a look at your log and it seems you have the same problem as @diedrichg here: https://github.com/mregni/EmbyStat/issues/879

 

At the moment there is not much that you can do, for some reason the authentication fails after a successful login on the tvdb server.

I'll have a look when I finished the missing episode detail screen.

[bluezog 7]

Hi @@reggi, thanks for the reply. I think the app is looking good and will keep testing.

Here is some more info for you.

I have installed both Emby and Embystat on a virtual machine, Win 10, same as my main system and added some test videos (a couple of TVShows and films). Once I had that all set up Embystat ran fine with no errors. I'm sure it must be something in my main Emby setup that's causing it(been using Emby for quite some time), just got no idea what though 

[reggi 409]

@@bluezog, you probably have luck with your test data The IndexOutOfRangeException only happens when you have a lot of data it seems. 

Anyway, I release beta.12 this night that should fixe that problem as well as other issues with tvdb authentication.

 

I also added a new feature, the show overview table is now expendable (click on a row) so you can see all missing episodes. The layout is still a work in progress, any ideas are more then welcome! 

Also ideas about other data that should be displayed there.

 

You also will see a new statistic box on the show and movie page displaying the total size of the selected collection.

 

- Fixed filter for shows overview table #871

- Various tvdb api issues #879

- Missing episode count should be correct now #911

 

Have fun testing!

[WilhelmStroker 96]

Installed the latest build, the media sync fails with the following error (possibly it is finding a duplicate?)

 

2019-09-26 12:31:28.5162 [ERROR] Error while running job System.InvalidOperationException: Sequence contains more than one matching element
   at System.Linq.Enumerable.SingleOrDefault[TSource](IEnumerable`1 source, Func`2 predicate)
   at EmbyStat.Common.Extensions.ShowExtension.GetNonSpecialEpisodeCount(Show show, Boolean includeMissing) in d:\a\1\s\EmbyStat.Common\Extensions\ShowExtension.cs:line 16
   at System.Linq.Enumerable.Sum[TSource](IEnumerable`1 source, Func`2 selector)
   at EmbyStat.Services.ShowService.TotalEpisodeCount(IReadOnlyList`1 shows) in d:\a\1\s\EmbyStat.Services\ShowService.cs:line 401
   at EmbyStat.Services.ShowService.CalculateGeneralStatistics(IReadOnlyList`1 shows) in d:\a\1\s\EmbyStat.Services\ShowService.cs:line 77
   at EmbyStat.Services.ShowService.CalculateShowStatistics(List`1 libraryIds) in d:\a\1\s\EmbyStat.Services\ShowService.cs:line 62
   at EmbyStat.Jobs.Jobs.Sync.MediaSyncJob.CalculateStatistics() in d:\a\1\s\EmbyStat.Jobs\Jobs\Sync\MediaSyncJob.cs:line 394
   at EmbyStat.Jobs.Jobs.Sync.MediaSyncJob.RunJobAsync() in d:\a\1\s\EmbyStat.Jobs\Jobs\Sync\MediaSyncJob.cs:line 93
   at EmbyStat.Jobs.BaseJob.Execute() in d:\a\1\s\EmbyStat.Jobs\BaseJob.cs:line 54

[reggi 409]

Mmm, looks like you have multiple special seasons for a certain show. I'll create a GitHub issue and fix for it today.

[WilhelmStroker 96]

Mmm, looks like you have multiple special seasons for a certain show. I'll create a GitHub issue and fix for it today.

any way to find out what show from the logs? Let me know once you have a build and I'll test again

[Happy2Play 8296]

This happened after in-app update from .11 to .12.
 
Getting the same/similar error on Looney Tunes.
 
Also why would it only check 5 shows for missing episodes?

2019-09-26 03:24:43.0741 [INFO] MEDIASYNC-JOB We need to check 5 shows for missing episodes 
2019-09-26 03:24:43.0835 [INFO] THETVDB-CLIENT Call to THETVDB: [url=https://api.thetvdb.com/series/72514/episodes?page=1 ]https://api.thetvdb.com/series/72514/episodes?page=1 [/url]
2019-09-26 03:24:43.6884 [INFO] THETVDB-CLIENT Call to THETVDB: [url=https://api.thetvdb.com/series/72514/episodes?page=2 ]https://api.thetvdb.com/series/72514/episodes?page=2 [/url]
2019-09-26 03:24:43.9098 [INFO] THETVDB-CLIENT Call to THETVDB: [url=https://api.thetvdb.com/series/72514/episodes?page=3 ]https://api.thetvdb.com/series/72514/episodes?page=3 [/url]
2019-09-26 03:24:44.1475 [INFO] THETVDB-CLIENT Call to THETVDB: [url=https://api.thetvdb.com/series/72514/episodes?page=4 ]https://api.thetvdb.com/series/72514/episodes?page=4 [/url]
2019-09-26 03:24:44.3824 [INFO] THETVDB-CLIENT Call to THETVDB: [url=https://api.thetvdb.com/series/72514/episodes?page=5 ]https://api.thetvdb.com/series/72514/episodes?page=5 [/url]
2019-09-26 03:24:44.6475 [INFO] THETVDB-CLIENT Call to THETVDB: [url=https://api.thetvdb.com/series/72514/episodes?page=6 ]https://api.thetvdb.com/series/72514/episodes?page=6 [/url]
2019-09-26 03:24:44.8511 [INFO] THETVDB-CLIENT Call to THETVDB: [url=https://api.thetvdb.com/series/72514/episodes?page=7 ]https://api.thetvdb.com/series/72514/episodes?page=7 [/url]
2019-09-26 03:24:45.1022 [INFO] THETVDB-CLIENT Call to THETVDB: [url=https://api.thetvdb.com/series/72514/episodes?page=8 ]https://api.thetvdb.com/series/72514/episodes?page=8 [/url]
2019-09-26 03:24:45.4575 [INFO] THETVDB-CLIENT Call to THETVDB: [url=https://api.thetvdb.com/series/72514/episodes?page=9 ]https://api.thetvdb.com/series/72514/episodes?page=9 [/url]
2019-09-26 03:24:45.7169 [INFO] THETVDB-CLIENT Call to THETVDB: [url=https://api.thetvdb.com/series/72514/episodes?page=10 ]https://api.thetvdb.com/series/72514/episodes?page=10 [/url]
2019-09-26 03:24:45.9317 [INFO] THETVDB-CLIENT Call to THETVDB: [url=https://api.thetvdb.com/series/72514/episodes?page=11 ]https://api.thetvdb.com/series/72514/episodes?page=11 [/url]
2019-09-26 03:24:46.0300 [INFO] THETVDB-CLIENT Call to THETVDB: [url=https://api.thetvdb.com/series/72514/episodes?page=12 ]https://api.thetvdb.com/series/72514/episodes?page=12 [/url]
2019-09-26 03:24:46.1380 [ERROR] MEDIASYNC-JOB Can't seem to process show Looney Tunes, check the logs for more details! 
2019-09-26 03:24:46.1380 [ERROR] System.InvalidOperationException: Sequence contains more than one matching element
   at System.Linq.Enumerable.SingleOrDefault[TSource](IEnumerable`1 source, Func`2 predicate)
   at EmbyStat.Jobs.Jobs.Sync.MediaSyncJob.ProgressMissingEpisodesAsync(Show show, CancellationToken cancellationToken) in d:\a\1\s\EmbyStat.Jobs\Jobs\Sync\MediaSyncJob.cs:line 307
   at EmbyStat.Jobs.Jobs.Sync.MediaSyncJob.GetMissingEpisodesFromTvdbAsync(IReadOnlyList`1 shows, CancellationToken cancellationToken) in d:\a\1\s\EmbyStat.Jobs\Jobs\Sync\MediaSyncJob.cs:line 278 System.InvalidOperationException: Sequence contains more than one matching element
   at System.Linq.Enumerable.SingleOrDefault[TSource](IEnumerable`1 source, Func`2 predicate)
   at EmbyStat.Jobs.Jobs.Sync.MediaSyncJob.ProgressMissingEpisodesAsync(Show show, CancellationToken cancellationToken) in d:\a\1\s\EmbyStat.Jobs\Jobs\Sync\MediaSyncJob.cs:line 307
   at EmbyStat.Jobs.Jobs.Sync.MediaSyncJob.GetMissingEpisodesFromTvdbAsync(IReadOnlyList`1 shows, CancellationToken cancellationToken) in d:\a\1\s\EmbyStat.Jobs\Jobs\Sync\MediaSyncJob.cs:line 278

Somehow I has duplicate season folders Season 1966 and Season1966.  Once I corrected this issue the error was resolved.

Edited September 26, 2019 by Happy2Play

[Happy2Play 8296]

On a new installation Missing Episodes worked for all series instead of 5 as shown above.  So there would appear to be a issue when auto updating.

2019-09-26 14:21:04.2338 [INFO] MEDIASYNC-JOB We need to check 387 shows for missing episodes

Still same issue with Looney Tunes on a clean/new install.  But I see you made GitHub Issue 950.

 

Minor title issue (TV and Movies)  

 

 

**Same with common.disksize when looking at missing episodes.

Edited September 26, 2019 by Happy2Play

[reggi 409]

@@Happy2Play and @@WilhelmStroker, I created a fix for both of your issue and it should be available in the nightly docker container (version 0.2.1-dev.422). I changed the implementation so it's possible to have duplcate season folders.

 

As for the issue that only 5 shows are checked for updates. That's because EmbyStat will only update shows that have new episodes on Emby, is a new show or tvdb has an update for that show. Meaning, if you would do 2 sync job runs after one another you will see that the second run will not check any show for missing episodes.

 

For the translation error, can you clear your cache @@Happy2Play (or just remove en-US.json from your cache)? I checked and the translation keys are in the en-US.json file so they should be visible.

[WilhelmStroker 96]

As I'm not using docker, any chance to get the fix in a windows build? Thanks!

[reggi 409]

In attachment you can find the 7z of the Windows artifact. I hope it works

 

I can't use zip because file is over 90Mb

EmbyStat-win10-x64-v0.2.1-dev.422.7z