Developing a standalone embystat server

[reggi 409]

@@pju, forgot to mention. At this moment you can't stop a sync. The only way is to just kill it like you said. Normally that should give a real issue because I clean the database on every sync job. But it can create some issues.

 

Remark to every with a large amount of libraries (movie and TV's). To speed up the loading time on the statistic pages I calculate all possible combinations between libraries in the last sync job step. BUT if you have a large amount of libraries (for example 20 movie libraries) it will start calculation 20! possibilities ( > 5 271 287 700 000 000 000 combinations). What I will do is decrease the amount to ALL + all separately so you don't burn your CPU to much

[Sammy 738]

EmbyStat is eating up some clock cycles here. I have a Ryzen 3700x and it is running about 6% to 8%. Is this to be expected? It seems high to me.

[reggi 409]

That's high indeed and I think that's because of the web socket connection not optimized (and not used at the moment).

Currently it's not really a good idea to let EmbyStat run 24/7. Also because it only doesn't calculate user stats at the moment so no need to run EmbyStat 24/7 at this stage

[Sammy 738]

I will shut it down until the improvements are made. User stats is probably my biggest desire to have. Thanks!

[pju 10]

@@reggi,
Do you shows have any other Id (TVDB, IMDB, TMDB)?
As a rule of thumb those without <id> will not have any other id. It's typically recordings which I have decided to organize as shows because that fits my logic   

Show has no TVDB id it will always be processed as 100% collected?

Yes, that would be fine with me. But no doubt, there will be Emby users having shows with no <id> but some other ID. I only have found a couple of shows with this problem. An old TV-show having been digitally enhanced and sent again with the new one not in TVDB. And the new one is not in TVDB, fair enough.
Another where TVDB rejects to accept is as a show while IMDB list it (Swedish Millenium from 2010). So no TVDB id but IMDB id.

Maybe some kind of option - what to do at empty <id> ...?

 

I would btw appreciate a message in the logfile when an empty <id> is found.

 

 

The only way is to just kill it

I'll just kill it, then. After all it's just a virtual server - falling back to a snapshot in case of trouble is pretty easy.

 

Thanks for the bonus info about the calculation - I really have wondered what's going on

[reggi 409]

Sounds like a good idea to put a information log when no id is found. 

Currently I only connect to TVDB for episode information so I can't use IMDB as a fallback or something at the moment. Could come in handy later on maybe if I have the time.

[pju 10]

@reggi: Fair enough - definitely fine with me

If you find a handy way to generate a pseudo <id>, I will be happy to hear about it.

[vaise 307]

Updated the docker container for this and re-ran.

I am getting a completely clean log now - no errors at all.

Unfortunately, the stats are still off for the movies (TV Shows are however correct).

 

I did a bit of comparisons :

 

The embystats log is spot on, i.e I stripped out the detail and just showed the totals in the logs below, then compared this to what is in emby and it is correct : 

 

2020-05-13 13:00:03.3218 [iNFO] MEDIASYNC-JOB Starting job 

2020-05-13 13:00:03.4116 [iNFO] MEDIASYNC-JOB First delete all existing media and root media libraries from database so we have a clean start. 

2020-05-13 13:00:03.4119 [iNFO] MEDIASYNC-JOB Asking MediaServer for all root folders 

2020-05-13 13:00:03.4979 [iNFO] MEDIASYNC-JOB Found 17 root items, getting ready for processing 

2020-05-13 13:00:03.4979 [iNFO] MEDIASYNC-JOB Lets start processing movies 

2020-05-13 13:00:03.8140 [iNFO] MEDIASYNC-JOB Found 58 movies for 3D Movies library 

2020-05-13 13:00:05.0365 [iNFO] MEDIASYNC-JOB Processed 58 / 58 movies 

2020-05-13 13:00:05.0409 [iNFO] MEDIASYNC-JOB Found 290 movies for 4K Movies library 

2020-05-13 13:00:10.9930 [iNFO] MEDIASYNC-JOB Processed 290 / 290 movies

2020-05-13 13:00:11.0011 [iNFO] MEDIASYNC-JOB Found 2414 movies for Movies library 

2020-05-13 13:02:05.6200 [iNFO] MEDIASYNC-JOB Processed 2414 / 2414 movies 

2020-05-13 13:02:05.6224 [iNFO] MEDIASYNC-JOB Found 19 movies for Music Videos library 

2020-05-13 13:02:05.8274 [iNFO] MEDIASYNC-JOB Processed 19 / 19 movies 

2020-05-13 13:02:05.8297 [iNFO] MEDIASYNC-JOB Found 13 movies for Sports library 

2020-05-13 13:02:06.0351 [iNFO] MEDIASYNC-JOB Processed 13 / 13 movies 

2020-05-13 13:02:06.0371 [iNFO] MEDIASYNC-JOB Found 18 movies for Stand Up Comedy library 

2020-05-13 13:02:06.3016 [iNFO] MEDIASYNC-JOB Processed 18 / 18 movies 

 

I conclude therefore that the backend is correct and this is some sort of 'adding up or display issue' because when I use the embystat gui, the TOTAL movies is correct (2812), but if I use the library dropdowns, they all dont add up to this - i.e  - show the movies tab, then change the drop down for each library, it is not matching the above processed count in the logs i.e :

 

3D Movies - Log processed 58, stats say there are 28.

4K Movies - Log processed 290, stats say there are 144.

Movies - Log processed 2414, stats say there are 1581.

The other three libraries are correct but they each have less than 20 items (19, 13. 18).

 

These therefore also show a reduced disk space (as I reported earlier).

 

As there are now no errors in the logs to point to this issue, where should I look to assist you with this debugging ?

 

Thanks,

Edited May 13, 2020 by vaise

[reggi 409]

I was hoping that your TV show count would be of and not your movie count. Maybe Emby is counting the streams and not the movie count. 

I'll have to check later on why this is happening. Currently there is no use of debugging more since you are showing all needed logs. 

 

I'm currently rewriting the frontend completely because it was already getting buggy Let's hope that will go fast

[rechigo 293]

Hi reggi, can you provide the steps to build and run from source?

 

I opened the sln in vs 2019 -> Build Solution -> Ran executable in EmbyStat.Web\bin\Debug\netcoreapp3.1\EmbyStat.exe, it runs, says it's listening on 6555, but when I visit 6555 I get a 500. Nothing in the console indicating why, nothing in the logs indicating why either

 

I'm guessing there's an extra step that I'm missing?

[vaise 307]

I was hoping that your TV show count would be of and not your movie count. Maybe Emby is counting the streams and not the movie count. 

I'll have to check later on why this is happening. Currently there is no use of debugging more since you are showing all needed logs. 

 

I'm currently rewriting the frontend completely because it was already getting buggy Let's hope that will go fast

 

No worries.

Let me know if you want anything tested/done.  I am using the docker release on unraid.

From a personal note of fixes/enhancements :

- I would like a tick box in the config to 'not' check for duplicates as the system sees multiple formats of the same movie in the same folder as duplicates (_SD and _Bluray for example)

- As I have 4k movies in one library called 'Movies 4K', and the same movies sometime in another 'Movies'  library, the movies landing page thinks they are all duplicates.

- I would love a 'readonly' landing page also that I could publish via my reverse proxy

Apart from it loosing about 1011 media files, everything else is fantastic.

[MediaPlxMby 0]

Hello,

Not sure if I'm at the right place. I installed EmbyStat on my Ubuntu server and unable to start the service

embystat.service - Emby Stats program
   Loaded: loaded (/etc/systemd/system/embystat.service; enabled; vendor preset: enabled)
   Active: activating (auto-restart) (Result: core-dump) since Wed 2020-05-13 22:02:42 EDT; 4s ago
  Process: 26820 ExecStart=/home/embystat/EmbyStat (code=dumped, signal=ABRT)
 Main PID: 26820 (code=dumped, signal=ABRT)

 

Where could I find the logs to try to identify the cause.

 

Thanks
 

[crusher11 857]

How long should the initial sync take? In about ten hours it's gone from 85% to 85.8%.

[darkassassin07 432]

With 24k epps (217 series) and 1500 movies, sync takes around 20min for me.

[crusher11 857]

I have no idea how much of anything I've got because the stats aren't working yet, because it's still scanning. But it's been over 24 hours now and it's gone from 85% to 86.8%. It's going to take months at this rate.

[neik 837]

I have no idea how much of anything I've got because the stats aren't working yet, because it's still scanning. But it's been over 24 hours now and it's gone from 85% to 86.8%. It's going to take months at this rate.

 

You can also see those numbers in Emby's WebUI in each library. ;-)

 

For me it took like half an hour for the first sync.

[Spaceboy 2500]

If he has embystat installed in docker on his Synology it does take infinitely longer. From experience

[crusher11 857]

You can also see those numbers in Emby's WebUI in each library. ;-)

 

For me it took like half an hour for the first sync.

I have like a dozen libraries, so not quite that simple.

 

And yeah, it's a Docker install.

[mbc0 63]

If he has embystat installed in docker on his Synology it does take infinitely longer. From experience

 

I am running a 32 core threadripper, 128GB RAM, NVME 1TB Cache Drive and 12GB/s SAS Drives and been also stuck on 85% since installing 3 hours ago so I imaging there is something wrong here?

[mbc0 63]

Just to further update, there is no disk access from Embystat and still on 85%

 

It appeared to fly through the TV Series and has been stuck on the Movies for 4 hours and not budged so will kill the docker for now, nothing in the logs other than the message below 4 hours ago, would be great to hear from someone regarding this :-)

 

2020-05-15 10:41:58.6990 [iNFO] MEDIASYNC-JOB Calculating movie statistics 

 

[pju 10]

@mcb0: Can't tell if this is "too long time" because I'm used to see very long time, running the test on an VMware Workstation on a network drive.

But it's normal for me to see a very long period of time staying at 85%.

Think EmbyStat is doing a lot of calculations - which is why you don't see disk access.

Have a look at post #776: https://emby.media/community/index.php?/topic/56640-developing-a-standalone-embystat-server/page-39&do=findComment&comment=879978

[mbc0 63]

 

@mcb0: Can't tell if this is "too long time" because I'm used to see very long time, running the test on an VMware Workstation on a network drive.

But it's normal for me to see a very long period of time staying at 85%.

Think EmbyStat is doing a lot of calculations - which is why you don't see disk access.

Have a look at post #776: https://emby.media/community/index.php?/topic/56640-developing-a-standalone-embystat-server/page-39&do=findComment&comment=879978

 

 

Thanks for the reply, I haven't killed the docker yet so will leave running out of interest, considering I am running 4 VM's,  20+ Dockers I just can't compute why I am stuck at 85% for so long when I have such a massive amount of resources available and no disk access, probably due to me not understanding what is happening :-D

 

[crusher11 857]

Mine is continuing to trudge along at about 1% per day.

[mbc0 63]

yup same here, 85.2% after 30 hours, time to kill the docker lol

[crusher11 857]

Not sure what you think that's going to achieve, although yours seems to be going vastly slower than even mine. I'm up to 88.3% after three and a half days.