Developing a standalone embystat server

[mgworek 121]

never mind. I realized I wasn't running the nightly build with the new wizard. 

 

Everything good now.

 

Thanks

log20190417.txt

Edited April 17, 2019 by mgworek

[mgworek 121]

Is anyone having a problem getting it to open in Safari? It just sits on the splash screen cycling through the logo. Runs fine with Chrome and Firefox. I used to use it with safari.

 

Update:

 

Works fine in Safari Tech Preview. I will investigate more. It's either the current safari version or something on my end.

 

Thanks.

Edited April 17, 2019 by mgworek

[Pog22 50]

Right I got this running behind Nginx reverse proxy and all scanned in which took a while (~100TB). It's running in docker so I've linked the Emby server by hostname and local port to bypass Nginx. One thing, could you possibly add in the option to use the Emby wan address in the library links

[mgworek 121]

My first scan scan crashed yesterday but 2nd try everything was good. This morning I noticed my docker was crashed. I started again and had no movies anymore. Ran the scan again and it crashes each time. 

 

I get about 5 of these in the log and then the docker crashes again.

 

"Microsoft.Data.Sqlite.SqliteException (0x80004005): SQLite Error 5: 'database is locked'.

 

obviously  my db got locked, any idea how or how to unlock it?

 

Thanks.

2019-04-18.log

Edited April 18, 2019 by mgworek

[jaycedk 382]

Check out the git

https://github.com/mregni/EmbyStat/issues/645

 

Same issue but on windows.

[mgworek 121]

Check out the git

https://github.com/mregni/EmbyStat/issues/645

 

Same issue but on windows.

 

 

Sorry about that, I did a quick search in this topic for "locked" but had no results.

[Spaceboy 2494]

hi reggi, i'm still having difficulty getting embystat to complete the initial sync with emby. sometimes it seems like it has completed but then it wipes everything and starts again. i had one similar error in emby to that which you pointed out on hitman 47 but i can't see any other things to look at and it seems to stop at different places. the longest it has ran for so far is 37 hours. i do have a fair size library but i imagine other people have much more. i wondered if you had thought about breaking the initial sync down in some way? either by batch or perhaps media type?

 

last 2 days logs attached. 2019-04-18.log2019-04-19 (1).log

 

cheers

[Pog22 50]

I have 100TB, a full sync is taking 6.5 hours

[reggi 409]

Hey guys, sorry for not responding sooner. I have some busy days now with my work and will have time on sunday to check the thread again.

 

As for @@Spaceboy, said to hear it's not running! I do have to see how I can refactor the initial sync into smaller pieces and make it incremental as well so EmbyStat doesn't DDOS your Emby every night.

[Spaceboy 2494]

I have 100TB, a full sync is taking 6.5 hours

cool, I have more

[adrianwi 238]

Is anyone noticing an alarming increase in the HDD space used by the Docker host when running this?

 

I seem to have something consuming HDD at an alarming rate, and whilst it's easy enough to increase the space, I don't think anything I'm dockerising requires that much space.  It might not be related to this, but seems to have started around the same time I installed the latest version to test.

[Pog22 50]

I like this, but I've had to give up on it as my server is a remote server behind an Nginx reverse proxy and I'm also using Cloudflare DNS. To get images and links working I would need to use the Emby Wan URL which with a 100TB library creates a huge amount of traffic out through Cloudflare and back when Embystat runs in docker on the same host.

 

Feature request, It would be great if I could use the local port and hostname and have the option to amend the public domain URL to the image and library links, Ombi has this Externally Facing Hostname setting

[reggi 409]

@adrianwi 

Strange that the HDD space is increasing fast in your docker, this shouldn't happen because the only thing that is written to DB is the database file.

 

But it looks like everyone is having major performance issues with the current version, so best thing is to restructure the media sync job and see what can be done. 

Currently I'm looking to switch SQLite with LiteDB because it seems more thread/process safe. And I have to create some paging in the http calls to Emby so the memory doesn't blow up every time you do a sync.

Or you guys just have way to much media, hehehe  

 

Sorry for having this kind of major issues still, I hope we will get to a better version soon.

 

Or you guys just have way to much media, hehehe  

[reggi 409]

So, yesterday I was working on a new solution for the performance and memory issues with EmbyStat and SQLite and I came u with LiteDB, it's a file database but document based and it looks very promising.

I did some tests with my collections (1061 movies and 195 shows) and it takes me now just over 9 minutes instead of > 30 minutes. So that looks very good.

 

Maybe I can create a standalone app for this console application so people with huge collections can test the performance as well before I implement this in EmbyStat. I'll create a package later today.

 

EDIT: forgot to add test result screenshot

Edited April 21, 2019 by reggi

[jaycedk 382]

Nice

Looking forward to testing that

[reggi 409]

So, I have 2 version of the testing tool packages. The packages are self contained so you don't need to install the dotnet core framework yourself.

 

Usage of the test app is fairly easy

tester --username <Emby username> --url <url>:<port> --pass <Emby user password>

Let it run for a while, at the end it will show some sync timings. The app will generate a data.db file, you can always use this viewer to check the data => https://github.com/falahati/LiteDBViewer

It's possible you get a 404 when checking for missing episodes, that's probably because there is a unknown thetvdb id in the Emby database.

 

 

It would be create if some of you could run this app and share the results. That way I get a better idea if it is a good idea to switch to LiteDB.

 

Tester win-x64.zip

Tester linux-x64.zip

[kesm 25]

I had to launch this command before the test, also your command should be Tester and not tester : 

 

 

chmod +x Tester

 

but here is the result : 

Collecting run statistics
---------------------------------------------------
RESULTS         Count   Time
Collection      6       278 ms
Movies          996     6587 ms
Shows           156     172411 ms
Episodes        12561   157093 ms       (count => total missing episodes, time => time to check for missing episodes)
People          30813                   (no time, people are synced while syncing media
Total                   336369 ms
---------------------------------------------------
All done!

with some 404 before

Edited April 22, 2019 by kesm

[reggi 409]

Thx for the output and extra instructions! Your sync time looks very good as well.

[jaycedk 382]

Hi there

Just finished with your test script of liteDB.

 

Collecting run statistics
---------------------------------------------------
RESULTS         Count   Time
Collection      11      330 ms
Movies          4086    10050 ms
Shows           381     11067205 ms
Episodes        27      308001 ms       (count => total missing episodes, time => time to check for missing episodes)
People          86626                   (no time, people are synced while syncing media
Total                   11385586 ms
---------------------------------------------------
All done!

[Happy2Play 8292]

With 404 error

Collecting run statistics
---------------------------------------------------
RESULTS         Count   Time
Collection      8       119 ms
Movies          3976    13742 ms
Shows           360     2172045 ms
Episodes        3278    879808 ms       (count => total missing episodes, time => time to check for missing episodes)
People          129017                  (no time, people are synced while syncing media
Total                   3065714 ms
---------------------------------------------------
All done!

[neik 837]

With one 404 error:

Collecting run statistics
---------------------------------------------------
RESULTS         Count   Time
Collection      9       577 ms
Movies          625     11805 ms
Shows           121     743127 ms
Episodes        1494    139092 ms       (count => total missing episodes, time => time to check for missing episodes)
People          30475                   (no time, people are synced while syncing media
Total                   894601 ms
---------------------------------------------------
All done!

[hshah 19]

I run EmbyStat in Docker on my Synology NAS, but the below test was done in Windows where both my Plex and Emby servers run:

Start syncing collections
Start syncing movies
Start syncing shows
Start syncing missing episodes
Collecting run statistics
---------------------------------------------------
RESULTS         Count   Time
Collection      5       210 ms
Movies          1395    4506 ms
Shows           229     1253548 ms
Episodes        7516    209482 ms       (count => total missing episodes, time => time to check for missing episodes)
People          56640                   (no time, people are synced while syncing media
Total                   1467746 ms
---------------------------------------------------
All done!

[reggi 409]

Thanks guys for the responses. I'll have a look today but seems like a good solution. Shows are still some kind of a bottleneck.

@@jaycedk, I see the shows sync itself is taking just over 3 hours on your machine. Can you tell me how long it took with the current EmbyStat implementation?

[jaycedk 382]

Im current at work, but will have a look in about 9 hours

[jaycedk 382]

Hmm well the total sync takes 2h50m with the old sql lite.