Large Libraries - Constant Time Outs

[bakes82 167]

It seems like if you have a larger library Emby cant handle doing some basic things.

1 - Resetting watched status, my TV/Movie folder always gets a 504 time out when I want to reset the view state for the whole library from testing

2 - Filtering trying to filter to "Watched" movies also times out

 

I assume this uses sqllite?  is it index?  Can we use mysql instead?

[Carlo 4560]

Hi, mind if I ask you what you consider a "large" library?

How many movies do you have?  How many total episodes? How many total albums?

[bakes82 167]

> 12k Movies

> 166k Episodes

 

Smaller libraries like 4k TV (5kish episodes), 4k Movies (1kish) are fine

Edited October 8, 2020 by bakes82

[ebr 16169]

And you are trying to change the watched status for all of those at once?

[bakes82 167]

Not sure what that means lol.  You guys built a UI that lets me select 1 to N options on the home screen, I select "Movies" library then click the ... click unplayed.  Times out.

Your UI for multi selection of movies is horrible lol, you cant even shift click, you think Im going to manually click on thousands of movies?  I want an easy way to reset my watched status, I assumed this was the best way, seems its not,  Like even resetting TV doesnt seem to reset episodes, you need to filter TV to episodes and then do it that way.

[Luke 42077]

Hi, can you please attach the emby server log from when you experienced this? Thanks.

[Carlo 4560]

OK thanks for those sizes. You're system is up there in size but no where near outgrowing SQLite.  I know this because my system is quite a bit bigger and it's still quite snappy with a bit of TLC given it, every once in a while.

One thing I would recommend when you start getting into larger sizes like this is to occasionally do a VACUUM on the library.db file (after making a backup) especially if you've not done this in a while (or ever) or if you've been deleting/adding files in bulk.  You could think of this as an "unfragment" operation similar to unfragmenting an HDD only in a database.

What OS are you running your Emby Server on? I'll see if I can give you directions to do this.  Then if this doesn't help speed things up, I'd suggest sending the logs as requested to Luke.

[PenkethBoy 2068]

also the spec of the machine and type of disc the db is located on

[bakes82 167]

Where is the vaccum library button?  If its not in the UI someplace I wont be doing it. 

[bakes82 167]

Log attached.  Sever specs IDK off top of by head, this is hosted remotely in a docker, Im just testing Emby to see how it works.  Plex seems to better overall so far on the same box.  It would be writing to a NVME and should have at least 6core and 32gb.

Yes I know people have large libraries 30k movies isnt a far stretch lol.  But others have 3k and think its alot so why I said "larger" lol

Plex does have built in db optimization, that not here?

 

Since it looks like this kicks of a scheduled job does it still run after the 504 error?  Why can i see it running some place.   It "looks" like after a bunch of time the UI finally catches up and the "continue" watching section gets updated and the rest of the UI catches up but again to me as a user, I dont know it still running.  But I filtered again for "played" episodes and found none.  But I have weird error where a show is saying its played but ive never watched an eipisode of it, but also looks like there no files for it.

embyserver.txt

Edited October 8, 2020 by bakes82

[PenkethBoy 2068]

well that logs dominated by trakt api calls

also the cpu only has 2 cores

so you slowness and 504's very likely to be related to that

also you have loads of plugins loaded - prune that down to the minimum - will get you a more responsive server

and get some more cpu power - as you dont know what you have - did you sepcifiy what you wanted or did you assume

also where is the media located relative to the emby server - local or remote from the hosted server

 

basically you need to spend some time tuning the setup - but most of your choices have introduced delays over a local (home) server - if thats what you want then fine but a bit of work and config changes could improve things markedly - although a local server will always be faster and more responsive like Carlo's

 

[bakes82 167]

Well this is a dev box its not my main box lol, I assumed its the same as my primary but im not paying for it so yea know.    The media is local.  I'll see about getting cpu increased   could explain a few issues.

[Ronstang 293]

1 hour ago, bakes82 said:

Where is the vaccum library button?

I'd like to know this too since lately I have been completely reorganizing my hard drives and almost all my files have moved several times to and from EMBY shares in order to get the order I wanted.

[PenkethBoy 2068]

its a switch in the config.xml file - gets reset on server boot

[CBers 7450]

2 hours ago, bakes82 said:

Where is the vaccum library button?  If its not in the UI someplace I wont be doing it. 

 

28 minutes ago, Ronstang said:

I'd like to know this too since lately I have been completely reorganizing my hard drives and almost all my files have moved several times to and from EMBY shares in order to get the order I wanted.

 

[Carlo 4560]

2 hours ago, bakes82 said:

Where is the vaccum library button?  If its not in the UI someplace I wont be doing it. 

 

30 minutes ago, Ronstang said:

I'd like to know this too since lately I have been completely reorganizing my hard drives and almost all my files have moved several times to and from EMBY shares in order to get the order I wanted.

Two ways to "easily run VACUUM; Either way shut down emby.

1st method is to modify the system.xm file and change false to true and restart Emby.

<VacuumDatabaseOnStartup>false</VacuumDatabaseOnStartup>

You can find this on Windows in C:\Users\Carlo\AppData\Roaming\Emby-Server\programdata\config     (change Carlo to your username Emby runs under).

2nd method is with use of a sqlite database util.

You need to have a sqlite db manager and there are several from GUI to text based and any will work for this. "DB Browser for SQLite" is an easy to use version for Windows.  There are 2 EXEs but you want to use "DB Browser for SQLite.exe". You open the database in the GUI just like any other windows program.  library.db for most windows system will be in 

C:\Users\Carlo\AppData\Roaming\Emby-Server\programdata\data    (change Carlo to your username Emby runs under).

Once you have library.db open. Click the EXECUTE SQL tab and type

VACUUM;

then click the execute button and wait until it says it done which could take seconds to minutes.  You'll know it's done when you get a result similar to:

Execution finished without errors.
Result: query executed successfully. Took 35667ms

Exit and restart Emby.  Not much to it at all and it can make a huge difference in performance.

Obviously the first method is easiest but the 2nd method might be easier for those who know and use DBs or just like to know what's actually happening.

-----------

I completely agree with other comments on cores but there are only two plugins I'd remove as a test and they are:

InfuseSync 1.2.1.0
Trakt 3.4.4.0

Any "syncing" plugin can slow you down if not configured correctly and it's easy to test by removing those two plugins.

Edited October 8, 2020 by cayars

[bakes82 167]

So based on that post you guys dont run any process that do this automatically like plex and I now have to schedules these if I want to keep emby working fine?

/var/packages/EmbyServer/target/sqlite/bin/sqlite3 library.db "VACUUM" 
/var/packages/EmbyServer/target/sqlite/bin/sqlite3 library.db "ANALYZE" 
/var/packages/EmbyServer/target/sqlite/bin/sqlite3 library.db "REINDEX" 

[Carlo 4560]

The way Emby does it puts YOU in control so you know what DB operations are being performed.  Unless you get to a big library this really isn't needed and even then it's a matter of how you add your data.

Having exclusive access to the database to do a full VACUUM is better than how Plex does it ongoing which only removes space but doesn't reorganize the tables or indexes.

It takes longer to write this post then change the option and restart Emby to run it!  LOL

[Happy2Play 9780]

Ideally this switch should be added to the Dashboard somewhere.  

[bakes82 167]

Except plex has it built in and does whatever magic they want it to do w/out me knowing or having to be involved.  If I wanted to be a DBA I would be but I wouldnt be using sqlite either.  Im not sure all the magic it does when it does it stuff but guess what, I can do it from the UI, its easy and intuitive.  Some one needs to write a plugin for this already.

 

I could run the same vacuum command on the plex db lol.  

 

@chef Can you make a simple plugin to this

This should be it.  If you can add the other options also.

using (SQLiteCommand command = m_connection.CreateCommand())
{
    command.CommandText = "vacuum;";
    command.ExecuteNonQuery();
} 

[bakes82 167]

Just now, Happy2Play said:

Ideally this switch should be added to the Dashboard somewhere.  

I dont hold my breath for anything here.  Looks like it was added in 2019 its almost 2021 and its not in the UI yet.

[Happy2Play 9780]

At the same time this doesn't really apply to a Home user does it?

[bakes82 167]

You tell me?  Dont many home users have very large cloud backed libraries now days   I assume everyone has access to like 1PB cloud drives for cheap.  Also Emby seems to have the framework for large scale where plex caps at 1000, so honestly Emby seems to be targeting those people since thats how you get $$$.  So you would think keeping the db clean and running great should be a higher focus than the home user base which can get and use emby for free.

[Happy2Play 9780]

Well if I never use this feature and never have a issue or use this feature and see no difference, then what benefit is it to me?  Only difference I have seen is size of db.

But my library is less than half the size.

Edited October 8, 2020 by Happy2Play

[bakes82 167]

1 hour ago, Happy2Play said:

Well if I never use this feature and never have a issue or use this feature and see no difference, then what benefit is it to me?  Only difference I have seen is size of db.

But my library is less than half the size.

The feature should be a system task that runs weekly honestly, why wouldnt you want to run it?  If it was enable and running and things ran better would you be happy?  Most things like optimization help everyone.