Emby Performace drops after adding new media

[nagetech 66]

Howdy everyone!

Recently got new gear and migrated my Emby server on over. I installed Emby and used the backup plugin to import all my users and settings. Bing Bang Boom!

BUT..... I have noticed a new issue I didn't have with my old server. Emby loads up fast on all my clients (both internally and externally when it is initially launched. Even after a bit, I didn't notice any drops in performance.

Now, the trouble starts when I add new files to my collection. If I throw in, say 12 new movies and 3 seasons for a new show, emby springs into action and starts importing them into the library. During that time, WebUI performance drops and all my clients show a lag when browsing the site. Just clicking on a movie to load up the info on it takes a bit. A restart of the Emby app, and all is well.

Hoping someone can help out

@LukeI have PM'ed you my server logs.

 

Looking forward to hearing back from you guys!

[rbjtech 4330]

During the import, Emby is working to create the metadata and possibly the thumbnails (if you opted to do that) - this can be intensive if you have a low powered cpu.  If performance returns to normal after the import (check the logs), then this is likely all it is.

Some pointers to help with this is to ensure your emby system (database, cache etc) is ideally on an SSD drive and the media itself is on separate disks/controllers.

Edited January 30, 2021 by rbjtech

[nagetech 66]

Thanks for the response !

 

Server does have SSD drives for the OS and emby lives there as well. All my media are in a Supermicro 45 bay case with Stablebit Drivepool pooling them.

Server is a R730, Don't have the specs off the top of my head, but dual xeon with 48 cores and 84GB RAM. Unit definitely has the horsepower.

Even after the scan the performance never bounces back. I'm trying really hard to think logically why emby would stay slow even after it is done importing the data.

 

Hoping @Luke Finds a clue in my logs.

Edited January 30, 2021 by nagetech

[rbjtech 4330]

ok - agree it sound like you have more than enough horsepower for emby .. haha.

Have you looked at performance metrics such as CPU idle after adding media - does it drop to the same level as before adding media ?  How about the emby process itself ?  Are there any ffmpeg processes running ?

It is possible that Drivepool is rebalancing drives perhaps - I'm not familiar with the product but I do know that some pooling solutions try and balance data across striped platters for example.

[RanmaCanada 360]

It's more than likely the balancing from Drivepool.  As you are adding the files, Drivepool overwhelms your system while trying to allocate the files over your pool for redundancy.  I run into a similar problem during mover on UNRAID.  As files are being moved off the cache drive and onto the array, parity is being built and it causes everything to crap out (monthly parity checks are just as bad).  Your processors may be fast, but you are still limited to the speed of your slowest drive.

[Luke 37247]

You've enabled thumbnail image extraction, and the help text underneath the option does warn that this will consume a fair amount of resources.

[nagetech 66]

@Luke

Thanks for jumping in amigo!

 

I have been running emby for years so do know of the performance tax for certain features. What I'm not understanding is Luke, once its done scanning the library, no tasks are running and the system is idle (I confirm by checking drivepool console and see there are no reads or writes going on with the pool) emby still runs sluggish as heck.

 

I 100% get if it was scanning and extracting, then sure the slowness would make sense.

 

But if emby isn't doing anything anymore, it should spring back.

 

Concerning Drive Pool, I have it setup to replicate in real time, but balancing is scheduled for 1AM. While I do understand the notion having 40 disks spin up and start working, I have it designed as a split pool. One is for TV Shows and one is for movies. So what one pool is doing has nothing to do with the other. If emby lets say is scanning my Movies and I'm adding tons of files & Drive Pool is going basaltic thrashing those discs, in theory, it shouldn't affect the drive pool performance of the TV show drives. Hope all that made sense!

[Carlo 4331]

Except for all the IO going through the buss transferring data from one disc to another which can severely bottleneck your IO system.

What does your CPU look like when it's sluggish?  Do you see high CPU use?
How about log activity?  If you look at your log do you see a lot going on there?

[nagetech 66]

Sorry for the late reply!

When Emby is dragging, Task manager shows everything ok. CPU and Memory Usage isn't spiked, DrivePool doesn't show any drive activity at all.

As a test one day, I didn't add an new media at all to emby, and it ran slick all day. Went back to turning on auto organize and I added my own media files, and the issue came back. Emby will run strong at first, but the more media it ingests into the library, the higher the chance it will get sluggish and stay there till a restart.

Could my media library just be too big for emby? I have well over 12k movies, never mind my other collections.

[Luke 37247]

No, it's not too big.

[nagetech 66]

Just had a thought, don't know if this matters.

 

The server has quad NIC, and I'm using 3 out of 4.

1 Nic is direct connected to another server to transfer files between them. Has no gateway set in the IP settings

1 Nic I designated to serve just the inside of my network. Gateway is set to my main router

Final Nic is designated as my "for outside clients" . Gateway is the same, but this is the one the router forwards the ports to the emby server.

 

I don't think that would matter, and reading the logs shows emby does route internal clients to the internal nic, and outside clients are on the "external" NIC

 

Looking forward to more ideas!

[PenkethBoy 2063]

What controllers are you using with DP and your disks?

Is DP up to date? are you on the beta version of DP?

Do you run Stablebit Scanner and/or Cloud Drive?

Do any of your pools also include a clouddrive?

Do you backup to the cloud? or Locally and if so is this set to run on a file change etc

As you have a large number of drives and i suspect a large amount of data - any process that runs like above would take hours (possibly days to complete) which might be what you are seeing and this is what appears to slow emby down. _ might not be this but we need more detail to try and find the slow down.

 

[rbjtech 4330]

Lets try something - make a 'change' in emby, but don't write new files to the DrivePool.

If you pick a folder - rename the video file to .tmp - let emby pick this up, 'remove' the entry and it should no longer be available.  Once it has disappeared, go back to it and rename it back to the original file type (.mkv etc).  emby should then pick this back up again and re-add it.

If the slowdown then happens as before, then I would suggest it is not anything to do with disk I/O/Drivepool - as other than a rename, nothing has changed on the file system.

[PenkethBoy 2063]

That wont eliminate an IO related problem though

[rbjtech 4330]

26 minutes ago, PenkethBoy said:

That wont eliminate an IO related problem though

We don't know if it is I/O related - that's what we are trying to identify here.

The other way is to dump a load of data on the Drivepool but outside the emby libraries - if emby doesn't slowdown during this operation then it's likely not the I/O.

Logs would also be good - as they show latency on each http request and may give clues.

[PenkethBoy 2063]

we dont know much but your test wont eliminate it either way

[Carlo 4331]

I like the "tmp" test to rule out base/core Emby issue.

I'd also try adding media manually without Auto Organize as maybe that could be causing the issues as well.

In situations like this it's often a process of elimination to find the culprit. 

[nagetech 66]

Howdy everyone!

 

Sorry for the late reply, been nursing a bum back.

Just a little feedback, I think I MAY have figured it out. I forgot to mention I also had Primo Cache Installed on the server. I was trying it out, but quite frankly it didn't feel like it was doing anything!

Uninstalled it since I believe if you ain't gonna use it, then why leave it installed. Since then, I think emby has been running way way better. Not sure how Primo Cache was holding it up, but going to keep trying it for the next few days while I'm in bed and let you guys know!

 

I super appreciate the help, support & ideas. next time I have a problem, I can use a lot of this as Pre-emptive steps before coming to the forums.

 

 

[Carlo 4331]

I can see how this would affect things.
Give us an update in a few days.

[nagetech 66]

Howdy everyone!

 

Just wanted to report back. Sadly, looks like Primocache wasn't the issue.

 

It looks like I need to reboot emby at least once a day to keep it "fresh", otherwise, it starts to slow down on all clients both locally and remotely. The movies stream fine, but browsing through emby has noticeable delays in between selections. The second I restart emby, everything works like a snap!

 

I can confirm these symptoms happen both with the system being busy adding new media, or just sitting idle only serving requested media. Emby is the only application on the server showing these symptoms. It's a fresh install of Windows Server 2016 on a beefy server, so I'm 99.9% confident it's not the environment.

 

The only other thing I can note is, right before I restart emby each time, it always appears that the memory is hovering around 900mb of usage, not sure if that matters.

 

@Luke, I'm PM'img you some fresh logs in the hopes they give you more clues

[Luke 37247]

You could try removing the auto organize plugin and see if that helps.

[nagetech 66]

Unfortunately, it looks like the only time emby ever runs ok for long periods of time is when I don't add any new media at all. Even then, after some days it starts to show some lag.

 

Is there anyway to setup a scheduled task to auto restart the server each day lets say 5AM? That way emby is fresh for each starting day

[Happy2Play 8356]

7 minutes ago, nagetech said:

Is there anyway to setup a scheduled task to auto restart the server each day lets say 5AM? That way emby is fresh for each starting day

Not without a plugin guru reviving the Restart plugin on GitHub.

You can create a script with the command from API (link at bottom of dashboard).  There are a few topics out there

 

Edited February 11, 2021 by Happy2Play

[Happy2Play 8356]

When was the last time you vacuumed the database?  Still wish there was a switch via the UI.

4.2.0.32 - Vacuum database config switch - Emby Server - Emby Community

[rbjtech 4330]

8 hours ago, nagetech said:

Is there anyway to setup a scheduled task to auto restart the server each day lets say 5AM? That way emby is fresh for each starting day

So two options here -

1) if you want to gracefully restart the entire server - then depending on your base Operating System, just setup a schedule job to restart it.  In Windows, this is 'Task Scheduler' in other Linux based OS it will be 'cron' or some similar alternative.  As long as it is a graceful restart, then all applications are properly closed and emby will shutdown properly.  If emby is not run as a service (see below), then you'll need some way of auto logging back on again to re-start emby - using Windows, this is done by using netplwiz - a quick google will tell you how.  The windows immediate restart command is 'shutdown /r /t 0' and must be run as Administrator.

2) the other option is to run emby as a 'Service' and then simply use the same schedule to just restart the service, there is no need to restart the entire machine. (not sure if a restart fixes your issue or not ?  If not, then don't use this option ... 

--

Maybe there are other options to restart certain aspects of your system - I know 'netsh' can be used to reset the tcp/ip stack on Windows for example - maybe there are ways to just restart the IO system which is more likely to be your issue, but certainly worth testing 'netsh' on the command line when you get a 'slowdown'.