Emby Performace drops after adding new media

[nagetech 66]

This is running a Dell R720 Server, running windows 2016 on bare metal. Radarr, sonarr & lidarr are the only other apps on the server. The C:\ Drive is 2 x Samsung 860 1TB drives in Raid 1, while all the hard drives for storage are split into 2 pools using Drive pool.

 

 

Hope that helps!

[Carlo 4331]

I'd try shutting down everything but Emby to see if the problem goes away.  If so add one program back at a time.

If you still see a problem with only Emby running we need to dive in further but right now the logs make it appear that Emby is starved of resources and this creates high retrieval times.

[nagetech 66]

I actually did try that already, before I reached out to you guys. This server install is only about a month old, and I have removed everything and left just emby, drive pool running on the unit. Emby starts of SUPER fast, but as mentioned before, it's performance degrades over time.

[Carlo 4331]

Something is very wrong then as the times shown in our logs to do things are very high.
This could very well be from the performance hit you get running RAID 1 which is not a good idea for DB apps and things that write a lot.

[nagetech 66]

Do you recommend I move the emby installation elsewhere?

 

is it possible to shut down emby, move the whole directory over to a drive that isn’t in raid 1 and try that ?

 

im surprised to bear though the raid 1 could be causing that, this has a h730 controller with 1gb of ram. My benchmark write speeds to the raid array is around 600mb. I don’t have a lot of users and the server only serves emby. 
 

starting to become a real noodle scratcher  

 

[PenkethBoy 2063]

are the samsung drives pro or evo?

and do you have trim enabled - and have you checked its working as on win 10 it does not run automatically 

[nagetech 66]

Drives are evo I believe. 
 

I will check later about the trim status in windows. I want to say yes, but never hurts to make sure 

[PenkethBoy 2063]

then being evo's is possibly your problem - not sure - but dont think samsung ssd are raid rated/compatible - you would need to check that

evo's have a small cache of ~11gb after that the disk only performs at the native speed of the disks which being evo is not 500+mb/s - think its a lot less

with pro's they dont have a cache and can perform at 500+mb/s all the time - so more consistent results and why they cost more

also if the trim is not working that will slow things done as well - its a bug in windows 10 that although you can set it to trim the drives every week - it does not do it - suspect win 2016 will be the same

[nagetech 66]

I can follow that logic.

 

I did just check, and Server 2016 does have trim enabled.

 

My older server (another poweredge) also had 2 x samsung drives (850s) in the same config and it didn't show these symptoms.

Even if the SSD slows down a bit, so much so that emby has that issue? Wouldn't I feel that across the whole OS/system?

If you feel the cache may be an issue, should I disable the cache for the Raid 1 Mirror in the Raid config? If it's not enabled, I believe that means the controller doesn't use the Drive's cache.

[PenkethBoy 2063]

evo's slow down quite a bit

what controller are you using for the raid - is it HW raid or software Raid?

do you have the os and emby installed on your os drive?

the cache on evo's cant be disabled as far as i know

and the cache on the controller - is it a true cache or "just" for power outages assuming you have the battery as well

[nagetech 66]

H730 Hardware Raid

 

Yes Windows & Emby live on the same Raid 1

On the raid controller, there are options for Disk Cache (enable / disable)

Not sure if that is beneficial though.

I've run benchmarks on the drive a few times when I was having the problem (just to see if it was a drive issue) and even then my benchmarks are the same before, during, and after emby acts up.

Just sharing information about my testing.

[PenkethBoy 2063]

benchmarks wont tell you that much as they a synthetic and dont write that much to a disk - so should sit in the cache of the ssds

what i would do is copy a large file - say a 40gb+ movie to the raid - watch what speed windows says its writing at and do you get an initial high speed and then it drops down 

then copy the file on the raid so its read and writing to the same disk - again what speeds do you get

do this second copy immediately after the initial copy finishes - this will not give the ssd's time to flush their cache 

[PenkethBoy 2063]

you dont have the raid included in your DP setup do you?

[nagetech 66]

The 40 drives I have are in a Supermicro enclosure hooked up the the server via a LSI HBA Card. No raid no nothing, each disk is just passed through straight to Windows. I'd never try to run DP on top of a RAID set. I feel one goof and the Data Recovery on that will cause the hair to insta-fallout my head.

 

[PenkethBoy 2063]

ok late here pick up tomorrow

[nagetech 66]

Thanks for jumping in to help out! Super appreciate it

[nagetech 66]

Just wanted to share the benchmark I ran. Figured may help with the diagnostics. In my humble opinion, it looks like the drives are doing well, but if there's something I'm missing let me know guys!

[PenkethBoy 2063]

Morning

ignore the benchmarks for now

and do the tests i posted last night to swamp the cache

[nagetech 66]

Howdy everyone!

Well I have great news. In troubleshooting this (with the help of you guys giving me ideas) I ended up fixing the problem plus tweaking a setting that's helped give my new servers a little pep.

First, the fix..... You guys might as well start rolling up the newspapers and magazines to crack me across the head. I had completely forgotten that I had OMBI installed on the system. I started playing with it, but got busy and left it on the back burner. Turns out, for whatever reason, give or take every 5 hours or so, ombi will just start slowing down emby. It just so happen that I was there using it to witness it happen just as I added movies. Talk about timing........ I've had ombi turned off and Emby is performing even faster than before!

Sorry about being a bonehead and forgetting about that app....

 

The auxiliary good news (at least for me) While I was chasing down the possibility of it being something with my Raid controller or Raid config, I brushed up a bit on the H700 controllers and RAID in general. Turned out on both my servers I had Direct I/O instead of Cached I/O. When I switched them, I noticed an immediate improvement, not only on the emby server, but on my other database server for my business software. Unexpected win!

 

This community is awesome, and all you guys always jump in to help out a fellow nerd. Thank you for all the help, never would've stumbled my way into the fix without you guys!

[Happy2Play 8356]

Glad you figured it out.  But yes I believe there are outstanding issue with OMBI.

Ombi Causing Emby GUI To Drastically Slow Down · Issue #3687 · Ombi-app/Ombi (github.com)

[nagetech 66]

Yep, that about explains all my symptoms.

I'll just wait until OMBI figures out that issue and enjoy my newly reclaimed speed

[Carlo 4331]

The need, the need for speed.

So glad you got it figured out.