4.8.10.0 high system load

[DarinM 11]

I've been running Emby now for a little under a year and things have been good until this latest release.

It starts out fine, but after videos have been streamed for a while the load average on the system jumps.  Here is a load average example:
 

top - 20:35:40 up 12 days, 23:41,  1 user,  load average: 60.00, 60.00, 60.00
Tasks: 287 total,   2 running, 285 sleeping,   0 stopped,   0 zombie
%Cpu(s):  0.1 us,  0.1 sy,  0.0 ni, 99.8 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st
MiB Mem :  63529.2 total,   3031.1 free,   1003.0 used,  59495.1 buff/cache
MiB Swap:  15260.0 total,  15259.5 free,      0.5 used.  61425.8 avail Mem

    PID USER      PR  NI    VIRT    RES    SHR S  %CPU  %MEM     TIME+ COMMAND
1784893 emby      20   0 9258336 368100  91228 S   0.7   0.6   1:23.27 EmbyServer
 

Doing systemctl restart emby-server clears it out and brings the load average down.  Last night when I checked on it load average was 174.  

This is a Rocky Linux 8 system with 64 GB memory.  The processor is AMD Ryzen 5 5600G.  It also has an RTX 3050 for transcoding.

I had restarted Emby just before 6PM tonight and the load average was like this by 8:35PM.

Any suggestions or info are appreciated.

 

embyserver-63864189406.txt

[Carlo 4560]

Hi,

I didn't see anything in the log file.  Everything seemed to be running very quick as well.

    PID USER      PR  NI    VIRT    RES    SHR S  %CPU  %MEM     TIME+ COMMAND
1784893 emby      20   0 9258336 368100  91228 S   0.7   0.6   1:23.27 EmbyServer

What are you trying to show here? Looks very low to me.

[DarinM 11]

8 hours ago, Carlo said:

What are you trying to show here? Looks very low to me.

 load average: 60.00, 60.00, 60.00

This is the issue.  It only happens when something is played.  Restarting emby takes it to 0 where it normally sits.

[Q-Droid 989]

Something isn't right there.

top - 20:35:40 up 12 days, 23:41,  1 user,  load average: 60.00, 60.00, 60.00   <-- this
Tasks: 287 total,   2 running, 285 sleeping,   0 stopped,   0 zombie   <-- doesn't jibe with this
%Cpu(s):  0.1 us,  0.1 sy,  0.0 ni, 99.8 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st   <-- or this

Do you have other utilities you can use to check, like sar or vmstat? They should all get their info from the same place but these numbers don't make sense. You would want to capture this while Emby is active IF it is indeed the one causing the problem. The value 60 across the board is suspect and would reflect a very busy host.

 

 

[DarinM 11]

Sar is showing this:
 

08:00:00 AM     all      0.28      0.00      0.33      0.16      0.00     99.23
08:10:00 AM     all      0.10      0.00      0.29      0.09      0.00     99.52

08:10:00 AM     CPU     %user     %nice   %system   %iowait    %steal     %idle
08:20:00 AM     all      0.21      0.00      0.27      0.07      0.00     99.46
08:30:00 AM     all      0.11      0.00      0.24      0.04      0.00     99.62
08:40:00 AM     all      0.10      0.00      0.21      0.03      0.00     99.66
Average:        all      0.12      0.00      0.19      0.02      0.00     99.67
 

Not showing much.. The system is not very busy at the moment.   What I have figured out is that the load average goes up when transcoding is done.  Is there a way to get a thread count from the Emby process?

[Q-Droid 989]

Is that from the same timeframe?  The load average in top is a rolling 1, 5 and 15 minute value while sar is listing 10 minute intervals. There is no time period in your list with high usage which should be there if you had a load of 60 over 1, 5 and 15. In my opinion a very consistent value of 60.00 for all time slots is nearly impossible.

The load average from top is the number of processes in the run queue waiting for CPU. Meaning that on your host there should be 12 on CPU and another  60 (more like another 48) waiting and the usage would be reflected as 0% idle, a high %user and high %system because of the overhead of so many processes queued up for work.

Is your OS up to date? The kernel isn't very new.

I'm on openSUSE so the commands might be slightly different. To show the threads when emby is the runtime user:
top -H -u emby
ps -Lf -u emby -U emby
 

 

Edited October 11, 2024 by Q-Droid
Correcting the load average meaning.

[DarinM 11]

Sorry not from the same timeframe.. I've been trying to recreate the issue.. I've found that watching transcoded material causes it.  The OS is Rocky Linux, so Red Hat based and yes, it will use an older kernel.  Not bleeding edge like Ubuntu.

Here's current info:
top - 10:59:50 up 13 days, 14:05,  1 user,  load average: 21.87, 19.39, 14.30

Here's output from top -H -u emby:

    PID USER      PR  NI    VIRT    RES    SHR S  %CPU  %MEM     TIME+ COMMAND
2640958 emby      20   0 6726148 354412  92460 S   0.3   0.5   0:00.04 .NET ThreadPool
2531976 emby      20   0 6726148 354412  92460 S   0.0   0.5   0:01.08 EmbyServer
2531977 emby      20   0 6726148 354412  92460 S   0.0   0.5   0:00.00 EmbyServer
2531978 emby      20   0 6726148 354412  92460 S   0.0   0.5   0:00.00 EmbyServer
2531979 emby      20   0 6726148 354412  92460 S   0.0   0.5   0:00.00 EmbyServer
2531980 emby      20   0 6726148 354412  92460 S   0.0   0.5   0:00.00 EmbyServer
2531981 emby      20   0 6726148 354412  92460 S   0.0   0.5   0:00.05 .NET Finalizer
2531983 emby      20   0 6726148 354412  92460 S   0.0   0.5   0:00.32 .NET Long Runni
2531984 emby      20   0 6726148 354412  92460 S   0.0   0.5   0:00.00 EmbyServer
2531985 emby      20   0 6726148 354412  92460 S   0.0   0.5   0:00.00 .NET Network Ad
2531987 emby      20   0 6726148 354412  92460 S   0.0   0.5   0:00.34 .NET ThreadPool
2531989 emby      20   0 6726148 354412  92460 S   0.0   0.5   0:00.47 Kestrel Timer
2531990 emby      20   0 6726148 354412  92460 S   0.0   0.5   0:00.09 .NET Sockets
2531994 emby      20   0 6726148 354412  92460 S   0.0   0.5   0:01.08 .NET Timer
2532004 emby      20   0 6726148 354412  92460 S   0.0   0.5   0:00.00 .NET File Watch
2532005 emby      20   0 6726148 354412  92460 S   0.0   0.5   0:00.00 .NET File Watch
2532006 emby      20   0 6726148 354412  92460 S   0.0   0.5   0:00.00 .NET File Watch
2532007 emby      20   0 6726148 354412  92460 S   0.0   0.5   0:00.00 .NET File Watch
2532008 emby      20   0 6726148 354412  92460 S   0.0   0.5   0:00.00 .NET File Watch
2532009 emby      20   0 6726148 354412  92460 S   0.0   0.5   0:00.00 .NET File Watch
2532010 emby      20   0 6726148 354412  92460 S   0.0   0.5   0:00.00 .NET File Watch
2532011 emby      20   0 6726148 354412  92460 S   0.0   0.5   0:00.00 .NET File Watch
2532040 emby      20   0 6726148 354412  92460 S   0.0   0.5   0:00.00 .NET File Watch
2609923 emby      20   0 6726148 354412  92460 D   0.0   0.5   0:00.07 .NET ThreadPool
2610004 emby      20   0 6726148 354412  92460 D   0.0   0.5   0:00.00 .NET ThreadPool
2610005 emby      20   0 6726148 354412  92460 D   0.0   0.5   0:00.00 .NET ThreadPool
2610739 emby      20   0 6726148 354412  92460 D   0.0   0.5   0:00.00 .NET ThreadPool
2616825 emby      20   0 6726148 354412  92460 D   0.0   0.5   0:00.06 .NET ThreadPool
2616867 emby      20   0 6726148 354412  92460 D   0.0   0.5   0:00.00 .NET ThreadPool
2617611 emby      20   0 6726148 354412  92460 D   0.0   0.5   0:00.01 .NET ThreadPool
2618030 emby      20   0 6726148 354412  92460 D   0.0   0.5   0:00.03 .NET ThreadPool
2618415 emby      20   0 6726148 354412  92460 D   0.0   0.5   0:00.00 .NET ThreadPool
2618456 emby      20   0 6726148 354412  92460 D   0.0   0.5   0:00.00 .NET ThreadPool
2624595 emby      20   0 6726148 354412  92460 D   0.0   0.5   0:00.04 .NET ThreadPool
2624596 emby      20   0 6726148 354412  92460 D   0.0   0.5   0:00.02 .NET ThreadPool
2624597 emby      20   0 6726148 354412  92460 D   0.0   0.5   0:00.12 .NET ThreadPool
2626267 emby      20   0 6726148 354412  92460 D   0.0   0.5   0:00.00 .NET ThreadPool
2631752 emby      20   0 6726148 354412  92460 D   0.0   0.5   0:00.08 .NET ThreadPool
2632565 emby      20   0 6726148 354412  92460 D   0.0   0.5   0:00.02 .NET ThreadPool
2632576 emby      20   0 6726148 354412  92460 D   0.0   0.5   0:00.02 .NET ThreadPool
2633378 emby      20   0 6726148 354412  92460 D   0.0   0.5   0:00.00 .NET ThreadPool
2636445 emby      20   0 6726148 354412  92460 D   0.0   0.5   0:00.17 .NET ThreadPool
2637653 emby      20   0 6726148 354412  92460 D   0.0   0.5   0:00.03 .NET ThreadPool
2638428 emby      20   0 6726148 354412  92460 D   0.0   0.5   0:00.02 .NET ThreadPool
2638437 emby      20   0 6726148 354412  92460 D   0.0   0.5   0:00.00 .NET ThreadPool
2641704 emby      20   0 6726148 354412  92460 S   0.0   0.5   0:00.02 .NET ThreadPool
 

And output from ps -Lf -u emby -U emby:

# ps -Lf -u emby -U emby
UID          PID    PPID     LWP  C NLWP STIME TTY          TIME CMD
emby     2531976       1 2531976  0   48 08:41 ?        00:00:01 /opt/emby-server/system/EmbyServer -programdata /var/lib/emby -ffdetect /opt/emby-server/bin/ffdetect -ffmpeg /opt/emby-server/bin/ffmpeg -ffprobe
emby     2531976       1 2531977  0   48 08:41 ?        00:00:00 /opt/emby-server/system/EmbyServer -programdata /var/lib/emby -ffdetect /opt/emby-server/bin/ffdetect -ffmpeg /opt/emby-server/bin/ffmpeg -ffprobe
emby     2531976       1 2531978  0   48 08:41 ?        00:00:00 /opt/emby-server/system/EmbyServer -programdata /var/lib/emby -ffdetect /opt/emby-server/bin/ffdetect -ffmpeg /opt/emby-server/bin/ffmpeg -ffprobe
emby     2531976       1 2531979  0   48 08:41 ?        00:00:00 /opt/emby-server/system/EmbyServer -programdata /var/lib/emby -ffdetect /opt/emby-server/bin/ffdetect -ffmpeg /opt/emby-server/bin/ffmpeg -ffprobe
emby     2531976       1 2531980  0   48 08:41 ?        00:00:00 /opt/emby-server/system/EmbyServer -programdata /var/lib/emby -ffdetect /opt/emby-server/bin/ffdetect -ffmpeg /opt/emby-server/bin/ffmpeg -ffprobe
emby     2531976       1 2531981  0   48 08:41 ?        00:00:00 /opt/emby-server/system/EmbyServer -programdata /var/lib/emby -ffdetect /opt/emby-server/bin/ffdetect -ffmpeg /opt/emby-server/bin/ffmpeg -ffprobe
emby     2531976       1 2531983  0   48 08:41 ?        00:00:00 /opt/emby-server/system/EmbyServer -programdata /var/lib/emby -ffdetect /opt/emby-server/bin/ffdetect -ffmpeg /opt/emby-server/bin/ffmpeg -ffprobe
emby     2531976       1 2531984  0   48 08:41 ?        00:00:00 /opt/emby-server/system/EmbyServer -programdata /var/lib/emby -ffdetect /opt/emby-server/bin/ffdetect -ffmpeg /opt/emby-server/bin/ffmpeg -ffprobe
emby     2531976       1 2531985  0   48 08:41 ?        00:00:00 /opt/emby-server/system/EmbyServer -programdata /var/lib/emby -ffdetect /opt/emby-server/bin/ffdetect -ffmpeg /opt/emby-server/bin/ffmpeg -ffprobe
emby     2531976       1 2531987  0   48 08:41 ?        00:00:00 /opt/emby-server/system/EmbyServer -programdata /var/lib/emby -ffdetect /opt/emby-server/bin/ffdetect -ffmpeg /opt/emby-server/bin/ffmpeg -ffprobe
emby     2531976       1 2531989  0   48 08:41 ?        00:00:00 /opt/emby-server/system/EmbyServer -programdata /var/lib/emby -ffdetect /opt/emby-server/bin/ffdetect -ffmpeg /opt/emby-server/bin/ffmpeg -ffprobe
emby     2531976       1 2531990  0   48 08:41 ?        00:00:00 /opt/emby-server/system/EmbyServer -programdata /var/lib/emby -ffdetect /opt/emby-server/bin/ffdetect -ffmpeg /opt/emby-server/bin/ffmpeg -ffprobe
emby     2531976       1 2531994  0   48 08:41 ?        00:00:01 /opt/emby-server/system/EmbyServer -programdata /var/lib/emby -ffdetect /opt/emby-server/bin/ffdetect -ffmpeg /opt/emby-server/bin/ffmpeg -ffprobe
emby     2531976       1 2532004  0   48 08:41 ?        00:00:00 /opt/emby-server/system/EmbyServer -programdata /var/lib/emby -ffdetect /opt/emby-server/bin/ffdetect -ffmpeg /opt/emby-server/bin/ffmpeg -ffprobe
emby     2531976       1 2532005  0   48 08:41 ?        00:00:00 /opt/emby-server/system/EmbyServer -programdata /var/lib/emby -ffdetect /opt/emby-server/bin/ffdetect -ffmpeg /opt/emby-server/bin/ffmpeg -ffprobe
emby     2531976       1 2532006  0   48 08:41 ?        00:00:00 /opt/emby-server/system/EmbyServer -programdata /var/lib/emby -ffdetect /opt/emby-server/bin/ffdetect -ffmpeg /opt/emby-server/bin/ffmpeg -ffprobe
emby     2531976       1 2532007  0   48 08:41 ?        00:00:00 /opt/emby-server/system/EmbyServer -programdata /var/lib/emby -ffdetect /opt/emby-server/bin/ffdetect -ffmpeg /opt/emby-server/bin/ffmpeg -ffprobe
emby     2531976       1 2532008  0   48 08:41 ?        00:00:00 /opt/emby-server/system/EmbyServer -programdata /var/lib/emby -ffdetect /opt/emby-server/bin/ffdetect -ffmpeg /opt/emby-server/bin/ffmpeg -ffprobe
emby     2531976       1 2532009  0   48 08:41 ?        00:00:00 /opt/emby-server/system/EmbyServer -programdata /var/lib/emby -ffdetect /opt/emby-server/bin/ffdetect -ffmpeg /opt/emby-server/bin/ffmpeg -ffprobe
emby     2531976       1 2532010  0   48 08:41 ?        00:00:00 /opt/emby-server/system/EmbyServer -programdata /var/lib/emby -ffdetect /opt/emby-server/bin/ffdetect -ffmpeg /opt/emby-server/bin/ffmpeg -ffprobe
emby     2531976       1 2532011  0   48 08:41 ?        00:00:00 /opt/emby-server/system/EmbyServer -programdata /var/lib/emby -ffdetect /opt/emby-server/bin/ffdetect -ffmpeg /opt/emby-server/bin/ffmpeg -ffprobe
emby     2531976       1 2532040  0   48 08:41 ?        00:00:00 /opt/emby-server/system/EmbyServer -programdata /var/lib/emby -ffdetect /opt/emby-server/bin/ffdetect -ffmpeg /opt/emby-server/bin/ffmpeg -ffprobe
emby     2531976       1 2609923  0   48 10:21 ?        00:00:00 /opt/emby-server/system/EmbyServer -programdata /var/lib/emby -ffdetect /opt/emby-server/bin/ffdetect -ffmpeg /opt/emby-server/bin/ffmpeg -ffprobe
emby     2531976       1 2610004  0   48 10:21 ?        00:00:00 /opt/emby-server/system/EmbyServer -programdata /var/lib/emby -ffdetect /opt/emby-server/bin/ffdetect -ffmpeg /opt/emby-server/bin/ffmpeg -ffprobe
emby     2531976       1 2610005  0   48 10:21 ?        00:00:00 /opt/emby-server/system/EmbyServer -programdata /var/lib/emby -ffdetect /opt/emby-server/bin/ffdetect -ffmpeg /opt/emby-server/bin/ffmpeg -ffprobe
emby     2531976       1 2610739  0   48 10:22 ?        00:00:00 /opt/emby-server/system/EmbyServer -programdata /var/lib/emby -ffdetect /opt/emby-server/bin/ffdetect -ffmpeg /opt/emby-server/bin/ffmpeg -ffprobe
emby     2531976       1 2616825  0   48 10:29 ?        00:00:00 /opt/emby-server/system/EmbyServer -programdata /var/lib/emby -ffdetect /opt/emby-server/bin/ffdetect -ffmpeg /opt/emby-server/bin/ffmpeg -ffprobe
emby     2531976       1 2616867  0   48 10:29 ?        00:00:00 /opt/emby-server/system/EmbyServer -programdata /var/lib/emby -ffdetect /opt/emby-server/bin/ffdetect -ffmpeg /opt/emby-server/bin/ffmpeg -ffprobe
emby     2531976       1 2617611  0   48 10:30 ?        00:00:00 /opt/emby-server/system/EmbyServer -programdata /var/lib/emby -ffdetect /opt/emby-server/bin/ffdetect -ffmpeg /opt/emby-server/bin/ffmpeg -ffprobe
emby     2531976       1 2618030  0   48 10:31 ?        00:00:00 /opt/emby-server/system/EmbyServer -programdata /var/lib/emby -ffdetect /opt/emby-server/bin/ffdetect -ffmpeg /opt/emby-server/bin/ffmpeg -ffprobe
emby     2531976       1 2618415  0   48 10:31 ?        00:00:00 /opt/emby-server/system/EmbyServer -programdata /var/lib/emby -ffdetect /opt/emby-server/bin/ffdetect -ffmpeg /opt/emby-server/bin/ffmpeg -ffprobe
emby     2531976       1 2618456  0   48 10:31 ?        00:00:00 /opt/emby-server/system/EmbyServer -programdata /var/lib/emby -ffdetect /opt/emby-server/bin/ffdetect -ffmpeg /opt/emby-server/bin/ffmpeg -ffprobe
emby     2531976       1 2624595  0   48 10:39 ?        00:00:00 /opt/emby-server/system/EmbyServer -programdata /var/lib/emby -ffdetect /opt/emby-server/bin/ffdetect -ffmpeg /opt/emby-server/bin/ffmpeg -ffprobe
emby     2531976       1 2624596  0   48 10:39 ?        00:00:00 /opt/emby-server/system/EmbyServer -programdata /var/lib/emby -ffdetect /opt/emby-server/bin/ffdetect -ffmpeg /opt/emby-server/bin/ffmpeg -ffprobe
emby     2531976       1 2624597  0   48 10:39 ?        00:00:00 /opt/emby-server/system/EmbyServer -programdata /var/lib/emby -ffdetect /opt/emby-server/bin/ffdetect -ffmpeg /opt/emby-server/bin/ffmpeg -ffprobe
emby     2531976       1 2626267  0   48 10:41 ?        00:00:00 /opt/emby-server/system/EmbyServer -programdata /var/lib/emby -ffdetect /opt/emby-server/bin/ffdetect -ffmpeg /opt/emby-server/bin/ffmpeg -ffprobe
emby     2531976       1 2631752  0   48 10:48 ?        00:00:00 /opt/emby-server/system/EmbyServer -programdata /var/lib/emby -ffdetect /opt/emby-server/bin/ffdetect -ffmpeg /opt/emby-server/bin/ffmpeg -ffprobe
emby     2531976       1 2632565  0   48 10:49 ?        00:00:00 /opt/emby-server/system/EmbyServer -programdata /var/lib/emby -ffdetect /opt/emby-server/bin/ffdetect -ffmpeg /opt/emby-server/bin/ffmpeg -ffprobe
emby     2531976       1 2632576  0   48 10:49 ?        00:00:00 /opt/emby-server/system/EmbyServer -programdata /var/lib/emby -ffdetect /opt/emby-server/bin/ffdetect -ffmpeg /opt/emby-server/bin/ffmpeg -ffprobe
emby     2531976       1 2633378  0   48 10:50 ?        00:00:00 /opt/emby-server/system/EmbyServer -programdata /var/lib/emby -ffdetect /opt/emby-server/bin/ffdetect -ffmpeg /opt/emby-server/bin/ffmpeg -ffprobe
emby     2531976       1 2636445  0   48 10:54 ?        00:00:00 /opt/emby-server/system/EmbyServer -programdata /var/lib/emby -ffdetect /opt/emby-server/bin/ffdetect -ffmpeg /opt/emby-server/bin/ffmpeg -ffprobe
emby     2531976       1 2637653  0   48 10:55 ?        00:00:00 /opt/emby-server/system/EmbyServer -programdata /var/lib/emby -ffdetect /opt/emby-server/bin/ffdetect -ffmpeg /opt/emby-server/bin/ffmpeg -ffprobe
emby     2531976       1 2638428  0   48 10:56 ?        00:00:00 /opt/emby-server/system/EmbyServer -programdata /var/lib/emby -ffdetect /opt/emby-server/bin/ffdetect -ffmpeg /opt/emby-server/bin/ffmpeg -ffprobe
emby     2531976       1 2638437  0   48 10:56 ?        00:00:00 /opt/emby-server/system/EmbyServer -programdata /var/lib/emby -ffdetect /opt/emby-server/bin/ffdetect -ffmpeg /opt/emby-server/bin/ffmpeg -ffprobe
emby     2531976       1 2641704  0   48 11:00 ?        00:00:00 /opt/emby-server/system/EmbyServer -programdata /var/lib/emby -ffdetect /opt/emby-server/bin/ffdetect -ffmpeg /opt/emby-server/bin/ffmpeg -ffprobe
emby     2531976       1 2641753  0   48 11:01 ?        00:00:00 /opt/emby-server/system/EmbyServer -programdata /var/lib/emby -ffdetect /opt/emby-server/bin/ffdetect -ffmpeg /opt/emby-server/bin/ffmpeg -ffprobe
emby     2531976       1 2641755  0   48 11:01 ?        00:00:00 /opt/emby-server/system/EmbyServer -programdata /var/lib/emby -ffdetect /opt/emby-server/bin/ffdetect -ffmpeg /opt/emby-server/bin/ffmpeg -ffprobe
emby     2531976       1 2641756  0   48 11:01 ?        00:00:00 /opt/emby-server/system/EmbyServer -programdata /var/lib/emby -ffdetect /opt/emby-server/bin/ffdetect -ffmpeg /opt/emby-server/bin/ffmpeg -ffprobe
 

Restarting emby clears the high load average..

 

[Q-Droid 989]

Your best bet is to capture all of this while it's happening and also include the relevant emby server log, ffmpeg logs and hardware detection log. The info after restarting Emby showing idle procs and no CPU time on them isn't useful. Start with a clean slate and reboot the server first.