Server becoming unresponsive. RAM is very bloated..

[Carlo 4483]

It's quite possible there are a 2 different memory issues. I know the issue I've been trying to chase down happens on bare metal or any type of container. I'm pretty sure I know the general cause but need to narrow it down a bit more specific so it can be found/fixed.

If you're sure it's a docker issue you may be able to solve this yourself by switching from Docker to Podman.  It's command line interface is identical to Docker’s, including the arguments. You can simply alias the docker command to podman without noticing the difference, making it easy for existing Docker users to transition to Podman:

# .bashrc
alias docker=podman

Podman does a couple things differently which actually make it more reliable I think. Unlike Docker, which uses the dockerd daemon to manage all the containers under its control, Podman is daemonless. Therefore, there’s no persistent connection to some long-living process, removing the single point of failure issue in Docker, where an abrupt crash in the daemon process can kill running containers or cause them to become orphaned. Podman containers are started as child processes of the Podman process, heavily using user namespaces and network namespaces.

Podman also differentiates from Docker by using rootless containers by default. Root access is not necessary for launching and operating a container, but it helps to mitigate potential vulnerabilities in the container runtime that can cause privilege escalation. I think that simplifies adjusting privileges and makes it a little easier to understand.

A really cool built in feature of Podman that is not yet present in Docker is the ability to create and run pods. A pod is a collection of one or more containers that utilize a shared pool of resources and work closely together to achieve a specific function. Pods are also the smallest execution unit in Kubernetes, making the transition to Kubernetes easier should the need arise.

Summing up, it should be fairly strait forward on most OS (NAS aside) to install Podman and even run it onside Docker but I'd suggest for testing to shut down docker, do the alias and bring your whole stack up running in Podman just as it was running under Docker. If the memory issue was truly Docker related we will know shortly.  With any luck the issue is fixed and you got a bit of an upgrade in the process.

 

[Neminem 698]

Thanks for you valued input

But i think I will give Unraid a month to see if they can come up with a fix.

Right now unraid uses docker, and I would not know how to use Podman in that environment.

I have a HP ProDesk 400 G3 mini laying around, where i can play around with Linux + Podman, if all else failes. 

I might call on you for advice when and if that time comes

[seanbuff 1015]

6 hours ago, jaycedk said:

But i think I will give Unraid a month to see if they can come up with a fix.

UnRaid 6.12.2 was just released. They have reverted the Docker version back to an older one to address memory usage issues, etc. See how you go with that.

[Neminem 698]

Updating right now

Will report back tomorrow, when i have tested.

Edited June 30, 2023 by jaycedk

[Neminem 698]

As promised here is my findings.

Unraids docker roll back did not do it.

But I have now notices that its only during media streaming, trascoded , streamed or Directplay i see the high mem usage.

That usually drops down when the stream is ended.

I have attached some log files, I case you want them, to look through.

 

embyserver-63823766400.txt embyserver-63823939199.txt embyserver (9).txt

Edited July 3, 2023 by jaycedk

[Luke 38842]

5 hours ago, jaycedk said:

As promised here is my findings.

Unraids docker roll back did not do it.

But I have now notices that its only during media streaming, trascoded , streamed or Directplay i see the high mem usage.

That usually drops down when the stream is ended.

I have attached some log files, I case you want them, to look through.

 

embyserver-63823766400.txt 2.94 MB · 0 downloads embyserver-63823939199.txt 3.61 MB · 0 downloads embyserver (9).txt 2.85 MB · 0 downloads

Hi, please try removing these plugins:

2023-07-02 03:03:06.574 Info App: Loading EpMetaRefresh, Version=1.0.0.1, Culture=neutral, PublicKeyToken=null from /config/plugins/EpMetaRefresh.dll
2023-07-02 03:03:06.574 Info App: Loading Lastfm, Version=1.0.27.0, Culture=neutral, PublicKeyToken=null from /config/plugins/Lastfm.dll
2023-07-02 03:03:06.574 Info App: Loading MediaBrowser.Plugins.SmtpNotifications, Version=3.1.6.0, Culture=neutral, PublicKeyToken=null from /config/plugins/MediaBrowser.Plugins.SmtpNotifications.dll
2023-07-02 03:03:06.574 Info App: Loading MediaBrowser.Plugins.TimeLord, Version=19.8.3.1105, Culture=neutral, PublicKeyToken=null from /config/plugins/MediaBrowser.Plugins.TimeLord.dll
2023-07-02 03:03:06.574 Info App: Loading PhillipsHue, Version=1.1.0.6, Culture=neutral, PublicKeyToken=null from /config/plugins/PhillipsHue.dll
2023-07-02 03:03:06.574 Info App: Loading Restart, Version=1.0.0.0, Culture=neutral, PublicKeyToken=null from /config/plugins/Restart.dll
2023-07-02 03:03:06.574 Info App: Loading statistics, Version=3.0.10.0, Culture=neutral, PublicKeyToken=null from /config/plugins/Statistics.dll
2023-07-02 03:03:06.574 Info App: Loading playback_reporting, Version=2.0.0.3, Culture=neutral, PublicKeyToken=null from /config/plugins/playback_reporting.dll

Then restarting the server and see how things compare.

[Neminem 698]

Ok will do and report back my findings.

 

[Neminem 698]

Hmm don't think its a plugin issue.

Removed the plugins restartet the emby docker.

Watched a movie, observed this.
 

embyserver (10).txt

[Neminem 698]

Ok I think I had an realization of what is going on.

Here is the memory usage from all my dockers.

Here we see the 2 ppl streaming.

 

During the same time span, I see this when looking at the server wide mem usage.

 

This leads me to believe that, when streaming, the file touched is being pulled into RAM Cache.

If we look at RAM Used server wide its siting nicely at around 19GB, but looking at all dockers usage it's well above 24GB at times.
And I see an increase in RAM cached.

So in short user starts a stream.

Video file loaded into RAM cached.

I think, I see combined RAM Used + RAM Cached, in my docker graf, after the kernel was updated with the Unraid update.

I think, I only saw RAM Used before this update.

Tbh I have not seen oom_kill at any time.

I was just confused by what I could see.
 

[Neminem 698]

Just for good measure.

Can we move my comments to a new topic, since I hijacked this one from op.

[andrewmcd7 5]

Can I please raise this issue again. It’s occurring on my server usually once a day. Last about 5-15 min. No scheduled tasks occurring. It logs in and you see the images for the libraries but everything else hangs. I have attached the log file. 

app2806145776409558552.zip

[andrewmcd7 5]

@Luke^^^

[Luke 38842]

On 10/8/2023 at 9:30 AM, andrewmcd7 said:

@Luke^^^

Hi, infuse is hitting the server pretty hard during it's syncing process. Try removing the Infuse plugin and also closing infuse apps on your devices and then see how things compare.

[andrewmcd7 5]

14 minutes ago, Luke said:

Hi, infuse is hitting the server pretty hard during its syncing process. Try removing the Infuse plugin and also closing infuse apps on your devices and then see how things compare.

Will removing the plugin but still having infuse clients still connect, will that make the problem worse or better?  I thought the plug in was supposed to lessen the DB hammering on emby.

[Luke 38842]

28 minutes ago, andrewmcd7 said:

Will removing the plugin but still having infuse clients still connect, will that make the problem worse or better?  I thought the plug in was supposed to lessen the DB hammering on emby.

I don't know enough about how InfuseSync works to answer that. I'm just commenting on what I see in the server log.

[andrewmcd7 5]

Unfortunately this didn’t resolve the issue. It might have made it less frequent but it’s still happening. Here are logs from a different server… libraries were loading but no media was able to be played. Time was 7:45am. I’m racking my brain here. Any insight would help @Luke

embyserver_3.txt

[Schleudertrauma 8]

There still are many issues related to your media in fact that emby can't find the media files in some cases. Have you set the permissions to your media files correctly? In some cases emby can't start it's native services and then crashes but i am not a dev, so i don't know why that happens. In most cases in my history with linux and emby were the permission related to emby.

In your case i would try to install emby natively from the debian package for ubuntu systems and test whether the issue persists or not. If not, then there could be a failure in your docker environment.

[andrewmcd7 5]

Yeah I realized after a mount dropped in that server… @schleudertrauma

[Luke 38842]

4 hours ago, andrewmcd7 said:

Yeah I realized after a mount dropped in that server… @schleudertrauma

Has this resolved your issue?

[andrewmcd7 5]

22 minutes ago, Luke said:

Has this resolved your issue?

No this was a second server. Thought it was the same issue. It wasn’t. 

[andrewmcd7 5]

Looks like the issue is still happening occasionally. Here is another server of mine that just had the same issue. Loads the thumbnails for libraries and that’s it. See attached logs. 

embyserver__24_.txt

[Luke 38842]

On 12/26/2023 at 4:31 PM, andrewmcd7 said:

Looks like the issue is still happening occasionally. Here is another server of mine that just had the same issue. Loads the thumbnails for libraries and that’s it. See attached logs. 

embyserver__24_.txt 1.25 MB · 1 download

OK we're looking into it. Thanks.

[andrewmcd7 5]

Hey Luke any progress on this? It’s a daily issue on many servers we manage. Hoping there’s something we can do

[Luke 38842]

1 minute ago, andrewmcd7 said:

Hey Luke any progress on this? It’s a daily issue on many servers we manage. Hoping there’s something we can do

The 4.8 release is coming soon and will be lighter on it's feet than 4.7. Thanks.