Jump to content
specsnow

High Memory Usage

Recommended Posts

specsnow

Hello. My Emby server is consuming over 35GB of RAM on a regular basis. After I reboot my server (or just the jail) the memory usage drops but will rise again within a few hours. The problem is that after a few days, my entire server will become unresponsive because of Emby consuming all the RAM  

  • Emby 4.4.2.0 (installed from ports)
  • Host: FreeNAS 11.3 U3.2
  • Jail: FreeBSD 11.3-RELEASE-p6
  • 64 GB RAM
  • Premium (not sure if this matters)

When Emby is running normally, I will have about 40 GB available RAM. I never have more than 1 or 2 clients connected to the server at one time. I do have Live TV and DVR configured but Live is not used very often. I’ve already reached out to IX for support and they agree that something is wrong with Emby/Mono because it’s using way too much RAM. 

Any help is appreciated!

Share this post


Link to post
Share on other sites
Luke

Hi there, can you please attach the emby server log from when this happened? Thanks.

Share this post


Link to post
Share on other sites
specsnow

Most recent log file attached. At the time I pulled this log file, Mono/Emby was consuming over 19GB of RAM. Thanks. 

embyserver.zip

Share this post


Link to post
Share on other sites
Luke

Thanks, we are looking into it.

Share this post


Link to post
Share on other sites
MRobi
On 6/25/2020 at 8:08 PM, specsnow said:

When Emby is running normally, I will have about 40 GB available RAM.

40Gb of available ram on FreeNAS is very very very rare. In fact it's unheard of. If you have 40Gb of available ram there is something wrong with your server. The ZFS file system by design will use ALL the ram you give it simply for cache purposes. It's not designed to leave you any free ram. I've attached a screenshot I just took from my system with 96Gb of ram showing 5.6Gb free. This is completely normal.

A few questions:

1) How are you determining a memory usage breakdown per jail? I'm not aware of how to do this.

2) With 64Gb of ram (ECC Ram??) I'm assuming you're OK here, but how many Tb of storage do you have? Rule of thumb for FreeNAS is a minimum 1Gb ram per Tb storage. Do you have any ram dedicated to VM's? 

3) Have you noticed a patern when your server becomes unresponsive? Is it at a certain time of day, etc.. etc.. 

Honestly, I'd be looking at hardward issues before looking at apps running inside of jails. Usually the jail would crash before the whole server. You may have a bad boot disk, bad stick of ram, etc.. etc.. 

Capture.PNG

Share this post


Link to post
Share on other sites
specsnow

@MRobi Thanks for the reply. Here are the answers to your questions. 

1) Using the command "ps -eo pid,rss,args -J <jail number>". That command will output all the running processes in a jail and how much memory they are using. That's how I know how much memory Mono/Emby is using. 

2) I have 4x 8TB drives with 24TB-ish of usable storage. I have 8GB dedicated to 1 VM. 

3) No pattern that I've noticed. Sometimes it can be within 1 day, sometimes within 2-3 days. Since the issue began happening, it's never gone more than just a couple days without becoming unresponsive. I did test bringing down the jail for a few days and the issue never happened so I am 100% confident it is Mono/Emby causing it. 

Also, you are definitely correct that ZFS will use as much RAM as it can get for cache but that is built up over time. It doesn't consume that much RAM immediately after a fresh boot. When I said I had 40GB available, that was after a fresh boot after all jails, VMs, and services have fired up and were running for about an hour. 

Thanks for the help. 

Share this post


Link to post
Share on other sites
Luke

What Emby server plugins have you installed?

Share this post


Link to post
Share on other sites
specsnow

@Luke

Bluray Folder Support 1.0.0.0
Cinema Intros 1.0.32.0
Dlna 1.0.38.0
Dvd Folder Support 10.0.0.
Email Notifications 3.1.6.0
Fanart.tv 1.0.8.0
IMVDb 1.0.8.0
MovieDb 1.3.5.0
MussicBrainz 1.0.13.0
Nfo metadata 1.0.30.0
OMDb 1.0.9.0
Open Subtitles 1.0.27.0
Port Mapper 1.0.4.0
Reports 1.0.38.0
Server Configuration Backup 1.4.5.0
Studio Images 1.0.3.0
TheAudioDb 1.0.10.0
TheTVDB 1.0.34.0
Webhooks 1.0.12.0

Share this post


Link to post
Share on other sites
Luke

As a test, can you try removing the dlna plugin, then restart the server and see if that helps. thanks.

Share this post


Link to post
Share on other sites
specsnow

@Luke I deleted the DLNA plugin, and a few others I never use, and am still having the issue. 

Share this post


Link to post
Share on other sites
Luke
3 minutes ago, specsnow said:

@Luke I deleted the DLNA plugin, and a few others I never use, and am still having the issue. 

Can you please attach a new emby server log? Thanks.

Share this post


Link to post
Share on other sites
level20peon

I think the mono version in 11.3 is leaking memory, try:

mono -V

If it says that you running 5.10.1.57 look for ways to upgrade (or downgrade, if your emby installation runs on such version).

Edited by level20peon

Share this post


Link to post
Share on other sites
Luke
12 minutes ago, level20peon said:

I think the mono version in 11.3 is leaking memory, try:


mono -V

If it says that you running 5.10.1.57 look for ways to upgrade (or downgrade, if your emby installation runs on such version).

Is there even any newer version available on freebsd? I've been checking freshports every so often but haven't seen anything.

Share this post


Link to post
Share on other sites
level20peon

I am currently building 5.20 thanks to this (still building, will get back once this finishes):

https://www.ixsystems.com/community/threads/how-to-manually-upgrade-mono-from-5-10-to-5-20-in-a-freenas-jail.78871/page-6

At any rate, this is probably a workaround that most "I just use the emby plugin"-users won't be able to follow. And I guess that just taking that prebuilt attachment from the linked post should be considered as an unknown binary and for regulatory reasons not be posted here anyways. Difficult to find a "good" solution for just anybody here - my advice, however, would be to just restart the mono instance / the jail emby runs in with a cronjob or similar on a regular basis, in order to flush the memory consumption.

Edited by level20peon

Share this post


Link to post
Share on other sites
level20peon

Took a little longer than expected, but I finished that 5.20 build and included the kevent patch (https://bz-attachments.freebsd.org/attachment.cgi?id=176608), since this still seems to be an issue in 5.20. Anyways, it will take quite some time (1-2 weeks) until I know if this fixed the memory leak. I'll get back once I know more.

Share this post


Link to post
Share on other sites
Luke

That's great, thanks!

Share this post


Link to post
Share on other sites
MRobi
9 hours ago, level20peon said:

Took a little longer than expected, but I finished that 5.20 build and included the kevent patch (https://bz-attachments.freebsd.org/attachment.cgi?id=176608), since this still seems to be an issue in 5.20. Anyways, it will take quite some time (1-2 weeks) until I know if this fixed the memory leak. I'll get back once I know more.

I'm running mono 6.8.0.105 for my Sonarr, Radarr and Lidarr jails. Haven't seen the need to do it yet for Emby. But the process is quite simple.

Download mono-6.8.0.105 from this link (password is "Mono")

https://sansonehowell-my.sharepoint.com/:f:/p/jsansone/EpfQbi-zytpLhYraiy74XuUBzWXb6l_XwkzUVoc8BzOf1g?e=l702tr

Move it to the /tmp folder in your jail. I use winscp to do it

Run this command: pkg install /tmp/mono-6.8.0.105.txz 

Then restart your jail. Done. Mono 6.8.0.105 in under 3mins

Share this post


Link to post
Share on other sites
level20peon

I try to avoid prebuilt binaries from an unknown source.

Share this post


Link to post
Share on other sites
level20peon

After watching the memory usage of 5.20 for around two weeks now I can say with certainty that this version fixed the memory leak I was experiencing with 5.10.

  • Like 1

Share this post


Link to post
Share on other sites
Luke

That's great news. Thanks for the feedback.

Share this post


Link to post
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now

×
×
  • Create New...