Jump to content

High Memory Usage


specsnow

Recommended Posts

specsnow

Hello. My Emby server is consuming over 35GB of RAM on a regular basis. After I reboot my server (or just the jail) the memory usage drops but will rise again within a few hours. The problem is that after a few days, my entire server will become unresponsive because of Emby consuming all the RAM  

  • Emby 4.4.2.0 (installed from ports)
  • Host: FreeNAS 11.3 U3.2
  • Jail: FreeBSD 11.3-RELEASE-p6
  • 64 GB RAM
  • Premium (not sure if this matters)

When Emby is running normally, I will have about 40 GB available RAM. I never have more than 1 or 2 clients connected to the server at one time. I do have Live TV and DVR configured but Live is not used very often. I’ve already reached out to IX for support and they agree that something is wrong with Emby/Mono because it’s using way too much RAM. 

Any help is appreciated!

Link to comment
Share on other sites

Hi there, can you please attach the emby server log from when this happened? Thanks.

Link to comment
Share on other sites

  • 3 weeks later...
MRobi
On 6/25/2020 at 8:08 PM, specsnow said:

When Emby is running normally, I will have about 40 GB available RAM.

40Gb of available ram on FreeNAS is very very very rare. In fact it's unheard of. If you have 40Gb of available ram there is something wrong with your server. The ZFS file system by design will use ALL the ram you give it simply for cache purposes. It's not designed to leave you any free ram. I've attached a screenshot I just took from my system with 96Gb of ram showing 5.6Gb free. This is completely normal.

A few questions:

1) How are you determining a memory usage breakdown per jail? I'm not aware of how to do this.

2) With 64Gb of ram (ECC Ram??) I'm assuming you're OK here, but how many Tb of storage do you have? Rule of thumb for FreeNAS is a minimum 1Gb ram per Tb storage. Do you have any ram dedicated to VM's? 

3) Have you noticed a patern when your server becomes unresponsive? Is it at a certain time of day, etc.. etc.. 

Honestly, I'd be looking at hardward issues before looking at apps running inside of jails. Usually the jail would crash before the whole server. You may have a bad boot disk, bad stick of ram, etc.. etc.. 

Capture.PNG

Link to comment
Share on other sites

specsnow

@MRobi Thanks for the reply. Here are the answers to your questions. 

1) Using the command "ps -eo pid,rss,args -J <jail number>". That command will output all the running processes in a jail and how much memory they are using. That's how I know how much memory Mono/Emby is using. 

2) I have 4x 8TB drives with 24TB-ish of usable storage. I have 8GB dedicated to 1 VM. 

3) No pattern that I've noticed. Sometimes it can be within 1 day, sometimes within 2-3 days. Since the issue began happening, it's never gone more than just a couple days without becoming unresponsive. I did test bringing down the jail for a few days and the issue never happened so I am 100% confident it is Mono/Emby causing it. 

Also, you are definitely correct that ZFS will use as much RAM as it can get for cache but that is built up over time. It doesn't consume that much RAM immediately after a fresh boot. When I said I had 40GB available, that was after a fresh boot after all jails, VMs, and services have fired up and were running for about an hour. 

Thanks for the help. 

Link to comment
Share on other sites

specsnow

@Luke

Bluray Folder Support 1.0.0.0
Cinema Intros 1.0.32.0
Dlna 1.0.38.0
Dvd Folder Support 10.0.0.
Email Notifications 3.1.6.0
Fanart.tv 1.0.8.0
IMVDb 1.0.8.0
MovieDb 1.3.5.0
MussicBrainz 1.0.13.0
Nfo metadata 1.0.30.0
OMDb 1.0.9.0
Open Subtitles 1.0.27.0
Port Mapper 1.0.4.0
Reports 1.0.38.0
Server Configuration Backup 1.4.5.0
Studio Images 1.0.3.0
TheAudioDb 1.0.10.0
TheTVDB 1.0.34.0
Webhooks 1.0.12.0

Link to comment
Share on other sites

As a test, can you try removing the dlna plugin, then restart the server and see if that helps. thanks.

Link to comment
Share on other sites

3 minutes ago, specsnow said:

@Luke I deleted the DLNA plugin, and a few others I never use, and am still having the issue. 

Can you please attach a new emby server log? Thanks.

Link to comment
Share on other sites

  • 1 month later...
level20peon

I think the mono version in 11.3 is leaking memory, try:

mono -V

If it says that you running 5.10.1.57 look for ways to upgrade (or downgrade, if your emby installation runs on such version).

Edited by level20peon
Link to comment
Share on other sites

12 minutes ago, level20peon said:

I think the mono version in 11.3 is leaking memory, try:


mono -V

If it says that you running 5.10.1.57 look for ways to upgrade (or downgrade, if your emby installation runs on such version).

Is there even any newer version available on freebsd? I've been checking freshports every so often but haven't seen anything.

Link to comment
Share on other sites

level20peon

I am currently building 5.20 thanks to this (still building, will get back once this finishes):

https://www.ixsystems.com/community/threads/how-to-manually-upgrade-mono-from-5-10-to-5-20-in-a-freenas-jail.78871/page-6

At any rate, this is probably a workaround that most "I just use the emby plugin"-users won't be able to follow. And I guess that just taking that prebuilt attachment from the linked post should be considered as an unknown binary and for regulatory reasons not be posted here anyways. Difficult to find a "good" solution for just anybody here - my advice, however, would be to just restart the mono instance / the jail emby runs in with a cronjob or similar on a regular basis, in order to flush the memory consumption.

Edited by level20peon
Link to comment
Share on other sites

9 hours ago, level20peon said:

Took a little longer than expected, but I finished that 5.20 build and included the kevent patch (https://bz-attachments.freebsd.org/attachment.cgi?id=176608), since this still seems to be an issue in 5.20. Anyways, it will take quite some time (1-2 weeks) until I know if this fixed the memory leak. I'll get back once I know more.

I'm running mono 6.8.0.105 for my Sonarr, Radarr and Lidarr jails. Haven't seen the need to do it yet for Emby. But the process is quite simple.

Download mono-6.8.0.105 from this link (password is "Mono")

https://sansonehowell-my.sharepoint.com/:f:/p/jsansone/EpfQbi-zytpLhYraiy74XuUBzWXb6l_XwkzUVoc8BzOf1g?e=l702tr

Move it to the /tmp folder in your jail. I use winscp to do it

Run this command: pkg install /tmp/mono-6.8.0.105.txz 

Then restart your jail. Done. Mono 6.8.0.105 in under 3mins

Link to comment
Share on other sites

  • 2 weeks later...
level20peon

After watching the memory usage of 5.20 for around two weeks now I can say with certainty that this version fixed the memory leak I was experiencing with 5.10.

  • Like 2
Link to comment
Share on other sites

  • 5 weeks later...
FreeBSD_Lover

Thanks guys. I think I am in the same boat; running Emby version 4.5.2.0, and mono 5.10 on FreeBSD 11.4-RELEASE.  The weird thing is, I have another serve (at someone else’s house I set up for them) and this issue doesn't seem to be happening there. What is different is on my server I have the integration with HD HomeRun.  No need to respond; I will monitor my useage, and will try to get mono updated soon.

Just some notes for anyone else reading:

My symptoms are: memory useage keeps growing, and ends up consuming swap, as well it seems to consume about 100% of one cpu core. This morning swap was over 5GB and when I restarted Emby, swap went down to 282 MB, and CPU usage went back to 0.01%.

 

Link to comment
Share on other sites

There is a review in phabricator to update mono to a newer version, at the moment I'm using the patch from the review.
Take care, there are still some rough edges to the update, like port options not working and other architectures than AMD64.

Link to comment
Share on other sites

  • 2 weeks later...
Baenwort
On 11/2/2020 at 6:05 AM, Duffyx said:

There is a review in phabricator to update mono to a newer version, at the moment I'm using the patch from the review.
Take care, there are still some rough edges to the update, like port options not working and other architectures than AMD64.

There is a newer guide if you want Mono 6.8 that seems straight forward: https://www.truenas.com/community/threads/how-to-manually-upgrade-mono-from-5-10-to-5-20-in-a-freenas-jail.78871/post-613962 

Link to comment
Share on other sites

  • 9 months later...

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now
×
×
  • Create New...