specsnow 3 Posted June 25, 2020 Share Posted June 25, 2020 Hello. My Emby server is consuming over 35GB of RAM on a regular basis. After I reboot my server (or just the jail) the memory usage drops but will rise again within a few hours. The problem is that after a few days, my entire server will become unresponsive because of Emby consuming all the RAM Emby 4.4.2.0 (installed from ports) Host: FreeNAS 11.3 U3.2 Jail: FreeBSD 11.3-RELEASE-p6 64 GB RAM Premium (not sure if this matters) When Emby is running normally, I will have about 40 GB available RAM. I never have more than 1 or 2 clients connected to the server at one time. I do have Live TV and DVR configured but Live is not used very often. I’ve already reached out to IX for support and they agree that something is wrong with Emby/Mono because it’s using way too much RAM. Any help is appreciated! Link to comment Share on other sites More sharing options...
Luke 37064 Posted June 26, 2020 Share Posted June 26, 2020 Hi there, can you please attach the emby server log from when this happened? Thanks. Link to comment Share on other sites More sharing options...
specsnow 3 Posted July 15, 2020 Author Share Posted July 15, 2020 Most recent log file attached. At the time I pulled this log file, Mono/Emby was consuming over 19GB of RAM. Thanks. embyserver.zip Link to comment Share on other sites More sharing options...
Luke 37064 Posted July 20, 2020 Share Posted July 20, 2020 Thanks, we are looking into it. Link to comment Share on other sites More sharing options...
MRobi 159 Posted July 20, 2020 Share Posted July 20, 2020 On 6/25/2020 at 8:08 PM, specsnow said: When Emby is running normally, I will have about 40 GB available RAM. 40Gb of available ram on FreeNAS is very very very rare. In fact it's unheard of. If you have 40Gb of available ram there is something wrong with your server. The ZFS file system by design will use ALL the ram you give it simply for cache purposes. It's not designed to leave you any free ram. I've attached a screenshot I just took from my system with 96Gb of ram showing 5.6Gb free. This is completely normal. A few questions: 1) How are you determining a memory usage breakdown per jail? I'm not aware of how to do this. 2) With 64Gb of ram (ECC Ram??) I'm assuming you're OK here, but how many Tb of storage do you have? Rule of thumb for FreeNAS is a minimum 1Gb ram per Tb storage. Do you have any ram dedicated to VM's? 3) Have you noticed a patern when your server becomes unresponsive? Is it at a certain time of day, etc.. etc.. Honestly, I'd be looking at hardward issues before looking at apps running inside of jails. Usually the jail would crash before the whole server. You may have a bad boot disk, bad stick of ram, etc.. etc.. Link to comment Share on other sites More sharing options...
specsnow 3 Posted July 20, 2020 Author Share Posted July 20, 2020 @MRobi Thanks for the reply. Here are the answers to your questions. 1) Using the command "ps -eo pid,rss,args -J <jail number>". That command will output all the running processes in a jail and how much memory they are using. That's how I know how much memory Mono/Emby is using. 2) I have 4x 8TB drives with 24TB-ish of usable storage. I have 8GB dedicated to 1 VM. 3) No pattern that I've noticed. Sometimes it can be within 1 day, sometimes within 2-3 days. Since the issue began happening, it's never gone more than just a couple days without becoming unresponsive. I did test bringing down the jail for a few days and the issue never happened so I am 100% confident it is Mono/Emby causing it. Also, you are definitely correct that ZFS will use as much RAM as it can get for cache but that is built up over time. It doesn't consume that much RAM immediately after a fresh boot. When I said I had 40GB available, that was after a fresh boot after all jails, VMs, and services have fired up and were running for about an hour. Thanks for the help. Link to comment Share on other sites More sharing options...
Luke 37064 Posted July 21, 2020 Share Posted July 21, 2020 What Emby server plugins have you installed? Link to comment Share on other sites More sharing options...
specsnow 3 Posted July 21, 2020 Author Share Posted July 21, 2020 @Luke Bluray Folder Support 1.0.0.0 Cinema Intros 1.0.32.0 Dlna 1.0.38.0 Dvd Folder Support 10.0.0. Email Notifications 3.1.6.0 Fanart.tv 1.0.8.0 IMVDb 1.0.8.0 MovieDb 1.3.5.0 MussicBrainz 1.0.13.0 Nfo metadata 1.0.30.0 OMDb 1.0.9.0 Open Subtitles 1.0.27.0 Port Mapper 1.0.4.0 Reports 1.0.38.0 Server Configuration Backup 1.4.5.0 Studio Images 1.0.3.0 TheAudioDb 1.0.10.0 TheTVDB 1.0.34.0 Webhooks 1.0.12.0 Link to comment Share on other sites More sharing options...
Luke 37064 Posted July 26, 2020 Share Posted July 26, 2020 As a test, can you try removing the dlna plugin, then restart the server and see if that helps. thanks. Link to comment Share on other sites More sharing options...
specsnow 3 Posted July 31, 2020 Author Share Posted July 31, 2020 @Luke I deleted the DLNA plugin, and a few others I never use, and am still having the issue. Link to comment Share on other sites More sharing options...
Luke 37064 Posted July 31, 2020 Share Posted July 31, 2020 3 minutes ago, specsnow said: @Luke I deleted the DLNA plugin, and a few others I never use, and am still having the issue. Can you please attach a new emby server log? Thanks. Link to comment Share on other sites More sharing options...
level20peon 9 Posted September 18, 2020 Share Posted September 18, 2020 (edited) I think the mono version in 11.3 is leaking memory, try: mono -V If it says that you running 5.10.1.57 look for ways to upgrade (or downgrade, if your emby installation runs on such version). Edited September 18, 2020 by level20peon Link to comment Share on other sites More sharing options...
Luke 37064 Posted September 18, 2020 Share Posted September 18, 2020 12 minutes ago, level20peon said: I think the mono version in 11.3 is leaking memory, try: mono -V If it says that you running 5.10.1.57 look for ways to upgrade (or downgrade, if your emby installation runs on such version). Is there even any newer version available on freebsd? I've been checking freshports every so often but haven't seen anything. Link to comment Share on other sites More sharing options...
level20peon 9 Posted September 18, 2020 Share Posted September 18, 2020 (edited) I am currently building 5.20 thanks to this (still building, will get back once this finishes): https://www.ixsystems.com/community/threads/how-to-manually-upgrade-mono-from-5-10-to-5-20-in-a-freenas-jail.78871/page-6 At any rate, this is probably a workaround that most "I just use the emby plugin"-users won't be able to follow. And I guess that just taking that prebuilt attachment from the linked post should be considered as an unknown binary and for regulatory reasons not be posted here anyways. Difficult to find a "good" solution for just anybody here - my advice, however, would be to just restart the mono instance / the jail emby runs in with a cronjob or similar on a regular basis, in order to flush the memory consumption. Edited September 18, 2020 by level20peon Link to comment Share on other sites More sharing options...
level20peon 9 Posted September 19, 2020 Share Posted September 19, 2020 Took a little longer than expected, but I finished that 5.20 build and included the kevent patch (https://bz-attachments.freebsd.org/attachment.cgi?id=176608), since this still seems to be an issue in 5.20. Anyways, it will take quite some time (1-2 weeks) until I know if this fixed the memory leak. I'll get back once I know more. Link to comment Share on other sites More sharing options...
Luke 37064 Posted September 19, 2020 Share Posted September 19, 2020 That's great, thanks! Link to comment Share on other sites More sharing options...
MRobi 159 Posted September 19, 2020 Share Posted September 19, 2020 9 hours ago, level20peon said: Took a little longer than expected, but I finished that 5.20 build and included the kevent patch (https://bz-attachments.freebsd.org/attachment.cgi?id=176608), since this still seems to be an issue in 5.20. Anyways, it will take quite some time (1-2 weeks) until I know if this fixed the memory leak. I'll get back once I know more. I'm running mono 6.8.0.105 for my Sonarr, Radarr and Lidarr jails. Haven't seen the need to do it yet for Emby. But the process is quite simple. Download mono-6.8.0.105 from this link (password is "Mono") https://sansonehowell-my.sharepoint.com/:f:/p/jsansone/EpfQbi-zytpLhYraiy74XuUBzWXb6l_XwkzUVoc8BzOf1g?e=l702tr Move it to the /tmp folder in your jail. I use winscp to do it Run this command: pkg install /tmp/mono-6.8.0.105.txz Then restart your jail. Done. Mono 6.8.0.105 in under 3mins Link to comment Share on other sites More sharing options...
level20peon 9 Posted September 20, 2020 Share Posted September 20, 2020 I try to avoid prebuilt binaries from an unknown source. 1 Link to comment Share on other sites More sharing options...
level20peon 9 Posted October 2, 2020 Share Posted October 2, 2020 After watching the memory usage of 5.20 for around two weeks now I can say with certainty that this version fixed the memory leak I was experiencing with 5.10. 2 Link to comment Share on other sites More sharing options...
Luke 37064 Posted October 2, 2020 Share Posted October 2, 2020 That's great news. Thanks for the feedback. Link to comment Share on other sites More sharing options...
FreeBSD_Lover 13 Posted November 2, 2020 Share Posted November 2, 2020 Thanks guys. I think I am in the same boat; running Emby version 4.5.2.0, and mono 5.10 on FreeBSD 11.4-RELEASE. The weird thing is, I have another serve (at someone else’s house I set up for them) and this issue doesn't seem to be happening there. What is different is on my server I have the integration with HD HomeRun. No need to respond; I will monitor my useage, and will try to get mono updated soon. Just some notes for anyone else reading: My symptoms are: memory useage keeps growing, and ends up consuming swap, as well it seems to consume about 100% of one cpu core. This morning swap was over 5GB and when I restarted Emby, swap went down to 282 MB, and CPU usage went back to 0.01%. Link to comment Share on other sites More sharing options...
Duffyx 25 Posted November 2, 2020 Share Posted November 2, 2020 There is a review in phabricator to update mono to a newer version, at the moment I'm using the patch from the review. Take care, there are still some rough edges to the update, like port options not working and other architectures than AMD64. Link to comment Share on other sites More sharing options...
Baenwort 97 Posted November 14, 2020 Share Posted November 14, 2020 On 11/2/2020 at 6:05 AM, Duffyx said: There is a review in phabricator to update mono to a newer version, at the moment I'm using the patch from the review. Take care, there are still some rough edges to the update, like port options not working and other architectures than AMD64. There is a newer guide if you want Mono 6.8 that seems straight forward: https://www.truenas.com/community/threads/how-to-manually-upgrade-mono-from-5-10-to-5-20-in-a-freenas-jail.78871/post-613962 Link to comment Share on other sites More sharing options...
Luke 37064 Posted August 19, 2021 Share Posted August 19, 2021 Using a newer version of the mono runtime may potentially help: Link to comment Share on other sites More sharing options...
Recommended Posts
Create an account or sign in to comment
You need to be a member in order to leave a comment
Create an account
Sign up for a new account in our community. It's easy!
Register a new accountSign in
Already have an account? Sign in here.
Sign In Now