Emby server freezes/hangs a lot (often when users do an initial or manual sync in Kodi)

[jant90 15]

I've been having a lot of freezes on my Emby server (note: Emby itself freezes, not the system). I just updated to version 3.2.22.0 today and before I was running version 3.2.20.0, but also the versions before that one had these freezes. After a while of collecting complaints from my friends that are using the server I noticed that these freezes often occur when somebody does a initial or manual sync using the Emby for Kodi plugin. However, these freezes also seem to occur randomly (without anybody syncing) and I can not consistently reproduce a freeze by calling a sync action from within Kodi.

 

I did a clean install of this server (when I switched to new hardware and from Ubuntu to Manjaro) a few weeks ago and after filling the library with my content I tried syncing my first device (a LibreELEC device) and even then, without any other users connected I noticed these freezes when doing my initial sync. However a few server restarts and sync attempts later everything was synced to Kodi and I didn't pay much attention to it anymore. But recently Emby seems to freeze more and more often, I find myself restarting the Emby server service 3 to 5 times a day now. And that is with maybe 3 active users spread over the day.

 

What I think happens is that the HTTP server/process within Emby hangs/freezes, basically rendering user interaction with Emby impossible. When opening the webinterface during a freeze often the gray background is visible and sometimes the loader icon in the center of the screen too but other than that nothing on the page loads (while my webbrowser keeps trying to load it). It's pretty much the same when opening the Android app. There are a lot of HTTP response entries in the log with "(slow)" behind them, even when accessing the server myself (locally), that look a little like:

HTTP Response 200 to X.X.X.X. Time: 9006ms (slow) https://myembydomain.com/XXXXXXX

There are also a lot of Mono errors in there but I think ever since I've started using Emby I saw those kind of multi-line Mono "error reports" in there so they never bothered me, buy maybe they show what's wrong here?

 

As a test I just removed the Emby.Kodi Sync Queue plugin from the server but as soon as one user started Kodi, which triggered a sync of course, another user's playback was interrupted and the webinterface was again unreachable. 

 

Anyways, I have the log files of 3 of these freeze occurrences, I masked private info (IP addresses and hostname) in the one I'm sharing here, other than that the log is original and complete. The Sync Queue plugin has been removed during the freeze in this log. Events leading to the freeze:

1. At 17:05 I uninstalled the Sync Queue plugin and restarted the emby-server service.
2. A little later user d2fb16b9d2394a9096a530080b54bfac (IP A.A.A.A) starts playing some media using Emby for Kodi.
3. At around 17:22 user f0bd0f89fde648e0b9807052f0ed0727 (IP B.B.B. started Kodi that triggered a sync and froze Emby. The webinterface was not/hardly reachable and playback for the other user stopped.
4. At 17:38 the sync for the user hung at 100% for about 4 minutes, but from 17:38 onward the webinterface was responsive again. Playback also worked again. This was without restarting the emby-server service.
5. During the freeze user b3af460ff2284930ae24b4e5d5b797c1 (that's me, IP C.C.C.C, also the external IP of the server), tried to load the webinterface constantly.

I hope the cause of these freezes becomes clear from my log file or the information provided. If I need to supply any more info I would be more than happy to do so.

Thanks!

 

server-63634871115 (sync queue plugin uninstalled).txt

[Luke 37183]

What happens if a sync does not occur from Kodi? Like for example, what if you just use other apps such as the Emby web app?

[jant90 15]

Well the freezes seem to occur at random moments as well (when no one is syncing) so it's not exclusively triggered by a sync but most of the time a sync seems to be the cause.

 

When the server is not frozen playback using Kodi and the apps and webinterface works fine and the Emby server is responsive/snappy and generally just works as expected.

 

So basically, apart from these freezes it just works like it should.

[mastrmind11 717]

What are your server specs?  Sounds like an OOM issue to me.

[jant90 15]

OOM = out of memory? Well I monitored the server (using htop) during the freeze that I uploaded the log file for and during that whole time the system load and memory weren't high at all. I have 8GB of RAM in my system in total and memory usage is only around 2 to 3GB at any time really (4GB tops at peak moments). I also watched CPU usage of the Emby process during the freeze and it was only a few percent during that time (usually around 0% and shortly jumping to 10-20%). So, in conclusion the system as a whole has never been out of memory, and on top of that, during the Emby freeze, the server was perfectly capable of doing multiple full HD streams using other media server software, I explicitly tested this during the Emby freeze that I uploaded the log for. From that I conclude that the system as a whole was stable during that freeze.

 

By the way, my system consists of a ASRock J4205-ITX motherboard with onboard quad core Pentium J4205 processor, 8GB of RAM and 500GB SSD (Crucial MX300). I know that's not high-end hardware by any means but for exactly that reason I disabled transcoding for all users.

 

Just now I remember that when I was running Emby on my Ubuntu server I increased Mono's max memory allowance with some variable in the Emby service settings file or something to speed Emby up. [Yep, found it after some Googling, here.] Could it help if I set the Mono nursery-size to 512MB for example? And what would be the proper way to add it to the overwrite.conf for emby-server file? Could I just add this line?

Environment=MONO_THREADS_PER_CPU="500",MONO_GC_PARAMS="nursery-size=512m"

Or would I need to add MONO_ENV and change it to this:

Environment=MONO_ENV="MONO_THREADS_PER_CPU=500 MONO_GC_PARAMS=nursery-size=512m"

Thanks!

[Luke 37183]

It looks like it's just a matter of editing those files. But I also see you posting in the EmbyCon topic. It's possible the syncing process might just be a little hard on your machine. 

[puithove 208]

Remember that you need to make sure you're using the "mono-sgen" executable to launch.  This includes the newer garbage collection engine.  http://www.mono-project.com/docs/advanced/garbage-collector/sgen/working-with-sgen/

[jant90 15]

So I set the Mono environment variables and I immediately noticed that the Emby started quicker and the webinterface was way more responsive than before. Playback of movies also started a lot quicker than in the old situation. The whole feel of Emby now is more snappy.
 
Next, I added a new user and did an initial sync from Kodi. I saw Emby's memory usage quickly increase to over 3GB during the sync (in the old situation that was around 400MB max) and while the webinterface slowed down a little it was still very well usable. Playback of another user continued like it should. Also, the load on the server running Emby is quite a bit higher now during this sync. Now, I can not be 100% sure until I have this setup running for a few days at least but my preliminary conclusion is that it was an OOM issue that caused the freezes and general sluggishness of Emby, but it wasn't my system that was out of memory/resources, it was Mono that held Emby back from using more system resources.

To set the environment variables first I edited the /etc/conf.d/emby-server file (which is EnvironmentFile set in the systemd service unit file) but as changes to that file are not persistent across Emby updates I played around some more. I think the best way to do it is by running sudo systemctl edit emby-server (which creates a systemd override.conf file for you) and add the environment variables like this:

Environment=MONO_THREADS_PER_CPU=500
Environment=MONO_GC_PARAMS=nursery-size=512m

Note that it's also important to not quote the values or Mono won't use them (strangely the values could be quoted when setting them inside the /etc/conf.d/emby-server file).

 

 

That leaves me with one last question: would adding the --optimize=all flag to Mono make any difference? I'm asking because I noticed it in the other thread I linked to.

[Guest plexman]

I'm interested in this solution. In my case I don't have the "/etc/conf.d/emby-server" file cause the directory "/etc/conf.d" doesn't exist. If I change the veriables with "systemctl edit emby-server" is it going to work anyways? How can I check that the variables are applied when the server is executing?

 

OS: Ubuntu 16.04.2

Emby: 3.22.0-45.1

[mastrmind11 717]

I'm interested in this solution. In my case I don't have the "/etc/conf.d/emby-server" file cause the directory "/etc/conf.d" doesn't exist. If I change the veriables with "systemctl edit emby-server" is it going to work anyways? How can I check that the variables are applied when the server is executing?

 

OS: Ubuntu 16.04.2

Emby: 3.22.0-45.1

You don't have /etc/conf.d because 16.04 uses systemd and you're probably on a pure install.   Editing the unit will work fine.  I recommend checking this link, which will answer all your questions regarding systemd.  https://www.digitalocean.com/community/tutorials/how-to-use-systemctl-to-manage-systemd-services-and-units

[jant90 15]

If you're using systemd (you can check by running systemctl status emby-server) then you can use systemctl edit emby-server indeed. After editing restart the service with systemctl restart emby-server.

 

Btw I lowered the values to 250 and 256mb just now because last night Emby consumed over 5GB of RAM and that's just way too much on my 8GB RAM system. So I guess it's a matter of finetuning for your particular system.

[thynan 5]

I can confirm that jant90's approach speeds up emby alot. I am on ubuntu 16.04, and had regular problems with random slow loading (loading-circle would run for 1min and more before page was loaded). I also had frequent crashes of the emby server. I wrote a script which restarted the service automatically when it didn't respond within 1 minute to mitigate that problem, but the long loading times were very annoying. 

 

After setting the mono-parameters the way jant90 described, the web interface and app interface is a lot faster, and i didn't have any pageloads taking more than 2 or 3 seconds for the last couple days. The funny thing is: I already tuned mono about a year ago, and it helped a lot. But after an emby upgrade the settings were overwritten, and performance declined, and I didn't remember that the fix was quite easy.

Edited July 31, 2017 by thynan

[Luke 37183]

Thanks for the feedback.