Nem 2 Posted March 15, 2021 Share Posted March 15, 2021 Server: Unraid 6.8.3, Emby docker 4.5.4.0 Client: Windows 10, Chrome player Whenever I watch a movie on my laptop, after about 1 hour the player freezes and nothing in Emby works. I check my server and it is sitting at 100% CPU usage on all cores. Ultimately I have to hard restart my server because everything inside unraid is unresponsive. After the restart, I play the movie and the same things happens again after an hour. Nothing else on my unraid server causes this behaviour, so likely specific to Emby I remember something similar has happened in the past when using my desktop, and when trying to play videos on my raspberry pi via Kodi as well What can I do to diagnose this problem? Any likely culprits I can turn on/off for testing? I realize that posting logs here might help (should I turn on debug logging for this?), but I have so many of them from around the same time period that I'm not sure which ones would be useful: Link to comment Share on other sites More sharing options...
Happy2Play 8332 Posted March 15, 2021 Share Posted March 15, 2021 Primarily need to see the Embyserver.txt log for when this is happening. How much free space do you have? Link to comment Share on other sites More sharing options...
Nem 2 Posted March 15, 2021 Author Share Posted March 15, 2021 @Happy2Play The share that the file is on has about 1.67TB free. the cache drive containing emby has 200GB free I think I've narrowed the problem down somewhat. It seems something is eating up RAM and not releasing it. I have 16GB RAM on the server and it usually sits at ~60% usage. When I play a movie in Emby and let it run, the RAM usage slowly creeps up over time. Once it hits about 100% RAM usage, the CPU spikes and stays at 100% Ive attached my log file embyserver-63751368770.txt Link to comment Share on other sites More sharing options...
Happy2Play 8332 Posted March 15, 2021 Share Posted March 15, 2021 If you disable OMBI, do you still have this issue? You can see issues with OMBI by searching the forum though. Link to comment Share on other sites More sharing options...
Nem 2 Posted March 17, 2021 Author Share Posted March 17, 2021 @Happy2PlayI just turned off Ombi, restarted everything, and tried playing the same movie. Same problem. RAM usage crept up from 60% -> 95% in about 90minutes and didn't go down until I closed the browser. So it doesn't appear to be Ombi related. I have 16GB RAM and the server sits at around 60% RAM usage normally, leaving about 6.4GB available. Just noticed the movie file I'm trying to play is quite large (5.4GB). This problem didn't come up when I tried to play a smaller file (3.2GB) Is the transcoded video stored in RAM while a movie is playing? Is there a cap on how much of the movie Emby is allowed to transcode at a time? Link to comment Share on other sites More sharing options...
Q-Droid 662 Posted March 17, 2021 Share Posted March 17, 2021 Unraid seems to think you have 8GB of RAM. When monitoring usage it's best to know the breakdown of Total RAM, Used, Cache and Free. I/O for large files like media and transcodes will cache in memory to improve performance. This is done by the OS, not Emby. But this cache memory is not locked in to process space, just borrowed until needed for something more important, but it makes graphs look like RAM is used up and Free is low. Actual available memory is Free+Cache. I can't say if Unraid is accounting for this in its monitor. Link to comment Share on other sites More sharing options...
Nem 2 Posted March 17, 2021 Author Share Posted March 17, 2021 Hmm maybe the 8GB in unraid is reporting the capacity of a single DIMM? Either way, I'm not sure how to get a breakdown like that in unraid Assuming this is not a memory problem, what else could be causing Emby/unraid to hit 100% CPU and freeze like that? Is there anything useful in the log I posted? Link to comment Share on other sites More sharing options...
Q-Droid 662 Posted March 17, 2021 Share Posted March 17, 2021 If the behavior is consistent then check to see which processes are pegging the CPU like that. Also get embyserver and ffmpeg logs from the same time when CPU is pegged. If unraid doesn't have a process monitor on its dashboard then a screen capture from top or htop could help. Link to comment Share on other sites More sharing options...
Nem 2 Posted March 19, 2021 Author Share Posted March 19, 2021 @Q-Droid ok I will give that a go tonight I made a mistake in my earlier post - I thought I had 16GB in the server but forgot that I removed some a while back. It currently only has 8GB RAM (60% of which is usually used up by other things). Could having a low RAM capacity and trying to transcode a 5GB+ video be a problem here? Link to comment Share on other sites More sharing options...
Luke 37180 Posted March 19, 2021 Share Posted March 19, 2021 On 3/17/2021 at 12:17 AM, Nem said: @Happy2PlayI just turned off Ombi, restarted everything, and tried playing the same movie. Same problem. RAM usage crept up from 60% -> 95% in about 90minutes and didn't go down until I closed the browser. So it doesn't appear to be Ombi related. I have 16GB RAM and the server sits at around 60% RAM usage normally, leaving about 6.4GB available. Just noticed the movie file I'm trying to play is quite large (5.4GB). This problem didn't come up when I tried to play a smaller file (3.2GB) Is the transcoded video stored in RAM while a movie is playing? Is there a cap on how much of the movie Emby is allowed to transcode at a time? Hi, can we please see the server log from that? Thanks. Link to comment Share on other sites More sharing options...
Q-Droid 662 Posted March 19, 2021 Share Posted March 19, 2021 Like everything, it depends on what else your server is doing. If your box is primarily Emby and general NAS duty then 8GB on Linux is usually fine. Unraid might add some overhead depending on filesystem type, cache disk/pool and mover activity so it's best to figure out what is going on with the server when the CPU gets maxed out. When transcoding the size of the file matters less than the bitrate and codecs. It's basically reading chunks, converting then writing them out to another file or files. It shouldn't use that much RAM for the process. Embyserver logs, ffmpeg logs, top and maybe syslog messages covering the times of the problems might show something. Link to comment Share on other sites More sharing options...
Nem 2 Posted March 19, 2021 Author Share Posted March 19, 2021 @Luke @Q-Droid Just reran the test (without Ombi). Have attached all available logs from Emby, as well as screenshot from htop (sorted by cpu and mem). Everything is from around the time of the crash/freeze. Couldn't grab a syslog as I didn't realize the file wouldn't persist after reboot From the image you can see: Both unraid and htop report max RAM usage Unraid reports max CPU, but htop doesnt. Maybe some display bug in unraid... There is no single process that is maxing out cpu or ram at the time of the crash, so I'm not sure where all this memory is going Highest memory consumption is from a Wordpress virtual machine, but that % value is always that high embyserver-63751702129.txt ffmpeg-remux-b309eb1e-13f2-480d-986b-808f5642a583_1.txt hardware_detection-63751702140.txt Link to comment Share on other sites More sharing options...
Q-Droid 662 Posted March 19, 2021 Share Posted March 19, 2021 Did the server freeze/crash again? Based on the logs it looks like it played for ~90 minutes but no signs of problems in Emby. I can barely read the htop screens so guessing at a few things. Let's ignore CPU for now. The fact that the memory bar is solid green is not a good sign. Pay attention to the Res column in htop, it shows the per process memory space. Emby mem usage looks to be low, ~200MB? And your Worpress VM is taking up more than 1/3 of the RAM. On top of that it looks like Unraid does not create swap space by default. So if you were to consume all memory instead of paging and swapping the OOM killer kicks in and starts killing processes to reclaim memory and things can get hairy from there. The syslog would show when OOM killer gets busy. The "free" command can show you how much much memory is used and available overall. Can you back down the RAM size for the Wordpress VM and test? You can also search the webs to see how to monitor the OOM killer on Unraid and look for signs of invocation. Check overall system memory with the free command, test with the VM down, test with smaller VM. If it turns out to be a memory issue the free options are to consider the swap file plugin for Unraid and/or run smaller VMs. The easiest but not free would be to re-add that 8GB stick. Link to comment Share on other sites More sharing options...
Luke 37180 Posted March 19, 2021 Share Posted March 19, 2021 Try removing these plugins to reduce the resource consumption of your emby server: 2021-03-18 20:15:18.124 Info App: Loading statistics, Version=2.0.19.0, Culture=neutral, PublicKeyToken=null from /config/plugins/Statistics.dll 2021-03-18 20:15:18.124 Info App: Loading MediaBrowser.Plugins.PushBulletNotifications, Version=3.1.4.0, Culture=neutral, PublicKeyToken=null from /config/plugins/MediaBrowser.Plugins.PushBulletNotifications.dll 2021-03-18 20:15:18.124 Info App: Loading playback_reporting, Version=1.7.0.3, Culture=neutral, PublicKeyToken=null from /config/plugins/playback_reporting.dll Then restart the server and see how that compares. thanks. Link to comment Share on other sites More sharing options...
Nem 2 Posted March 20, 2021 Author Share Posted March 20, 2021 @LukeSame crash after removing those plugins. Details below... Free before: total used free shared buff/cache available Mem: 8107524 3942400 124876 653016 4040248 3257536 Swap: 0 0 0 Free after: total used free shared buff/cache available Mem: 8107524 3900096 135420 3885712 4072008 93152 Swap: 0 0 0 Potentially more interesting test results in the next post... ffmpeg-remux-16f0cc24-dfe8-4c16-afb6-d56ef7c8a40f_1.txt syslog.txt embyserver-63751788617.txt Link to comment Share on other sites More sharing options...
Nem 2 Posted March 20, 2021 Author Share Posted March 20, 2021 @Q-DroidI did another test where I completely shut down the wordpress VM. So no Ombi, no VM, no emby plugins. Weird thing happened this time. RAM consumption never reached 100%, CPUs were never maxed out, but still about 90mins into the video, the same thing happened. Both Emby and Unraid became unresponsive, despite resources not being maxed. Details below... Free before: total used free shared buff/cache available Mem: 8107524 1522396 2162576 652424 4422552 5669372 Swap: 0 0 0 Free after: total used free shared buff/cache available Mem: 8107524 1753848 531956 3976580 5821720 2097788 Swap: 0 0 0 Kinda weird that a crash still happens despite having resources available. Could this still be caused by Emby, or more likely to be a systemic issue and hence start a post on the Unraid forums instead? embyserver-63751795200.txt ffmpeg-remux-d22a8b59-8163-4f08-b182-364d2569794a_1.txt syslog.txt Link to comment Share on other sites More sharing options...
Solution Q-Droid 662 Posted March 20, 2021 Solution Share Posted March 20, 2021 Maybe we have something now. It might not be a problem with Emby itself but it could very well be configuration that is triggering the freeze/crash. I'm not all that familiar with Unraid and how it deviates from typical Linux installations. I know there is usually memory-backed storage involved but not the details. You have a good number of docker containers running, including Emby. You should look into where this is mapped for the Emby container: /transcode/transcoding-temp/ Because of this: 23:56:42.238 Error writing trailer of /transcode/transcoding-temp/51EB01_%d.ts: No space left on device 23:56:42.238 frame=140726 fps= 25 q=-1.0 Lsize= 3204201kB time=01:37:49.40 bitrate=4488.0kbits/s throttle=100 speed=1.04x 23:56:42.238 video:3066637kB audio:137564kB subtitle:0kB other streams:0kB global headers:0kB muxing overhead: 0.000000% Last message repeated 1 times 23:56:42.238 EXIT 23:56:42.239 Conversion failed! 23:56:42.239 If /transcode/transcoding-temp/ is not mapped to a persistent storage pool and is instead using a system volume it could explain the full system freeze. If the volume is memory-backed it would also explain things going back to normal after reboot. The time mark is also consistent with the 90ish minutes for this movie. You should also look into the volume mapping of all containers that might generate significant output. 1 Link to comment Share on other sites More sharing options...
Happy2Play 8332 Posted March 20, 2021 Share Posted March 20, 2021 27 minutes ago, Q-Droid said: 23:56:42.238 Error writing trailer of /transcode/transcoding-temp/51EB01_%d.ts: No space left on device As this issues appeared to be consistent at a specific interval this is what I initially thought a free space issue, but wasn't seeing any issues in previous logs. Also not to familiar with platform. Emby does do some odd things when it runs out of space though. And a restart hides the issue as it purges the transcode folder. Link to comment Share on other sites More sharing options...
Nem 2 Posted March 20, 2021 Author Share Posted March 20, 2021 ah interesting. I'll have to look into that tonight. But how does that `/transcode` directory work? If a video is only transcoded a few chunks at a time, presumably the previous chunks are deleted from `/transcode` before the next chunks start, and space usage on that volume should never really grow in size? Link to comment Share on other sites More sharing options...
Happy2Play 8332 Posted March 20, 2021 Share Posted March 20, 2021 1 minute ago, Nem said: ah interesting. I'll have to look into that tonight. But how does that `/transcode` directory work? If a video is only transcoded a few chunks at a time, presumably the previous chunks are deleted from `/transcode` before the next chunks start, and space usage on that volume should never really grow in size? Nope, Emby maintains the entire session to preserve FF/RW, only upon a Stop command is the session deleted from folder. So transcode folder needs to be as large as the session playback. One reason some users have issue with transcoding Live TV the entire day eating hundreds of gigabits. Link to comment Share on other sites More sharing options...
Nem 2 Posted March 21, 2021 Author Share Posted March 21, 2021 Issue solved. I had /transcode mapped to my /tmp directory, which in Unraid is RAM. That explains the constantly rising RAM usage when watching videos. I set the transcoding path back to default (empty) and the movie plays fine all the way to the end now Thanks for helping me figure this out! Link to comment Share on other sites More sharing options...
Luke 37180 Posted March 21, 2021 Share Posted March 21, 2021 Thanks for the feedback. Link to comment Share on other sites More sharing options...
Recommended Posts
Create an account or sign in to comment
You need to be a member in order to leave a comment
Create an account
Sign up for a new account in our community. It's easy!
Register a new accountSign in
Already have an account? Sign in here.
Sign In Now