awkimball 17 Posted October 17, 2025 Posted October 17, 2025 Hi all, I have been chasing emby container hanging/crashing for the past few months and finally think I have narrowed down the specific root cause. Context: Dell R730 Proxmox 9 Host (48 CPU, 128GB RAM) Debian 13 LXC Container w/ Emby 4.9.1.80 (42 CPU cores, 32GB RAM) Periodically my emby container completely stops responding, becomes very sluggish and I am not able to login via SSH or the proxmox pct console tool. I originally thought this was an intel GPU issue as I switched from an NVIDIA GPU earlier this year and I anecdotally thought the issues begun around then. It was very hard to diagnose because I do not have access to the container when it is in this failed state. I finally broke down and setup Prometheus for monitoring and a Grafana dashboard, and this has finally indicated that I am really seeing a OOM RAM issue. For this specific issue repro I am streaming 2 4K movies simultaneously and both are Direct Play / Direct Stream Remuxing due the the file container not being compatible with macOS Safari. As you can see, as soon as the streams are started the container RAM completely fills up, and after a couple of minutes the container becomes completely unresponsive and the streams stop. At this point, I have to force stop the container and restart it to recover. I suspect this is because emby/ffmpeg is remuxing the video files into a container compatible with the streaming client, and as such is loading the entire (large) video files into memory. I have used emby for years, why is this only an issue within the past year? If this is simply a linux memory management issueI I would expect the kernel to handle this properly and dispose of cached memory when it gets too full. Is this an emby/ffmpeg issue? Swap is off on both my proxmox host and the emby lxc container as I have a lot of RAM, I would prefer to keep swap turned off. Here are some images from my grafana dashboard that shows the rapid memory fill up just a few minutes after the streams begin, and then the metrics cut off when the container becomes unresponsive. I will also upload the emby server and ffmpeg stream logs from this session, with enhanced logging turned on. It seems really odd that I can no longer serve 2x 4K streams simultaneously when they aren't even transcoding, just remuxing/direct playing Thanks! ffmpeg-remux-f64d60e1-684c-4e99-83d0-f5aece2eeb5c_1.txt embyserver-63896295956.txt ffmpeg-directstream-101e7ef9-94c8-4e6b-b8ec-0ada5e8a74d4_1.txt 1
awkimball 17 Posted October 17, 2025 Author Posted October 17, 2025 Obviously I can mitigate this by throwing more RAM at the emby container, but why is it loading the entire video file into RAM? I suspect if I give it more RAM it will just fill that up too and then crash again
Lessaj 467 Posted October 17, 2025 Posted October 17, 2025 Is your /tmp file system RAM backed or disk backed? This is where the temporary files will go when remuxing or transcoding. Do you monitor the space used for this path? Quote 2025-10-17 01:56:27.994 Info App: Transcoding temporary files path: /tmp/emby/transcoding-temp 1
awkimball 17 Posted October 17, 2025 Author Posted October 17, 2025 2 minutes ago, Lessaj said: Is your /tmp file system RAM backed or disk backed? This is where the temporary files will go when remuxing or transcoding. Do you monitor the space used for this path? Wow.. I think you nailed it. /tmp is indeed tmpfs which is RAM backed. This normally would be fine, but weirdly inside the container is says tmpfs is allocated 64GB even though the entire container only has 32GB. I will dive more into this, it seems like something is going wrong when proxmox determines how big the tmpfs should be
awkimball 17 Posted October 17, 2025 Author Posted October 17, 2025 OK! Problem pretty much solved. It looks like there's some issue with proxmox/cgroups/lxc which means that the tmpfs size is created with half of the host's RAM and completely ignores the actual container memory limit set in the proxmox lxc .conf. As a result, it tries to fill tmpfs to 64GB and when it hits the container limit (32GB) it OOMs and crashes. I am going to make a post on the proxmox forum and see if this is expected, or a known issue or something. In the meantime, I am going to override this with `lxc.mount.entry = tmpfs tmp tmpfs nodev,nosuid,size=16G 0 0` in my container config which should limit tmpfs to half of my containers RAM, rather than half of the host ram. 1
awkimball 17 Posted October 17, 2025 Author Posted October 17, 2025 @LessajThank you so much. I have been chasing this for months and your reply is exactly what I needed to put the last piece of the puzzle together. Currently streaming 4x 4K movies without issue! Only using 16GB of the containers overall 32GB RAM 1
Lessaj 467 Posted October 17, 2025 Posted October 17, 2025 The other thought I had was that it wasn't cleaning up the temp files, which seems like it would be the case if the reported size is larger than the actual size so it never tries. I don't remember what the log entry says exactly to see if it was trying to do it. Glad that resolved it. 1
awkimball 17 Posted October 17, 2025 Author Posted October 17, 2025 Also, for additional info I have had "Enable Throttling" disabled on transcoding which seems to be related to the tmpfs filling up super quickly. Even with my above fix, 2 streams would max out the new 16GB tmpfs limit. I have now enabled "Enable Throttling" and I can handle at LEAST 6x 4K remuxes so I am a happy camper. TLDR: You have to manually limit tmpfs on proxmox lxc containers, and also ENABLE THROTTLING 2
awkimball 17 Posted October 17, 2025 Author Posted October 17, 2025 (edited) Actually, I'm going to unmark this as solved. My new tmpfs memory limit is working, but even with throttling turned on it is rapidly filling up tmpfs when 1x remux stream is started since throttling only seems to kick in when multiple streams are going all at once. As a result, the 1st or 2nd streams completely fill up tmpfs, and this then blocks additional streams from starting, they just spin forever. It only worked before because I started all of the streams simultaneously so throttling kicked in almost immediately Anybody have any suggestions other than just increasing tmpfs significantly? Do I need to remove some cpu cores so throttling kicks in sooner/immediately? Edited October 17, 2025 by awkimball
Lessaj 467 Posted October 17, 2025 Posted October 17, 2025 (edited) Throttling is just limiting how hard it has to work into spurts, so instead of going full tilt until it's done it will have a ~2 minute buffer worth. It should still be providing opportunities for cleanup as long as the sizing is being reported correctly. I tried digging through my logs to try to find what it says when it's cleaning up segments but I couldn't find any, my temp path is quite large but I know I've seen the messages before even when it wasn't really necessary to clean up yet so I was hoping to find something to reference. I'd recommend putting your transcoding temp folder on a disk backed path. If you're concerned that it will "wear out" your drive, especially if it's an SSD, don't be - modern consumer grade hardware is quite resilient, unless it's super super cheap or something you'll replace that drive long before it wears out. As a point of reference my scratch array includes a Samsung 850 EVO which used to serve as an OS drive for my hypervisor so it saw A LOT of small writes from VMs - it has almost 8 years of power on time with over 150 TB written and far as I can tell from the SMART data is still quite healthy even though it's warranty only covers 5 years and 150 TBW. EDIT: I have 2 of the same drive with equivalent results, since I ran them as RAID 1 when they were the hypervisor drives, now they're part of a ZFS "raid 0" array. Edited October 17, 2025 by Lessaj 1
awkimball 17 Posted October 17, 2025 Author Posted October 17, 2025 (edited) I'm definitely not opposed to putting the transcoding folder on a disk, that being said I would expect that the transcoding cache garbage collection that emby does should work equally well on either tmpfs or a real disk. I could definitely give it more space if it's on a real disk though so presumably this issue won't be as noticeable. @Lukeor anyone from emby can you help me determine if something is going wrong that is preventing emby from making space in the /tmp/emby when adding another stream requires space? I can replicate this pretty easily, so I just need to know what to look for in the logs (if log entries for this cache operation exist at all) Edited October 17, 2025 by awkimball
TMCsw 247 Posted October 18, 2025 Posted October 18, 2025 As a side note: search: proxmox 9 oom This is why I’m still on 8.x (admittedly, I haven’t been following this for now).
Luke 42077 Posted October 18, 2025 Posted October 18, 2025 4 hours ago, awkimball said: I'm definitely not opposed to putting the transcoding folder on a disk, that being said I would expect that the transcoding cache garbage collection that emby does should work equally well on either tmpfs or a real disk. I could definitely give it more space if it's on a real disk though so presumably this issue won't be as noticeable. @Lukeor anyone from emby can you help me determine if something is going wrong that is preventing emby from making space in the /tmp/emby when adding another stream requires space? I can replicate this pretty easily, so I just need to know what to look for in the logs (if log entries for this cache operation exist at all) Throttling is stream specific and it doesn't matter how many concurrent streams are going. The important thing is that the transcoding position has to get very far ahead of the playback position in order for throttling to occur in the first place.
Recommended Posts
Create an account or sign in to comment
You need to be a member in order to leave a comment
Create an account
Sign up for a new account in our community. It's easy!
Register a new accountSign in
Already have an account? Sign in here.
Sign In Now