Emby Beta 4.10.0.15+: Crashing due to Out of Memory on DS918+ Package Center Install

[DarWun 364]

6 hours ago, DiabboVerdde said:

After collecting several days of logs and monitoring Emby's RSS memory usage with Splunk, I think I've narrowed the problem down considerably. Hopefully this helps identify the root cause. It seems my problem might be different from the one originally described in this post, but the consequence is the same.

Actually, there are some similarities to the issue I reported originally. On my NAS the scheduled task for "Scan media library" was set to run every six hours. Throughout the day it would execute automatically without an issue. Memory usage remained stable. But when I ran the task manually first thing in the morning after the NAS had sat idle overnight memory usage immediately jumped from 1GB to 2GB. The DS918+ only has 4GB of ram, and that sudden jump brings total memory utilized to around 90%. If I do anything else in Emby while that is happening, like navigating from the Dashboard to the Home screen, Emby will become unresponsive. Usually Emby shutdown at that point due to an OOM event. A couple of times recently it was able to recover and the memory was released, usually within the next 30minutes provided I didn't do anything else in Emby.

For me, this only started happening within the last month. It is occurring fairly frequently now (every couple of days). It occurs first thing in the morning after the NAS has sat idle overnight. You identified that "The process is usually sleeping after the scan completes while still holding all of the allocated memory." When the memory usage spike occurred again this morning, I checked the process status. Emby was sleeping.

One thing I noticed is that you are still on DSM 7.3.2. I started seeing the issue just after updating to DSM 7.4-90075. That was coincidentally around the same time that I updated to Emby Beta 4.10.0.15. So I couldn't rule out that it was the DSM upgrade that was the culprit. The fact that you are seeing this on an earlier version of DSM might be evidence that the newer version of DSM is not the culprit.

[DiabboVerdde 8]

42 minutes ago, DarWun said:

Actually, there are some similarities to the issue I reported originally. On my NAS the scheduled task for "Scan media library" was set to run every six hours. Throughout the day it would execute automatically without an issue. Memory usage remained stable. But when I ran the task manually first thing in the morning after the NAS had sat idle overnight memory usage immediately jumped from 1GB to 2GB. The DS918+ only has 4GB of ram, and that sudden jump brings total memory utilized to around 90%. If I do anything else in Emby while that is happening, like navigating from the Dashboard to the Home screen, Emby will become unresponsive. Usually Emby shutdown at that point due to an OOM event. A couple of times recently it was able to recover and the memory was released, usually within the next 30minutes provided I didn't do anything else in Emby.

For me, this only started happening within the last month. It is occurring fairly frequently now (every couple of days). It occurs first thing in the morning after the NAS has sat idle overnight. You identified that "The process is usually sleeping after the scan completes while still holding all of the allocated memory." When the memory usage spike occurred again this morning, I checked the process status. Emby was sleeping.

One thing I noticed is that you are still on DSM 7.3.2. I started seeing the issue just after updating to DSM 7.4-90075. That was coincidentally around the same time that I updated to Emby Beta 4.10.0.15. So I couldn't rule out that it was the DSM upgrade that was the culprit. The fact that you are seeing this on an earlier version of DSM might be evidence that the newer version of DSM is not the culprit.

You're right @DarWun, there are many similiarities. It's not something of DSM or even Emby version, because I'm using the latest "production" version of Emby while you are using the beta. I thought about installing an older version of Emby but I'm unsure this will solve the problem. It is definitely linked to the periodoc library scans/validation/refresh, which I have set for every 30 minutes on my server. Something related to this process is allocating a lot of memory and not giving it back. The scan of all my library takes about 2 minutes and then the process returns to "sleeping" state. When the problem is happening, during the scan the memory is allocated and never de-allocated so it will stay at that same level until the next execution and so on... until a few hours later it eats up all the memory in the box and the kernel kills emby.

it is happening again here as we speak. I am collecting as much data as I can but I don't think we mere mortals have access to what this library scan process does under the hood so we probably need someone from Emby to take a look. I have all logs created by Emby plus the full history of memory usage and allocation... everything.

Here's how it's looking right now:

"Emby" is how much memory the process is using right now, "Emby HighWaterMark" is the maximum amount of memory ever used by the process. I use it as a kill/restart detector. It only goes down if the process dies and a new process is opened for Emby. And "Total Used" is the total memory usage for the Synology box.

The library scans happen with no issue, allocating and de-allocating memory just fine for some time, then the mysterious thing happen and from that moment on it can't deallocate memory anymore. And it will keep eating up more and more memory until the OS kills it.

This is a challenging debug. But I feel the Emby logs alone don't have all the information to help me isolate what is the exact thing that's screwing the library scans. Anyway I have plenty of debug logs if anyone wants to take a crack at it.



 

[sa2000 791]

31 minutes ago, DiabboVerdde said:

It is definitely linked to the periodoc library scans/validation/refresh, which I have set for every 30 minutes

I know there is a problem with frequent scheduled scans and that is why, whilst this issue remains, the advice I have been given is to have very infrequent periodic scans. Normally a number of hours would be needed of not doing any scans before any memory gets purged

So suggestion is to set the frequency as once daily or not have it automatic and manually scan specific libraries

The first scan after the launch of emby server always uses up more memory than subsequent scans. The gap between scans needs to be high to allow memory to get released

It has been seen that sometimes memory does start climbing and climbing - but a period of no activity should result in memory being released - that may vary from one OS to another and the releasing of memory may only happen on some when there is need for it

 

[sa2000 791]

40 minutes ago, DiabboVerdde said:

The library scans happen with no issue, allocating and de-allocating memory just fine for some time, then the mysterious thing happen and from that moment on it can't deallocate memory anymore. And it will keep eating up more and more memory until the OS kills it.

You can send me logs and memory usage logging and I will have a look and see if there are clues as to why sometimes the scans lead to this

 

[DarWun 364]

41 minutes ago, DiabboVerdde said:

You're right @DarWun, there are many similiarities. It's not something of DSM or even Emby version, because I'm using the latest "production" version of Emby while you are using the beta. ...
 

Well, that part is is discouraging. I just rolled back to Beta 4.10.0.11 to see if the issue occurs as frequently as does it after updating to 4.10.0.15 and beyond. If you're seeing the issue on the Stable release, I suspect I'm not likely to see anything different.

It is very strange that this only started occurring for me mid-June. I don't recall this occurring prior to that. Or at least if it occurred I wasn't aware of it as Emby didn't crash. I checked Resource Monitor memory usage over the past six months. And although I can't see specifically what Emby's memory usage is, I can see that there aren't any sudden spikes in memory prior to mid-June. Having said that, if the spike occurred and memory was returned shortly afterwards it probably wouldn't show up.

[DarWun 364]

19 minutes ago, sa2000 said:

You can send me logs and memory usage logging and I will have a look and see if there are clues as to why sometimes the scans lead to this

@sa2000FYI, the memory spike occurred this morning again on the first scan of the morning. The previous scan had occurred six hours prior. What was different this time is it occurred while I was running scans of specific libraries instead of doing a manual scan using the scheduled task. And it was during scans of the movie/collections/playlists libraries.

I didn't have debug logging turned on in Emby, and I wasn't running the memory usage script at the time. So unfortunately I don't have any detailed logs or a csv of the memory usage to provide.

Anyway, I suspect you will get much better input from @DiabboVerddegiven the thoroughness of their past posts. But let me know if there is anything I else I can do to help troubleshoot this.

Edited 1 hour ago by DarWun

[DiabboVerdde 8]

@sa2000I have sent you my logs via DM, and I also sent you my script that's collecting the statistics.
 

I have been scanning every 30 minutes since I installed emby years ago

and never had any issues. That's mostly because the hardware is powerful and largely underutilized. I have 32G of ram and until this problem started emby had never used more than 1GB RAM even under heavy stress (serving simultaneous 4K videos to all four KODIs in the house while scanning at the same time). This points to something new, introduced by a plugin or one of the latest updates, as I keep my emby version up to date as much as I can.

Thanks for looking at my logs! I hope they're useful and lead to a solution of this issue to us all!

[DiabboVerdde 8]

1 hour ago, DarWun said:

Well, that part is is discouraging. I just rolled back to Beta 4.10.0.11 to see if the issue occurs as frequently as does it after updating to 4.10.0.15 and beyond. If you're seeing the issue on the Stable release, I suspect I'm not likely to see anything different.

It is very strange that this only started occurring for me mid-June. I don't recall this occurring prior to that. Or at least if it occurred I wasn't aware of it as Emby didn't crash. I checked Resource Monitor memory usage over the past six months. And although I can't see specifically what Emby's memory usage is, I can see that there aren't any sudden spikes in memory prior to mid-June. Having said that, if the spike occurred and memory was returned shortly afterwards it probably wouldn't show up.

That's more or less when I noticed it was happening to me too. I got to Kodi to watch something and it said the Emby server was offline... I checked Synology and it was. Had to start it manually a few times. Now I have to start it manually at least once a day, as sometimes when this crash happens Synology package manager doesn't bring Emby back up automatically. Sometimes it does. But i take this is a Synology thing and not an Emby thing.

I thought about setting a job to restart emby once a day, but that's pointless. I rather work towards a real solution, that's when I started collecting logs.

Now sa2000 has all my logs. Hopefully he (she?) can see something I'm not seeing and find out what's causing this.