QNAP NAS Emby server consuming too much memory

[sezsat 1]

Hello, quite unfamiliar with Emby/QNAP here, I just want some insights and support on the current issue we are facing.

I just want to ask a for some insights about the QNAP Emby server that me and my colleagues are managing. We are currently having some issues with the server's memory consumption. The NAS' memory is already upgraded to 128 GB but the server manages to reach up to 70-90 GB of memory consumption leaving the NAS no choice but to terminate the server in order to free up some memory. 

We are already suspecting that the reason for this are the users frequently transcoding because they are usually watching through a web browser. We are planning to make them use the Emby app on whatever device they are using in order to achieve direct play and avoid transcoding. 

The problem is, there are times that the server will consume a lot of memory even though there is only 1 active playback (which is usually transcoding). Not sure if there is a delay of or the temporary transcoding files are not being deleted fast enough for the NAS to have enough memory. 

Is there any database optimization we can do in order to avoid this issue as well? I am willing to provide more information regarding the issue.

I just want to confirm whether our conclusion was right or is there any other reason why the server is consuming too much memory on our QNAP NAS. Thank you so much in advance for your help

[sezsat 1]

Example of once instance where the Server is consuming too much memory with only 1 active playback.

Note: Prior to this, we tested the server by playing movies on multiple desktop/mobile phones with transcoding. (About 8-10 devices)

Edited November 13, 2024 by sezsat

[sa2000 674]

Suggest you enable debug logging on the Emby Server and restart the server and then monitor memory usage and when it is excessive, download the server logs and attach here together with indication of the times and memory usage you detected

See this article with respect to the debug logging setting

https://emby.media/support/articles/Log-Files.html

 

[sezsat 1]

Screenshot from QNAP NAS Resource Monitor after doing a test
Test details: 
Number of simultaneous playback (all direct play): 10
Devices: 1 apple tv and 9 windows PC running Emby theater

[sezsat 1]

@sa2000

23 minutes ago, sa2000 said:

Suggest you enable debug logging on the Emby Server and restart the server and then monitor memory usage and when it is excessive, download the server logs and attach here together with indication of the times and memory usage you detected

See this article with respect to the debug logging setting

https://emby.media/support/articles/Log-Files.html

 

Will do that thank you.

[sa2000 674]

Please also take a memory usage snapshot immediately after launching Emby Server before doing any playbacks

[sezsat 1]

Snapshot of QNAP NAS memory usage right after Emby server restart @sa2000

Edited November 13, 2024 by sezsat

[sezsat 1]

I also downloaded the server logs when the server is consuming more than 80GB of memory from the NAS @sa2000

embyserverlogs.txt

[sa2000 674]

51 minutes ago, sa2000 said:

Suggest you enable debug logging on the Emby Server and restart the server and then monitor memory usage and when it is excessive, download the server logs and attach here together with indication of the times and memory usage you detected

The log you provided is an existing log. I wanted to see the problem afresh from the time the server is launched up to when it reaches excessive usage and for having the time when that was noticed

So we know it starts at reasonable level of 226Mb

What I am after is debug logging from the start of that launch up to the time when it reaches 80Gb and an indication as to when you noticed the climb

 

 

[sa2000 674]

And to add to my note - if the problem does not arise before midnight after a fresh launch, then please include all the log files as we do switch log files at midnight

[sa2000 674]

Unrelated:

You appear to have diskspace issue with only 56Mb space for transcodes

2024-11-13 00:54:50.550 Info EncodingManager: Deleting segment /share/ZFS530_DATA/.qpkg/EmbyServer/programdata/transcoding-temp/4A63B7/4A63B7_223.ts due to disk space. Available space: 58769408
2024-11-13 00:54:50.550 Info EncodingManager: Deleting segment /share/ZFS530_DATA/.qpkg/EmbyServer/programdata/transcoding-temp/4A63B7/4A63B7_224.ts due to disk space. Available space: 58769408
2024-11-13 00:54:50.550 Info EncodingManager: Deleting segment /share/ZFS530_DATA/.qpkg/EmbyServer/programdata/transcoding-temp/4A63B7/4A63B7_225.ts due to disk space. Available space: 58769408
2024-11-13 00:54:50.550 Info EncodingManager: Deleting segment /share/ZFS530_DATA/.qpkg/EmbyServer/programdata/transcoding-temp/4A63B7/4A63B7_226.ts due to disk space. Available space: 58769408
2024-11-13 00:54:51.164 Info EncodingManager: Deleting segment /share/ZFS530_DATA/.qpkg/EmbyServer/programdata/transcoding-temp/D8AEFD/D8AEFD_1368.ts due to disk space. Available space: 58769408
2024-11-13 00:54:51.164 Info EncodingManager: Deleting segment /share/ZFS530_DATA/.qpkg/EmbyServer/programdata/transcoding-temp/D8AEFD/D8AEFD_1369.ts due to disk space. Available space: 58769408
2024-11-13 00:54:51.164 Info EncodingManager: Deleting segment /share/ZFS530_DATA/.qpkg/EmbyServer/programdata/transcoding-temp/D8AEFD/D8AEFD_1370.ts due to disk space. Available space: 58769408

 

 

[sezsat 1]

 

Please see the log-files downloaded when it has reached 81% memory utilization. And probably we should check the diskspace issue for transcodes as well. 

Edited November 14, 2024 by CBers
Link removed.

[sa2000 674]

13 hours ago, sezsat said:

Please see the log-files downloaded when it has reached 81% memory utilization. And probably we should check the diskspace issue for transcodes as well. 

Thank you. I have downloaded the logs. 

So Emby Server was launched at 2024-11-13 15:51:31 and we know from earlier screenshots that the server process starts at about 226 Mb memory usage

You captured the logs at 23:49 and memory usage had climbed by then to 81% which from earlier info would be about 88 Gb

I can see the following tasks starting and completing. The last task which was still in progress was the "Scan Media Library" task

2024-11-13 16:23:56.326 Debug TaskManager: Executing Refresh Emby Connect Data
2024-11-13 16:23:56.329 Debug TaskManager: Refresh Emby Connect Data Completed after 0 minute(s) and 0 seconds

2024-11-13 22:23:55.551 Debug TaskManager: Executing Vacuum Database
2024-11-13 22:23:55.553 Debug TaskManager: Vacuum Database Completed after 0 minute(s) and 0 seconds

2024-11-13 22:23:56.326 Debug TaskManager: Executing Refresh Emby Connect Data
2024-11-13 22:23:56.327 Debug TaskManager: Refresh Emby Connect Data Completed after 0 minute(s) and 0 seconds

2024-11-13 22:23:56.801 Debug TaskManager: Executing Refresh Internet Channels
2024-11-13 22:23:56.805 Debug TaskManager: Refresh Internet Channels Completed after 0 minute(s) and 0 seconds

2024-11-13 22:23:58.943 Debug TaskManager: Executing Refresh Users
2024-11-13 22:23:58.969 Debug TaskManager: Refresh Users Completed after 0 minute(s) and 0 seconds

2024-11-13 22:23:59.170 Debug TaskManager: Executing Log file cleanup
2024-11-13 22:23:59.363 Debug TaskManager: Log file cleanup Completed after 0 minute(s) and 0 seconds

2024-11-13 22:24:02.099 Debug TaskManager: Executing Cache file cleanup
2024-11-13 22:24:03.283 Debug TaskManager: Cache file cleanup Completed after 0 minute(s) and 1 seconds

2024-11-13 22:25:19.525 Debug TaskManager: Executing Scan media library

2024-11-13 23:04:50.216 Debug TaskManager: Executing Refresh Guide
2024-11-13 23:04:50.895 Debug TaskManager: Refresh Guide Completed after 0 minute(s) and 0 seconds

2024-11-13 23:21:03.739 Debug TaskManager: Executing Download subtitles
2024-11-13 23:39:34.755 Debug TaskManager: Download subtitles Completed after 18 minute(s) and 31 seconds

It is possible that the memory usage climb was following the start of the library media scan. This is not going to be easy to pin down.

We can get closer to it by comparing logged information against memory usage snapshots. If you have access through ssh to the linux command shell, you could run the following script and then give me the log and output file from the script. The script can be started before restarting Emby Server (with of course debug logging already enabled). The script will log memory usage every 1 second and it will log the memory usage of the new emby server process after the relaunch - until the time when you notice the high usage and capture the logs and output from this script.

The script will be writing to a file in the current directory - so create a directory for this and cd to it in linux shell session and then run the script

I don;t know if it needs adapting for your QNAP environment. I am grateful to @eldoctorfor providing me with it last time I looked into high memory usage - an issue that was to do with frequent scan media library scheduled tasks 

(echo "PID,time,VmPeak,VmSize,VmLck,VmPin,VmHWM,VmRSS,RssAnon,RssFile,RssShmem,VmData,VmStk,VmExe,VmLib,VmPTE,VmSwap";  while true; do PID=$(pgrep EmbyServer); echo -n "$PID,"; date "+%F %T" | tr '\n' ','; grep -E "^Vm|^Rss" /proc/${PID}/status | cut -d':' -f2 | sed -e 's/ kB$//g' | tr -d '\t' | tr -d ' ' | tr '\n' ',' | sed -e 's/,$//'; echo ""; sleep 1; done;) | tee sezsat_memory_1.csv

You can change the filename in the script everytime you run it

So try out the script, and get it to run writing to an output file - then double check debug logging is still enabled on emby server and with the script still running, restart emby server. It will pick the new process once it is up and running and log its memory use.  You could after few minutes go to the dashboard and select Scheduled Tasks and manually trigger the running of Scan Media Library task. If that gets the memory to climb to the 80Gb level, capture the logs and get zip of the script output file and let me have them. You can upload here anonymized logs but if providing raw original log files, please send to me by private message

If a manually triggered Scan Media Library task does not bring the problem on, then wait until it happens and capture the script output file and logs  

Thank You

[sezsat 1]

Hi! we tried out the script however it looks like it does not recognized the pgrep command or we missed something? Linux version of QNAP is Linux 5.10.-qnap x86_64.

 

[sa2000 674]

There are some references about pgrep and QNAP - here mentioning need for installing "procps-ng" suite of command line tools. Is it something you can look into?

 

[Carlo 4560]

12 hours ago, sa2000 said:

Thank you. I have downloaded the logs. 

So Emby Server was launched at 2024-11-13 15:51:31 and we know from earlier screenshots that the server process starts at about 226 Mb memory usage

You captured the logs at 23:49 and memory usage had climbed by then to 81% which from earlier info would be about 88 Gb

I can see the following tasks starting and completing. The last task which was still in progress was the "Scan Media Library" task

This sounds a lot like a problem I had a couple versions back on Synology. I couldn't reproduce on Linux or Windows. I had 8GB memory in the NAS but it started using all the memory. I upgraded to 20GB and it would just take longer but do the same thing. I couldn't get any decent profile or debugging tools installed on Synology due to conflicting dependencies.

I traced the problem to occur during library scanning. It was exception related. It seemed that memory used during exceptions didn't free up locally (code wise) and would build up. Once the code performed the inner loops and worked itself back out the memory could free up. Something related to the exception didn't clean up properly and would hold the memory until it got back to a parent calling block which could then dispose it and garbage collection to free up the memory.

I noticed having debug logging turned on aggravated the memory increase. Having the server log open in a different tab would consume a lot of memory as well as the log grew.

I tried testing on other platforms by changing file permissions o Emby would throw an exception when checking files during scans. This caused memory to rise quickly on Synology.

BTW, problem disappeared, one day not it just stopped doing it. Whatever the cause, it only affected two other Synology users that I'm aware of. Had to be something unique to our systems or maybe combination of settings, maybe even legacy settings not exposed anymire.

 

[sa2000 674]

6 hours ago, Carlo said:

It seemed that memory used during exceptions didn't free up locally (code wise) and would build up.

Yes I had my suspicions on this - seen it once before in a user log where there were thousands of exceptions. But it is probably not on all exceptions. 

[Carlo 4560]

Agree

I was getting close to figuring it out and knew roughly where to look. I was just tool challenged.
Ater striking out trying to install a few debug tools I was familiar with; I started researching and reading a lot online about similar issues and tools people had some level of success using on Synology. I was going to try a couple things, but my issue resolved it itself.

I still have mixed feelings about it, as I no longer have an issue, but I was close to figuring it out and wanted to find the cause. What is so unique about it that very few have any issue that we know about. I really wanted to figure out if it was something in my specific configuration, something in Emby Server, something specific to the NET framework deployed on Synology, something specific to a new Synology DSM or driver update or some combination of multiple things that created the "perfect storm".

[sezsat 1]

Thank you guys for your support and willingness to help resolve this issue. 
Yesterday, we decided to update the firmware of the QNAP NAS to QuTS hero h5.2.1.2929 build 20241025. 

During the day test, it was actually doing okay 8 windows PC playing on Emby theater - Direct play and 1 apple tv direct play as well. 
The memory consumption on the NAS was not reaching 1GB at all.
During the night where most people would watch, it crashed again due to high memory consumption - NAS terminated the process to free up memory space.

We are opting to relocate the server from QNAP NAS to an HP Proliant DL160 server running on Rocky Linux. At this stage we would take this chance to ask for some advice regarding configurations, database cache size recommendation and all that. 

Our library consists of mostly tv series and movies. It's around 138TB combined. 
The library is stored on another QNAP NAS (not the same model where the current Emby server is hosted).
We plan to use the SMB protocol to access the library from the NAS.
Here are the specification of the server:

 

[sezsat 1]

19 hours ago, Carlo said:

Agree

I was getting close to figuring it out and knew roughly where to look. I was just tool challenged.
Ater striking out trying to install a few debug tools I was familiar with; I started researching and reading a lot online about similar issues and tools people had some level of success using on Synology. I was going to try a couple things, but my issue resolved it itself.

I still have mixed feelings about it, as I no longer have an issue, but I was close to figuring it out and wanted to find the cause. What is so unique about it that very few have any issue that we know about. I really wanted to figure out if it was something in my specific configuration, something in Emby Server, something specific to the NET framework deployed on Synology, something specific to a new Synology DSM or driver update or some combination of multiple things that created the "perfect storm".

Yes, I agree that the issue came from a mixture of configuration and that is unfortunate. 
Maybe the fact that the library is located on another NAS and it was mounted using HybridMount to the QNAP NAS that is hosting the Emby server or some other reason. IT is hard to say with certainty because there are a lot of factors involved, and it will take a lot of time and commitment to pinpoint the main issue/cause. Luckily for us we have an option to move the Emby server from the QNAP machine to a more, we can say robust and stable server.

Regardless, thank you for your efforts. 

[Carlo 4560]

12 minutes ago, sezsat said:

Maybe the fact that the library is located on another NAS and it was mounted using HybridMount to the QNAP NAS that is hosting the Emby server or some other reason. IT is hard to say with certainty because there are a lot of factors involved, and it will take a lot of time and commitment to pinpoint the main issue/cause. Luckily for us we have an option to move the Emby server from the QNAP machine to a more, we can say robust and stable server.

Regardless, thank you for your efforts. 

I too had a remote mount, well couple of them, and thought that was part of the issue as well. My nightly library scanning task would take 6+ hours to run so there was always a lot of activities and streaming taking place during nightly tasks. I never got to see what would happen with no other use except scheduled tasks running.

After you get up on the other box and off the QNAP, do you think we could do a couple test with no one using it for a couple days? It would be fine to stop or kill Emby server so it doesn't crash or cause issues for other services still running on the QNAP. With no one using Emby, it should be much easier to identify what was taking place when any increase in memory is detected.

 

[sezsat 1]

45 minutes ago, Carlo said:

I too had a remote mount, well couple of them, and thought that was part of the issue as well. My nightly library scanning task would take 6+ hours to run so there was always a lot of activities and streaming taking place during nightly tasks. I never got to see what would happen with no other use except scheduled tasks running.

After you get up on the other box and off the QNAP, do you think we could do a couple test with no one using it for a couple days? It would be fine to stop or kill Emby server so it doesn't crash or cause issues for other services still running on the QNAP. With no one using Emby, it should be much easier to identify what was taking place when any increase in memory is detected.

 

Yes we can do that, we just need to finalize everything, and we will let you know once we can do some tests.

As for the new server, do you have any recommendations on the configuration? I posted the specs of the server and the library details above. 

Edited November 16, 2024 by sezsat

[Carlo 4560]

Thought I sent this last night, but something must have happened when I submitted it. Clicked reply and my message was back. At least I didn't habe to retype it.

15 hours ago, sezsat said:

Thank you guys for your support and willingness to help resolve this issue. 
Yesterday, we decided to update the firmware of the QNAP NAS to QuTS hero h5.2.1.2929 build 20241025. 

During the day test, it was actually doing okay 8 windows PC playing on Emby theater - Direct play and 1 apple tv direct play as well. 
The memory consumption on the NAS was not reaching 1GB at all.
During the night where most people would watch, it crashed again due to high memory consumption - NAS terminated the process to free up memory space.

We are opting to relocate the server from QNAP NAS to an HP Proliant DL160 server running on Rocky Linux. At this stage we would take this chance to ask for some advice regarding configurations, database cache size recommendation and all that. 

Our library consists of mostly tv series and movies. It's around 138TB combined. 
The library is stored on another QNAP NAS (not the same model where the current Emby server is hosted).
We plan to use the SMB protocol to access the library from the NAS.
Here are the specification of the server:

 

You mentioned you have a Proliant DL 160 but then posted info for an OptiPlex 7040 which is quite different.

Which generation is the 160? (8,9 or 10th)
How much memory do you have in the DL 160?
Single or Dual XEONs?
Which CPUs?
Any GPU in the DL 160?

Which exact model QNAP will this replace?
Does your NAS have a GPU that Emby uses for transcoding?

Let's start with these questions and go from there on storage.
If you already know what type of storage, you have in mind let us know ie 8 drives in ZFS raid2

What type of storage were you thinking?
Do you know yet type of interface, drives, controllers?
Will you have any NVMe drives?

Depending on the generation, it will be similar to one my Dell Servers, likely R730 or R740XD, so I'll have a good idea of your setup.