RAM memory leak in the Embe process.

[Santrex 7]

Good afternoon.

We have a problem:
We have about 5 servers on Prod. platforms for viewing incoming video content from our partners.
The amount of RAM on our servers is 50 GB+
We occasionally have a "memory leak" issue in the Emby process. (EMBY as a Windows Service).
It can grow uncontrollably over time and fills all 100% of the RAM available to it. (only restart helps)
This does not happen all the time, but on each of the 5 servers that we have such a BUG has been "noticed".
We thought to connect this with the version of Emby Server (we have 2 servers with the old version), but as practice has shown, the problem is the same with the latest version and with the version 2 years ago.
2 servers - 4.7.0.3 | 3 server - 4.7.6.0
We have a large set of "Automations and Scripts 10+" collecting various "Statistics" via the API and this happens quite often (5-10 min.)
Servers automatically restart at 3am.
We have connected GPUs on servers for transcoding.
The problem was even on those servers where during the day there was not one user and video viewing.

What can you advise for us?
Have you met such a problem as "Memory leak"?

EmbyRAM1.bmp

Edited November 8, 2022 by Santrex

[Luke 42078]

Hi there, can you please attach the emby server log from when this happened? Thanks.

[Santrex 7]

Oh sure. Here is the log from that server on the day the problem occurred.
If there is a more recent log, I will throw it off the same way.

EmbyLogRAM.7z

[Santrex 7]

And what in general in EMBY can use so much RAM 30 GB+?
I didn't see a service capable of doing that.

[chef 3810]

1 hour ago, Santrex said:

And what in general in EMBY can use so much RAM 30 GB+?
I didn't see a service capable of doing that.

Sometimes Plugins can do that.

[Santrex 7]

Here is my list

[rbjtech 5284]

I have experienced this as well - using 64Gb on my server is not good as it eventually crashes Emby.

I've tried to figure out what is causing it - but the log is not showing anything (for me).

I've removed my Pictures library as I understand there may be an issue with a 3rd party jpg scanner causing the issue - since removing it (hopefully temporarily), the issue has no longer occurred and it's been stable for 2-3 days.  

I believe it happened during the library scan (mine was once every 24 hrs) - so the first thing I would do is disable that - and see if the issue persists.  Or manually run it and see if you can recreate the issue.

Edited November 9, 2022 by rbjtech

[Santrex 7]

It is important for us to check new files for new uploads.
Our scanner should check the entire library once every 5 minutes.
And we use storage and SMB-ball to access files.
Over 2000+ sub-folder paths and over 20000+ files in total.
There might be a glitch somewhere in the scanner, but I can't turn it off. Otherwise it's meaningless to me.
I have removed ScreenGrabber from the libraries, as it takes time.

[rbjtech 5284]

38 minutes ago, Santrex said:

It is important for us to check new files for new uploads.
Our scanner should check the entire library once every 5 minutes.
And we use storage and SMB-ball to access files.
Over 2000+ sub-folder paths and over 20000+ files in total.
There might be a glitch somewhere in the scanner, but I can't turn it off. Otherwise it's meaningless to me.
I have removed ScreenGrabber from the libraries, as it takes time.

So does real time monitoring not work for you ?

If you are doing a main scan every 5 minutes - then that is putting a large load on your system - assuming it competes it in the 5 minutes before it starts again.

Did you possibly mean the 'Metadata Folder' scan ? (which is different to a library scan).  if you are just 'adding' new files - then this is all you need to do - you do not need to run a full scan every 5 minutes ..

edit - I can see in the logs that it's doing a full scan every few minutes - I don't believe this is necessary.   It may not be the cause of the issue - but it is certainly not helping things..  

Edited November 9, 2022 by rbjtech

[Santrex 7]

And what are the general principles of updating EMBY regarding MediaFolder.
What mechanism does EMBY use to keep track of new occurrences of files in directories?
All of our paths are network shares.
I have real time enabled for each library, but honestly I'm not sure how it works. (especially with network paths)
Not all storage systems are Windows servers. Part is Linux with SMB access.

[Santrex 7]

I wrote above that the problem is "floating".
That is, everything can be fine for 5 days, for 6 it eats up 100% RAM. Maybe 2 days in a row be "Bug" and then a week everything is fine.
I can't relate it to any specific scenarios.

[Happy2Play 9780]

How many libraries do you have as it looks like 1200?

[Santrex 7]

33 - Library (Root-folders)
2183 - Path (Sub-Folders) - Total

Edited November 9, 2022 by Santrex

[Santrex 7]

Here it is again.

EmbyLogRAM2.7z

Edited November 10, 2022 by Santrex

[Luke 42078]

Hi, I would try turning off the realtime monitor for each of your emby libraries, then restart the server and see how that compares. Thanks.

[Santrex 7]

59 minutes ago, Luke said:

Hi, I would try turning off the realtime monitor for each of your emby libraries, then restart the server and see how that compares. Thanks.

Thank you. I'll post back after a few days of observation.

[Santrex 7]

1 hour ago, Luke said:

Hi, I would try turning off the realtime monitor for each of your emby libraries, then restart the server and see how that compares. Thanks.

And I wanted to clarify. This Sheduler after all does what I think?
It allows you to scan folders and find new video files?

[Luke 42078]

Yes correct.

[Santrex 7]

Unfortunately, the decision to remove "turning off the realtime monitor" did not work. We had a "Leak" again today at 10:15 (see graph)

EmbyLogRAM3.7z

[Santrex 7]

Here is the second time 1 day. I don't understand what causes this behavior...

[Santrex 7]

I've looked at the log for everything related to the "error" tag and very often I see an error with System.Private.CoreLib.dll

Quote

2022-11-14 14:24:46.557 Error Server: Error processing request
    *** Error Report ***
    Version: 4.7.6.0
    Command line: D:\Emby\system\EmbyServer.dll
    Operating system: Microsoft Windows 10.0.17763
    Framework: .NET 6.0.7
    OS/Process: x64/x64
    Runtime: D:/Emby/system/System.Private.CoreLib.dll
    Processor count: 24
    Data path: D:\Emby\programdata
    Application path: D:\Emby\system
    System.IO.FileNotFoundException: System.IO.FileNotFoundException: Unable to find the specified file.
       at Emby.Server.MediaEncoding.Api.Hls.BaseHlsService.CreateRequestFromPlaySessionId(BaseSegmentRequest segmentRequest)
       at Emby.Server.MediaEncoding.Api.Hls.DynamicHlsService.Get(GetHlsSegment segmentRequest)
       at Emby.Server.Implementations.Services.ServiceController.Execute(HttpListenerHost appHost, Object requestDto, IRequest req)
       at Emby.Server.Implementations.Services.ServiceHandler.ProcessRequestAsync(HttpListenerHost appHost, IRequest httpReq, IResponse httpRes, RestPath restPath, String responseContentType, CancellationToken cancellationToken)
       at Emby.Server.Implementations.HttpServer.HttpListenerHost.RequestHandler(IRequest httpReq, ReadOnlyMemory`1 urlString, ReadOnlyMemory`1 localPath, CancellationToken cancellationToken)
    Source: Emby.Server.MediaEncoding
    TargetSite: System.Tuple`3[Emby.Server.MediaEncoding.Api.StreamRequest,System.String,MediaBrowser.Controller.Net.AuthorizationInfo] CreateRequestFromPlaySessionId(Emby.Server.MediaEncoding.Api.Hls.BaseSegmentRequest)

 

[ebr 16177]

Hi.  If you remove the Playback Reporting plug-in, does it make a difference?

[Santrex 7]

8 minutes ago, ebr said:

Hi.  If you remove the Playback Reporting plug-in, does it make a difference?

The advice is good, but the Plugin is essential to our work. Its statistics and reports are a key element in our BI Analytics.
The company's management will not access the EMBY project without this module.

[Santrex 7]

12 minutes ago, ebr said:

Hi.  If you remove the Playback Reporting plug-in, does it make a difference?

Perhaps too frequent API requests from this Plugin cause such an effect. I'll try to collect them not so often...

[ebr 16177]

1 hour ago, Santrex said:

The advice is good, but the Plugin is essential to our work. Its statistics and reports are a key element in our BI Analytics.
The company's management will not access the EMBY project without this module.

Can you just try it as a test so we can eliminate it as a possible issue?