Emby server memory usage

[pir8radio 1312]

I'm seeing this too, ive posted a memory usage probe below. You can see it hummed along nicely till Aug 21st. I'm unsure what exactly, may have been a plugin update. I didnt pick up on the memory gains and it sent the VM unresponsive around the 25th.

Since then ive had to regularly restart it. Prior to that the only real restarts were during plugin or application updates.

It does run as a service and uses the NextPVR plugin, and I'm always transcoding recorded TV shows. 

Library size is over 7TB, so i guess you would call that a large library, and my logs dont go that far back unfortunately.

Having said all that, it still runs great.

 

 

 

 

 

Just in case this helps,  Emby went from V 3.2.26 to 3.2.27 around that time, Stable version that is.

[Luke 42077]

@@TheHwyman, you're suggesting you saw high memory usage related to live tv, but without playing or recording anything. If that is the case, if you could try the next beta server and compare that would be helpful. thanks.

 

It's up now, thanks.

[TheHwyman 5]

@@TheHwyman, you're suggesting you saw high memory usage related to live tv, but without playing or recording anything. If that is the case, if you could try the next beta server and compare that would be helpful. thanks.

 

Hi Luke,

 

Looks like I was wishful thinking in my last post from a day and a half ago... after removing all traces of LiveTV and Schedule Guide config, 12 hrs later my memory use was only sitting at ~260 MB.

 

I only just got a chance to peek at the server just now since the last day and a half... Emby service is currently up to 3 GB memory in use. Checked the perflog trace and it's still climbing ~ 1 - 1.2 GB of memory consumed each 24 hrs of operation. Not a steady increase either, most of the 24 hr period it is relatively steady at its memory use + or - 100/200 MB but then usually at one point during the 24 hr period for a rough duration of 1-2 hrs it climbs in consumption on a 45 degree angle in the chart and the bulk of the increased memory consumption takes place. So now it's about determining exactly what is taking place in Emby at that time that is doing that...

 

All scheduled tasks only occur between midnight and 6 am daily. No user streaming has taken place all this week (last 4 days), but (2) client machines are online and connected 7/24. The memory increase period each day isn't consistent, it could be in the morning, afternoon or evening on any given day. When I look in the Emby server log just now and find the time period when the perflog shows the increased memory use taking place... the SyncQueue plugin is doing stuff and some new media is being added/processed.

 

So this is about the only thing that it appears is going on (during this time window), media is appearing in the folders (Emby is set to realtime folder detection), Emby is importing/processing them and then I imagine SyncQueue is pusing out updates to the online clients.

 

Fyi... I gotta travel out of town this evening, won't be back for a few days... but will check the forum when I can... but won't be able to muck around with this stuff on my server for a few days...

 

P.S. - Thinking about this (possibility of issue being around introduction of new media and/or SyncQueue) and the different user experiences (some experiencing memory consumption, others not)... it would also be interesting to know how much various folks are ADDing to their media collection on a daily/weekly basis.

 

 

Note: I have a number of automated processes running "massaging" my media. In addition to my SickRage/SAB setup downloading daily shows... I also have an AMD Ryzen 1800 system dedicated to converting my existing DVD/Bluray media sets to x265 and as it finishes stuff it's moving things into the server media library folders. So, for me personally, the last month my system is probably adding in the neighborhood of ~150 GB of media (tv shows and movies) into the media folders each and every day (and thus Emby is processing this intake) and come to think of it... I got the Ryzen box and started this automated process back in June and it's been doing its processing 7/24 since then.

Edited September 8, 2017 by TheHwyman

[Happy2Play 9780]

Hi Luke,

 

Looks like I was wishful thinking in my last post from a day and a half ago... after removing all traces of LiveTV and Schedule Guide config, 12 hrs later my memory use was only sitting at ~260 MB.

 

I only just got a chance to peek at the server just now since the last day and a half... Emby service is currently up to 3 GB memory in use. Checked the perflog trace and it's still climbing ~ 1 - 1.2 GB of memory consumed each 24 hrs of operation. Not a steady increase either, most of the 24 hr period it is relatively steady at its memory use + or - 100/200 MB but then usually at one point during the 24 hr period for a rough duration of 1-2 hrs it climbs in consumption on a 45 degree angle in the chart and the bulk of the increased memory consumption takes place. So now it's about determining exactly what is taking place in Emby at that time that is doing that...

 

All scheduled tasks only occur between midnight and 6 am daily. No user streaming has taken place all this week (last 4 days), but (2) client machines are online and connected 7/24. The memory increase period each day isn't consistent, it could be in the morning, afternoon or evening on any given day. When I look in the Emby server log just now and find the time period when the perflog shows the increased memory use taking place... the SyncQueue plugin is doing stuff and some new media is being added/processed.

 

Note: My SickRage / SABNZBd services are always downloading stuff periodically throughout each day and adding the stuff to the media folder libraries.

 

So this is about the only thing that it appears is going on (during this time window), media is appearing in the folders (Emby is set to realtime folder detection), Emby is importing/processing them and then I imagine SyncQueue is pusing out updates to the online clients.

 

Fyi... I gotta travel out of town this evening, won't be back for a few days... after this evening it may be awhile before I can reply back to anything...

 

Since you have other programs accessing your Library I wonder if Real time Monitoring could be a issue?

 

I know if I have MCM make changes I can see Emby use more memory from real time monitoring since there is no actual way to see this process/task.  But I have never seen excessive usage seen in this topic.

Edited September 8, 2017 by Happy2Play

[TheHwyman 5]

Since you have other programs accessing your Library I wonder if Real time Monitoring could be a issue?

 

I know if I have MCM make changes I can see Emby use more memory from real time monitoring since there is no actual way to see this process/task.  But I have never seen excessive usage seen in this topic.

 

You replied before I finished re-editing my post as I got thinking along the same lines and then added more detail... like the other x265 conversion processes that I have going on adding more media to the folders.

[dcook 299]

I add new content to my Libraries every day, and I have not experienced the memory issues.  

 

I don't use real time monitoring.

 

I don't think adding content has anything to do with it.

Edited September 8, 2017 by dcook

[TheKamakaZi 15]

Guys, I think I have found a suspect: FFMPEG. Or rather the way the forking is handled for it when performing chapter extraction.

 

I spun up a new container instance of Emby, leaving it empty for 24 hours, but with DLNA and blasting enabled. No increase in memory past the initial ~200MB.

 

Next, I pushed in ~3TB of movies, all on mkv, h264 and a combo of AC3 and DTS. After completing indexing, and chapter extraction, usage rose to 1.9GB.

 

At this point, I restarted the container, and left it for another 24 hours. Memory settled around 600MB.

 

I then pushed in my old media, avi, xvid, divx, mp3, etc.

 

And while watching the logs, I noticed that some of the ffmpeg processes never terminated, which leads me to believe that it may be at the core of our issue.

 

This would also explain why some people are experiencing the problem, and some don't. It's related to the type of content we have.

 

@@Luke, is there some way of tracking the threads used by ffmpeg to see their start, end, and resources consumed?

 

Sent from my SM-G935F using Tapatalk

[dcook 299]

If it was ffmpeg that was consuming the memory it would be that process showing, not Emby

For example, if I start watching something I see this:

 

 

Once I stop watching the ffmpeg process is gone.

[Happy2Play 9780]

FFMPEG has done this from day one, not as often these days thought.

 

So is FFMPEG using the resource or is "MediaBrowser.ServerApplication.exe"?

 

@@dcook beat me to it.

Edited September 8, 2017 by Happy2Play

[TheKamakaZi 15]

If it was ffmpeg that was consuming the memory it would be that process showing, not Emby

For example, if I start watching something I see this:

 

 

Once I stop watching the ffmpeg process is gone.

Please run the chapter extraction task, and see if ffmpeg also spawns under its own PID.

 

Sent from my SM-G935F using Tapatalk

[dcook 299]

I just ran Chapter Extraction, the task took 14 seconds to complete.  

During that time Emby increased from 481MB to just over 800MB once the task completed it went back down to 459MB

I did not see any ffmpeg process running during the Chapter Extraction task.

[Happy2Play 9780]

Please run the chapter extraction task, and see if ffmpeg also spawns under its own PID.

 

Sent from my SM-G935F using Tapatalk

 

Yep ffmpeg is spawned during my chapter extraction task.

[TheKamakaZi 15]

So dcook's memory usage rose, but spawned no ffmpeg.

 

@@Happy2Play, did your Emby process' memory usage rise during?

 

Sent from my SM-G935F using Tapatalk

[Happy2Play 9780]

Of course that is a given.  Task is still running but memory has only went up about 150MB from when task started.

 

[TheKamakaZi 15]

Of course that is a given. Task is still running but memory has only went up about 150MB from when task started.

 

But if ffmpeg is spawned outside of the main process, why would the main process' usage rise? The db should already be open, and any extracted information would be pushed to disk.

 

Sent from my SM-G935F using Tapatalk

[Happy2Play 9780]

But if ffmpeg is spawned outside of the main process, why would the main process' usage rise? The db should already be open, and any extracted information would be pushed to disk.

 

Sent from my SM-G935F using Tapatalk

 

Because the main process has to process the chapter images generated.

[TheKamakaZi 15]

Because the main process has to process the chapter images generated.

Are chapter images stored in the cache directory, or in db?

 

Does anyone know what is used to process the images, eg. ImageMagick?

 

Sent from my SM-G935F using Tapatalk

[Happy2Play 9780]

I know for sure they reside in cache\library\movieid folder once created, not sure on DB.

[dcook 299]

I think your on the wrong track here, if it was the chapter image extraction task causing the memory leak for some users then it should be effecting all users, not just some.

 

There has to be something in common with these small group of users who complain of this memory leak, such as:

 

Common Hardware

Emby plugins

LiveTV or PVR functions

Specific combination of Emby features enabled or disabled

OS version and patch level

File system type, and/or allocation unit size

etc, etc

[TheKamakaZi 15]

I think your on the wrong track here, if it was the chapter image extraction task causing the memory leak for some users then it should be effecting all users, not just some.

 

There has to be something in common with these small group of users who complain of this memory leak, such as:

 

Common Hardware

Emby plugins

LiveTV or PVR functions

Specific combination of Emby features enabled or disabled

OS version and patch level

File system type, and/or allocation unit size

etc, etc

My theory is reliant on the fact that we do not all have the same media.

 

I run Emby under a Linux container, and now have 2 containers at varying levels of patching, and different host OS, displaying the same symptom. The only commonality is my media library.

 

Sent from my SM-G935F using Tapatalk

[parrish 22]

Since I'm still following this thread (and still just as frustrated with the issue as when I first brought it up) here are my exact specs of the server this is running on.

 

Server Hardware:

SuperMicro H8DGI AMD Server Motherboard

AMD Opteron 6272 16 Core CPU

Matrox G200eW graphics (driver version 4.1.0.10)

64GB DDR3 ECC RAM

LSI 9650SE-8LPML SAS RAID controller

2 - WD3000HLFS 10krpm Velociraptor 280GB drives in RAID1 array for OS

2 - WD1004FBYZ 7200rpm Gold Enterprise 1TB drives in RAID1 for Data drive

Renesas USB3.0 PCI Express controller

Seagate USB3 2TB USB drive (this is where all my Emby media is stored)

Intel 82576 Gigabit NIC

 

Server Software:

Server 2016 Datacenter (fully current on all patches/updates) Acting as AD controller, file server, and DNS server

Avast CloudCare antivirus (formerly AVG CloudCare) - server modules only install

Hyper-V (with one Server 2016 Standard VM running MeshCentral- Emby is running on the host OS)

MS SQL Server 2012 (no current databases and service is not running)

Mdaemon v17 Email server

IIS10 hosting a static web site as well as a WebDAV site

Macrium Reflect 7 Server Edition

Serv-U FTP server version 15.1.3.3

iBackup version 11.1.0.0

Emby 3.2.30.0 (no plugins- a pure default installation and running as a service on host OS- not in a VM)

 

My media consists of about 28GB of MP3's, two AVI files, 133GB of MKV, one MOV, 195GB of MP4, and one WMV.  It's a very small collection with nothing unusual or crazy going on. 

 

Since I've set a task to restart the Emby service every morning I haven't noticed any memory usage issues.  Like right now, at 5:50pm (been running since about 3am) it's using 57mb of RAM.  But I haven't added any new media nor used it to watch any media.  It's just sitting there doing nothing at all. 

 

The media environment consists of a Dish Hopper3, two Joey 2's (all of which use DLNA), an Xbox One with the Emby app, and a Galaxy S6 phone with the Emby Android app.  If I log in to the Emby console it will be from a Windows 10 Pro PC using either Edge or IE11 as the browser. 

[Luke 42077]

Guys, I think I have found a suspect: FFMPEG. Or rather the way the forking is handled for it when performing chapter extraction.

 

I spun up a new container instance of Emby, leaving it empty for 24 hours, but with DLNA and blasting enabled. No increase in memory past the initial ~200MB.

 

Next, I pushed in ~3TB of movies, all on mkv, h264 and a combo of AC3 and DTS. After completing indexing, and chapter extraction, usage rose to 1.9GB.

 

At this point, I restarted the container, and left it for another 24 hours. Memory settled around 600MB.

 

I then pushed in my old media, avi, xvid, divx, mp3, etc.

 

And while watching the logs, I noticed that some of the ffmpeg processes never terminated, which leads me to believe that it may be at the core of our issue.

 

This would also explain why some people are experiencing the problem, and some don't. It's related to the type of content we have.

 

@@Luke, is there some way of tracking the threads used by ffmpeg to see their start, end, and resources consumed?

 

Sent from my SM-G935F using Tapatalk

 

The database query used by the chapter image scheduled task is a little but unoptimized. This will cause sqlite to have to perform a lot of unneeded work in order to resolve the query, and thus could increase memory consumption.

 

The improvement for this will be in tomorrow's beta.  Thanks.

[Luke 42077]

The subtitle download scheduled task also has a similar situation, and that is being corrected as well for tomorrow's beta. So now we will be up to three changes in the beta channel related to this. Thanks.

[jnheinz 17]

The subtitle download scheduled task also has a similar situation, and that is being corrected as well for tomorrow's beta. So now we will be up to three changes in the beta channel related to this. Thanks.

 

@@Luke This is not a bad theory.  Now that everyone brings it up, it does feel like when I run subtitle download & chapter image extraction tasks -- it causes it to spike & hold onto it more violently.  The subtitle download one was the worst of the two for me.  Ever since I disabled most scheduled tasks & disabled real-time monitoring, it has really slowed it down.  I did migrate my installation from Emby Server 3.2.29 for Windows to Emby Server 3.2.30 for Linux (Ubuntu) to validate or invalidate a suspicion of the OS or .NET Framework.  It took me a week to hit 2.2GB RAM.  This would previously only take about 4 days to reach that or 3GB+.  That was with scheduled tasks only running once in that week.

 

Maybe we should be disabling all scheduled tasks as a test?  I would be willing to test this beta.  Please let us know when it is released.

[dcook 299]

I don't understand how if it was those scheduled tasks why it would choose to only effect a few people.  Shouldn't it effect everyone if it was those tasks not releasing memory?