Emby server memory usage

[Luke 42077]

Everything always depends on usage patterns.

[dcook 299]

@@Luke Doesn't really make sense, there were plenty of users who reported no issues with memory who have much higher usage then some of those who reported the issue.  In fact one user who reported the issue had no usage at all, his server was not doing anything.

[ebr 16169]

@@Luke Doesn't really make sense, there were plenty of users who reported no issues with memory who have much higher usage then some of those who reported the issue.  In fact one user who reported the issue had no usage at all, his server was not doing anything.

 

Not everyone uses subtitles and chapter images are optional as well.

[dcook 299]

@@ebr  I understand that subtitles is probably not used much, but plenty of us use chapter images, I myself have them enabled, but my memory is usually less than 500MB  so if this was the issue, why is my memory stable and not climbing?

 

Seems to me if these 2 scheduled tasks are the cause of the memory issue it would effect a greater number of people than what has been reported.

[Happy2Play 9780]

There are really only two times Chapter extractrion may be a issue, but they are still one time usages.  During initial library build or you are adding an extreme amount of data.  And even then I have have not seen an extreme amount of memory ever used.  I don't think I have ever seen Emby use over 1.5GB of memory and has alwas returned to a normal level.

[Luke 42077]

A very complex or large database query can cause sqlite to put a lot of data into memory temporarily, into what it calls temp tables. This can cause memory to spike, so that's what my recent changes were about. nothing to do with the ffmpeg side of things.

[Happy2Play 9780]

A very complex or large database query can cause sqlite to put a lot of data into memory temporarily, into what it calls temp tables. This can cause memory to spike, so that's what my recent changes were about. nothing to do with the ffmpeg side of things.

 

So all users should see this spike on new installations during chapter extraction?  At lease on my systems and setup there has never been over a gigabit spike in memory.

 

 

As a side note to letting a unconfigured new server installation run.  I had one run for a week and it used 150-200MB just sitting there.

[Luke 42077]

Everything will vary depending on usage patterns.

[Happy2Play 9780]

Everything will vary depending on usage patterns.

 

True, but if a user is seeing this issue on a unconfigured system and I am not shouldn't the usage pattern be the same.  Is it now a system or hardware problem?

[Luke 42077]

I wouldn't call that an issue. Nothing is spiking and constantly climbing.

[Happy2Play 9780]

While some people have Live TV, not all do.  I had a tuner, but I have never used it.. nor did I have guide data.  When I deleted it, the leak persisted after a reboot.

 

For example (a test I have been doing), I have an Emby Server 3.2.29.0 installation in Windows with NO libraries and NO Live TV that has been creeping up from 350MB to 1.3GB over the course of a week.  Granted this is a little slower than installations WITH libraries..  There is absolutely zero media.  I am going to let it go to see if it can crash the server.  The scheduled tasks are still there along with a few plug-ins.  If a server with no libraries & no tuners can leak memory like that, there is clearly an issue with some underlying core component as @@Luke suggested like sqlite or the web server.  I can probably delete all plug-ins & remove the schedules from all tasks & I think it will still leak.

Did you do a test without plugins?

Edited September 11, 2017 by Happy2Play

[jnheinz 17]

Did you do a test without plugins?

 

@@Happy2Play Not yet.  I am going to check it now.

 

This is 10 days with an Emby Server for Windows, 3.2.29 installation.. no libraries, no Live TV.

 

 

 

I will remove all plug-ins & reboot.

Edited September 11, 2017 by jnheinz

[jnheinz 17]

All plug-ins removed & Emby Service was restarted.  Will check back in a few days.

 

[TheHwyman 5]

Ok, got home late yesterday... wasn't too tired so had some time and decided to dig into this...

 

End result, I now know EXACTLY what is causing this issue for me and can re-create it at will.

 

What I discovered is very much specific to my environment, if it is applicable to others or not is unknown.

 

Without knowing what is taking place within the Emby s/w, I can't say for sure but it "appears" like it has something to do with error handling / connection handling by the HTTP server functions in Emby with "failed queries".

 

What I am seeing is one of the Kodi client machines (my best friend's machine) accessing Emby remotely over the Internet via SSL is generating errors, no other client is exhibiting this. He has a Windows based Kodi (Emby for Kodi client) box (the culprit here) and a Raspberry Pi Kodi (Emby for Kodi client) box (accessed via Internet SSL) not showing any issues, my sister has an Android Kodi (Emby for Kodi client) box (accessed via Internet SSL) and is not showing any issues and I have one Windows based Kodi (Emby for Kodi client) box (internal network using SSL) with no issues either.

 

I believe my buddy has done something with his Windows Kodi box that has royally corrupted Kodi's DB or something. He's always messing around with Kodi plugins, adding, removing, etc. The end result is that some of the music content from my media library being served up by Emby has gotten "unattached" from Emby somehow and Kodi is trying to resolve the music records in the DB itself separately from the Emby for Kodi client. Each record appears to be linked to my Emby server DNS name and SSL port and Kodi is trying to resolve them (1,000s of them). It takes Kodi about 1.5 hrs to complete trying to resolve each and every record and this corresponds exactly to the timeframe of my Emby service memory increases.

 

I called my buddy and told him what was happening, then had him go to the Emby for Kodi client's menu and do a Repair Database, All libraries. I figured this would scratch rebuild Kodi's DBs and re-import all Emby media. He did that, took about 2 hrs and completed. He tried playing some video, music, etc... all was working fine. I told him to leave it on overnight and through today, I would check the logs later and see what happens. So I immediately stopped/started the Emby service to reset it down from its 4.5 gb memory usage at that time and it started back up at ~250mb.

 

Checked the server this morning, was still sitting at ~250mb and the logs showed no signs of the error... cool. Did a bunch of work, then at just after 4:00pm I connected into my server and Emby was still sitting at ~260mb. I was just about to open the log file to confirm no more error msgs from my buddy's client when BAM... task manager showed Emby's memory climbing fast...going up 100mb every 4-5 sec and was now hitting 1.2 gb. I opened the server log file to see what was happening and sure enough my buddy's client was generating all those errors again looking for audio files. I called him and asked what he did, he said he was physically re-arranging stuff and had shut down his client box 15 minutes earlier, unplugged it from power, moved it, reconnected it up and restarted it just after 4:00. So I asked him to email me his Kodi log file since starting it up, he did. Here are some exerpts (I've removed DNS/IP/Ports):

 

16:10:15 T:3768 WARNING: MUSIC_INFO::CMusicInfoScanner::Process directory 'https://[internetDNSNameForEmby]:[sSLPort]/emby/Audio/003383b788ed05d1959035454ca21be3/' does not exist - skipping scan.
16:10:15 T:3768   ERROR: CCurlFile::Stat - Failed: Couldn't resolve host name(6) for https://[internetDNSNameForEmby]:[sSLPort]/emby/Audio/003692ec530d4c4d57bdd1aa605e4087/
16:10:15 T:3768 WARNING: MUSIC_INFO::CMusicInfoScanner::Process directory 'https://[internetDNSNameForEmby]:[sSLPort]/emby/Audio/003692ec530d4c4d57bdd1aa605e4087/' does not exist - skipping scan.
16:10:15 T:3768   ERROR: CCurlFile::Stat - Failed: Couldn't resolve host name(6) for https://[internetDNSNameForEmby]:[sSLPort]/emby/Audio/003a1d05b1be7c7132dcf7af35b8e34c/
16:10:15 T:3768 WARNING: MUSIC_INFO::CMusicInfoScanner::Process directory 'https://[internetDNSNameForEmby]:[sSLPort]/emby/Audio/003a1d05b1be7c7132dcf7af35b8e34c/' does not exist - skipping scan.
16:10:15 T:3768   ERROR: CCurlFile::Stat - Failed: Couldn't resolve host name(6) for https://[internetDNSNameForEmby]:[sSLPort]/emby/Audio/003c4d3473a7829c7740d8795465f3f0/
...
16:11:18 T:3720  NOTICE: EMBY.service_entry -> Server is online and ready
16:11:18 T:3752  NOTICE: EMBY.userclient -> ----====# Starting UserClient #====----
...
16:14:31 T:3768 WARNING: MUSIC_INFO::CMusicInfoScanner::Process directory 'https://[internetDNSNameForEmby]:[sSLPort]/emby/Audio/13240a52bd586bd1ee714777ca8f1622/' does not exist - skipping scan.
16:14:32 T:3768 WARNING: MUSIC_INFO::CMusicInfoScanner::Process directory 'https://[internetDNSNameForEmby]:[sSLPort]/emby/Audio/1325f444d8840eb935b5029c99ae41ec/' does not exist - skipping scan.
16:14:32 T:3768 WARNING: MUSIC_INFO::CMusicInfoScanner::Process directory 'https://[internetDNSNameForEmby]:[sSLPort]/emby/Audio/132d6e373c3c0c7a794616b52977570b/' does not exist - skipping scan.
 

Note that the Emby for Kodi client is set to connect after a 10 second delay from Kodi startup. I've only pasted a couple of lines but his log had about 100 of those lines before the Emby for Kodi client even connected/logged in and to me appears that Kodi is trying to resolve those DB entries (and failing) outside of the Emby plugin). Then once the Emby for Kodi client logs in, the error lines continue (object does not exist) but the "resolve hostname" lines stop.

 

When I look at my server's log file, there are no error entries prior to the client's Emby for Kodi client logging into the server. But as soon as he's logged in, the corresponding errors to his queries start to appear. See below:

 

2017-09-11 16:14:40.0231 Info HttpServer: HTTP HEAD https://[internetDNSNameForEmby]:[sSLPort]/emby/Audio/13240a52bd586bd1ee714777ca8f1622/. UserAgent: Kodi/16.0 (Windows NT 6.1; WOW64) App_Bitness/32 Version/16.0-Git:20160220-a5f3a99
2017-09-11 16:14:40.0231 Error HttpServer: Could not find handler for /emby/Audio/13240a52bd586bd1ee714777ca8f1622/
2017-09-11 16:14:40.0231 Info HttpServer: HTTP Response 404 to [ClientInternetIPAddress]. Time: 0ms. https://[internetDNSNameForEmby]:[sSLPort]/emby/Audio/13240a52bd586bd1ee714777ca8f1622/
2017-09-11 16:14:40.3043 Info HttpServer: HTTP HEAD https://[internetDNSNameForEmby]:[sSLPort]/emby/Audio/1325f444d8840eb935b5029c99ae41ec/. UserAgent: Kodi/16.0 (Windows NT 6.1; WOW64) App_Bitness/32 Version/16.0-Git:20160220-a5f3a99
2017-09-11 16:14:40.3043 Error HttpServer: Could not find handler for /emby/Audio/1325f444d8840eb935b5029c99ae41ec/
2017-09-11 16:14:40.3043 Info HttpServer: HTTP Response 404 to [ClientInternetIPAddress]. Time: 0ms. https://[internetDNSNameForEmby]:[sSLPort]/emby/Audio/1325f444d8840eb935b5029c99ae41ec/
2017-09-11 16:14:40.5543 Info HttpServer: HTTP HEAD https://[internetDNSNameForEmby]:[sSLPort]/emby/Audio/132d6e373c3c0c7a794616b52977570b/. UserAgent: Kodi/16.0 (Windows NT 6.1; WOW64) App_Bitness/32 Version/16.0-Git:20160220-a5f3a99
2017-09-11 16:14:40.5543 Error HttpServer: Could not find handler for /emby/Audio/132d6e373c3c0c7a794616b52977570b/
2017-09-11 16:14:40.5543 Info HttpServer: HTTP Response 404 to [ClientInternetIPAddress]. Time: 0ms. https://[internetDNSNameForEmby]:[sSLPort]/emby/Audio/132d6e373c3c0c7a794616b52977570b/
 

I am not sure how to interpret those errors, either the Emby service can't find the audio object being referenced or it is malformed and doesn't know how to deal with the request. But either way, as each incoming request is causing the service to gobble memory up until the requests stop coming in and then never releases the memory afterwards. I got my buddy to poweroff his client and told him he will need to uninstall Kodi, delete its folders and do a fresh install. From my side, it's been just about 2.5 hrs since he turned his client off and the Emby service is holding steady at 1.5gb and not coming down.

 

I took an indepth look at the server logs going back 5 days, along with perfmon traces... and they exactly correlate. Every single time the service shoots up in memory usage, the logs contain those above errors from his client... all the identical error msg lines, with only the object id # after "/Audio/...." being different. The client only seems to try to resolve those objects once a day, but when it does... it takes almost 1.5 hrs to try and resolve them all. Other than those time windows, the Emby service's memory usage is quite steady or at least goes up some then eventually comes back down... but the memory used during the errors never ever recovers.

 

So I am hoping that points you guys at something you can identify...

[revengineer 142]

All plug-ins removed & Emby Service was restarted.  Will check back in a few days.

 

Very interested in hearing about the outcome of this test. If it is still leaking memory, I suggest to next disable all scheduled task. Emby should be completely idle at that point.

 

On my end, i see different rates of leaking. A fast rate apparently associated with Life TV especially when viewed remotely from Android app and a slower rate (100 MB/day) even when nearly idle. I still have libraries, plugins, and scheduled tasks and cannot easily narrow down the culprit without taking the movies away from the family. Hence my interest in this test.

[TylerV76 84]

Since I turned off the ability to transcode to mkv while recording, I've only had one instance of memory problems.

 

Saturday I was watching a football game in 2 rooms while doing some work in the house. After about 30 minutes the tv stopped playing so I checked the local server and it had a memory warning and Emby completely crashed. I rebooted Emby and haven't had an issue since.

 

For me it's that one setting that caused issues. Based on this thread it appears multiple things can trigger the memory issue.

 

Sent from my SM-G950U1 using Tapatalk

[jnheinz 17]

@@Luke Why is this scheduled task firing when I have no triggers configured?  No Live TV, Live TV plug-ins or guide providers configured.

 

2017-09-11 20:12:39.633 Info TaskManager: IntervalTrigger fired for task: Refresh Guide

2017-09-11 20:12:39.633 Info TaskManager: Queueing task RefreshChannelsScheduledTask

2017-09-11 20:12:39.633 Info TaskManager: Executing Refresh Guide

2017-09-11 20:12:42.654 Info App: Refreshing guide with 7 days of guide data

2017-09-11 20:12:42.659 Info TaskManager: Refresh Guide Completed after 0 minute(s) and 3 seconds

2017-09-11 20:12:42.660 Info ServerManager: Sending web socket message ScheduledTaskEnded

2017-09-11 20:12:42.660 Info TaskManager: ExecuteQueuedTasks

 

[jnheinz 17]

Very interested in hearing about the outcome of this test. If it is still leaking memory, I suggest to next disable all scheduled task. Emby should be completely idle at that point.

 

On my end, i see different rates of leaking. A fast rate apparently associated with Life TV especially when viewed remotely from Android app and a slower rate (100 MB/day) even when nearly idle. I still have libraries, plugins, and scheduled tasks and cannot easily narrow down the culprit without taking the movies away from the family. Hence my interest in this test.

 

@@revengineer I can't do that with my main installation either.  So, I am using the shell of my previous Windows installation in a virtual machine for this test.  Removing scheduled tasks will be my test for next week.  You are right, it should hypothetically be doing nothing.  I have NO devices attached to it, no libraries, no plugins.  Removing the triggers for the scheduled tasks should send it to completely idle.  DLNA is also disabled.

Edited September 12, 2017 by jnheinz

[Happy2Play 9780]

 

@@Luke Why is this scheduled task firing when I have no triggers configured?  No Live TV, Live TV plug-ins or guide providers configured.

 

2017-09-11 20:12:39.633 Info TaskManager: IntervalTrigger fired for task: Refresh Guide

2017-09-11 20:12:39.633 Info TaskManager: Queueing task RefreshChannelsScheduledTask

2017-09-11 20:12:39.633 Info TaskManager: Executing Refresh Guide

2017-09-11 20:12:42.654 Info App: Refreshing guide with 7 days of guide data

2017-09-11 20:12:42.659 Info TaskManager: Refresh Guide Completed after 0 minute(s) and 3 seconds

2017-09-11 20:12:42.660 Info ServerManager: Sending web socket message ScheduledTaskEnded

2017-09-11 20:12:42.660 Info TaskManager: ExecuteQueuedTasks

 

 

 

Because they are two different things.  "Refresh Guide" is not a channel.

[jnheinz 17]

Because they are two different things.  "Refresh Guide" is not a channel.

 

RefreshChannelsScheduledTask appears to be the name of the task & Refresh Guide appears to be a friendly name & appears to be synonymous when referencing the task.  That blurb from the log is referencing one thing.. one scheduled task that ran in 3 seconds.

[Happy2Play 9780]

RefreshChannelsScheduledTask appears to be the name of the task & Refresh Guide appears to be a friendly name & appears to be synonymous when referencing the task.  That blurb from the log is referencing one thing.. one scheduled task that ran in 3 seconds.

 

Different task numbers,  You can look at the different tasks in you data\scheduled tasks folder

 

Guide   3145e441-d9ba-27fc-391e-7ee15face581

Channel a27c5425-9520-b2e5-c509-def68c8f4c45

[jnheinz 17]

 

 

Different task numbers,  You can look at the different tasks in you data\scheduled tasks folder

 

Guide   3145e441-d9ba-27fc-391e-7ee15face581

Channel a27c5425-9520-b2e5-c509-def68c8f4c45

 

Then how would I disable that one?

[revengineer 142]

@@revengineer I can't do that with my main installation either.  So, I am using the shell of my previous Windows installation in a virtual machine for this test.  Removing scheduled tasks will be my test for next week.  You are right, it should hypothetically be doing nothing.  I have NO devices attached to it, no libraries, no plugins.  Removing the triggers for the scheduled tasks should send it to completely idle.  DLNA is also disabled.

BTW, are you running emby as application or service in your testing? (Maybe I missed this.)

[jnheinz 17]

BTW, are you running emby as application or service in your testing? (Maybe I missed this.)

 

Service.  I do not feel there should be an actual difference.  That is my opinion.

[ebr 16169]

Service.  I do not feel there should be an actual difference.  That is my opinion.

 

Well, the two run under completely different environments as far as Windows is concerned.  That is beyond our control.