Server Crashes after running out of memory

August 19, 2018

H!

Around once a day my emby-server crashes after running out of memory and starting to swap. The memory consumption increases gradually over some minutes. I made a screenshot of the memory consumption graph of my server before such a crash:

5b79b75008437_Screenshot_20180817220854.

You can clearly see when i restarted the server (around 22:00) and the memory was freed up once again. The server doesn't really "crash", it just becomes unresponsive when it starts to swap memory. Before it starts swapping, emby runs just fine.

I will not attach the whole log, because it''s pretty big, but i noticed that the following exception get's logged during the time the memory consumption increases:

2018-08-19 19:50:15.131 Error HttpServer: WebSocketException Success
        *** Error Report ***
        Version: 3.5.2.0
        Command line: /opt/emby-server/system/EmbyServer.dll -programdata /var/lib/emby -ffmpeg /opt/emby-server/bin/ffmpeg -ffprobe /opt/emby-server/bin/ffprobe -restartexitcode 3 -updatepackage emby-server-deb_{version}_amd64.deb
        Operating system: Unix 4.15.0.32
        64-Bit OS: True
        64-Bit Process: True
        User Interactive: True
        Processor count: 8
        Program data path: /var/lib/emby
        Application directory: /opt/emby-server/system
        System.Net.WebSockets.WebSocketException (0x80004005): net_WebSockets_InvalidState
           at SocketHttpListener.Net.WebSockets.WebSocketValidate.ThrowIfInvalidState(WebSocketState currentState, Boolean isDisposed, WebSocketState[] validStates)
           at SocketHttpListener.Net.WebSockets.ManagedWebSocket.ReceiveAsync(ArraySegment`1 buffer, CancellationToken cancellationToken)
        --- End of stack trace from previous location where exception was thrown ---
           at SocketHttpListener.Net.WebSockets.WebSocket.ReceiveAsync(Memory`1 buffer, CancellationToken cancellationToken)
           at EmbyServer.SocketSharp.SharpWebSocket.StartReceive()
        System.Net.WebSockets.WebSocketException
           at SocketHttpListener.Net.WebSockets.WebSocketValidate.ThrowIfInvalidState(WebSocketState currentState, Boolean isDisposed, WebSocketState[] validStates)
           at SocketHttpListener.Net.WebSockets.ManagedWebSocket.ReceiveAsync(ArraySegment`1 buffer, CancellationToken cancellationToken)
        --- End of stack trace from previous location where exception was thrown ---
           at SocketHttpListener.Net.WebSockets.WebSocket.ReceiveAsync(Memory`1 buffer, CancellationToken cancellationToken)
           at EmbyServer.SocketSharp.SharpWebSocket.StartReceive()

This message gets logged around twice every milisecond - that's the reason why the logfiles are so big I guess. I will attach a small portion of the log, until the time I restarted the server. If you need the whole log, please tell me.

embyserver-log.txt

August 20, 2018

I decided to attach the whole log anyway. I cut off the end, because it's just the same exception over and over again. The first time the exception occurs is on line 241851.

emby-server-before-memory-leak-2.txt

Edited August 20, 2018 by thynan

August 20, 2018

its these errors in your log that are probably eating memory:

2018-08-19 11:48:21.316 Error HttpServer: Error in ProcessAccept
	*** Error Report ***
	Version: 3.5.2.0
	Command line: /opt/emby-server/system/EmbyServer.dll -programdata /var/lib/emby -ffmpeg /opt/emby-server/bin/ffmpeg -ffprobe /opt/emby-server/bin/ffprobe -restartexitcode 3 -updatepackage emby-server-deb_{version}_amd64.deb
	Operating system: Unix 4.15.0.32
	64-Bit OS: True
	64-Bit Process: True
	User Interactive: True
	Processor count: 8
	Program data path: /var/lib/emby
	Application directory: /opt/emby-server/system
	System.IO.IOException: Authentication failed because the remote party has closed the transport stream.
	   at System.Net.Security.SslState.StartReadFrame(Byte[] buffer, Int32 readBytes, AsyncProtocolRequest asyncRequest)
	   at System.Net.Security.SslState.PartialFrameCallback(AsyncProtocolRequest asyncRequest)
	--- End of stack trace from previous location where exception was thrown ---
	   at System.Net.Security.SslState.ThrowIfExceptional()
	   at System.Net.Security.SslState.InternalEndProcessAuthentication(LazyAsyncResult lazyResult)
	   at System.Net.Security.SslState.EndProcessAuthentication(IAsyncResult result)
	   at System.Net.Security.SslStream.EndAuthenticateAsServer(IAsyncResult asyncResult)
	   at System.Net.Security.SslStream.<>c.<AuthenticateAsServerAsync>b__50_2(IAsyncResult iar)
	   at System.Threading.Tasks.TaskFactory`1.FromAsyncCoreLogic(IAsyncResult iar, Func`2 endFunction, Action`1 endAction, Task`1 promise, Boolean requiresSynchronization)
	--- End of stack trace from previous location where exception was thrown ---
	   at SocketHttpListener.Net.HttpConnection.Init()
	   at SocketHttpListener.Net.HttpEndPointListener.ProcessAccept(SocketAsyncEventArgs args)
	System.IO.IOException
	   at System.Net.Security.SslState.StartReadFrame(Byte[] buffer, Int32 readBytes, AsyncProtocolRequest asyncRequest)
	   at System.Net.Security.SslState.PartialFrameCallback(AsyncProtocolRequest asyncRequest)
	--- End of stack trace from previous location where exception was thrown ---
	   at System.Net.Security.SslState.ThrowIfExceptional()
	   at System.Net.Security.SslState.InternalEndProcessAuthentication(LazyAsyncResult lazyResult)
	   at System.Net.Security.SslState.EndProcessAuthentication(IAsyncResult result)
	   at System.Net.Security.SslStream.EndAuthenticateAsServer(IAsyncResult asyncResult)
	   at System.Net.Security.SslStream.<>c.<AuthenticateAsServerAsync>b__50_2(IAsyncResult iar)
	   at System.Threading.Tasks.TaskFactory`1.FromAsyncCoreLogic(IAsyncResult iar, Func`2 endFunction, Action`1 endAction, Task`1 promise, Boolean requiresSynchronization)
	--- End of stack trace from previous location where exception was thrown ---
	   at SocketHttpListener.Net.HttpConnection.Init()
	   at SocketHttpListener.Net.HttpEndPointListener.ProcessAccept(SocketAsyncEventArgs args)

Unfortunately you have removed lines directly before these started at the top of the log so unsure how far server got when starting up before these errors started.

@@Luke will have to give guidance.

Its obviously to do with ssl and there were posts about issues with ssl on Linux but thought they were all docker related.

Krs

Mark

Edited August 20, 2018 by maegibbons

August 20, 2018

Hi! Thanks for your reply. I don't think these errors are the problem. They happen the during the whole time the server is online, and the memory usage doesn't increase. As you can see in the above graphic, the memory usage starts increasing rapidly over around 30 minutes.

I think it's more likely the error message I pasted in my first post is the culprint, because they start around the time when the memory consumption goes up. I don't know where that comes from however.

PS: The Screenshot of the memory consumption is not from the same crash as the log - so don't get confused if the times don't match up exactly.

August 20, 2018

You still should get to the bottom of these errors. A Log full of errors is unhelpful at the very least and these errors/frequency are not expected in a correctly operating system.

Krs

Mark

August 20, 2018

Yes, that's certainly true, and I already tried to solve this some time ago, but was not able to find a solution as of yet. At the moment I want to fix the more serious errors which actually crash my server every day, before fixing the rest.

August 21, 2018

Hi, we're looking into this, thanks.

August 22, 2018

Hi! Thanks for the info. Yesterday it happened again. It was right after a user restarted his chromecast. At the same time a user connected and started streaming with kodi. I didn't have time to check yesterday, but I will try to reproduce both scenarios and see if the error can be reproduced as well. It's a long shot, but I recall someone posting about server crashes after a user restarted a chromecast while it was still connected to emby.

August 24, 2018

Emby on my synology quit working today, a restart of the server didn't help, had to restart the NAS.

Here's the errors in the log.txt that kept repeating when trying to play live tv.

Thanks, and let me know if you need any more information to help diagnose.

2018-08-24 09:39:57.775 Error HttpServer: Error processing request

*** Error Report ***

Version: 3.5.2.0

Command line: /var/packages/EmbyServer/target/server/MediaBrowser.Server.Mono.exe -package synology -programdata /var/packages/EmbyServer/target/var -ffmpeg /var/packages/EmbyServer/target/ffmpeg/bin/ffmpeg -ffprobe /var/packages/EmbyServer/target/ffmpeg/bin/ffprobe -restartpath /var/packages/EmbyServer/scripts/restart

Operating system: Unix 4.4.59.0

64-Bit OS: True

64-Bit Process: True

User Interactive: False

Mono: 5.10.1.47 (tarball Sat Apr 21 19:46:45 CST 2018)

Processor count: 2

Program data path: /var/packages/EmbyServer/target/var

Application directory: /volume1/@appstore/EmbyServer/releases/v3.5.2.0

System.ComponentModel.Win32Exception (0x80004005): ApplicationName='/var/packages/EmbyServer/target/ffmpeg/bin/ffprobe', CommandLine='-analyzeduration 3000000 -i "http://127.0.0.1:8096/LiveTv/LiveStreamFiles/fd4fdc9276d147b598ae77ab78f1c39c/stream.ts"-threads 0 -v info -print_format json -show_streams -show_format', CurrentDirectory='', Native error= Out of memory

at System.Diagnostics.Process.StartWithCreateProcess (System.Diagnostics.ProcessStartInfo startInfo) [0x002dc] in <d6c698d00e4945d0b7ff1f172eba44c7>:0

at System.Diagnostics.Process.Start () [0x0003a] in <d6c698d00e4945d0b7ff1f172eba44c7>:0

at (wrapper remoting-invoke-with-check) System.Diagnostics.Process.Start()

at Emby.Server.Implementations.Diagnostics.CommonProcess.Start () [0x00000] in <ea7199aea78e429a84e168a1e154646a>:0

at Emby.Server.MediaEncoding.Encoder.MediaEncoder+ProcessWrapper.Start () [0x00015] in <c2048c69b69a418e87e22afd7487c2f5>:0

at Emby.Server.MediaEncoding.Encoder.MediaEncoder.StartProcess (Emby.Server.MediaEncoding.Encoder.MediaEncoder+ProcessWrapper process) [0x00000] in <c2048c69b69a418e87e22afd7487c2f5>:0

at Emby.Server.MediaEncoding.Encoder.MediaEncoder+<GetMediaInfoInternal>d__69.MoveNext () [0x00120] in <c2048c69b69a418e87e22afd7487c2f5>:0

--- End of stack trace from previous location where exception was thrown ---

at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw () [0x0000c] in <9f46da0973534d9c8737c50aeebf5ecb>:0

at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (System.Threading.Tasks.Task task) [0x0003e] in <9f46da0973534d9c8737c50aeebf5ecb>:0

at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Threading.Tasks.Task task) [0x00028] in <9f46da0973534d9c8737c50aeebf5ecb>:0

at System.Runtime.CompilerServices.TaskAwaiter.ValidateEnd (System.Threading.Tasks.Task task) [0x00008] in <9f46da0973534d9c8737c50aeebf5ecb>:0

at System.Runtime.CompilerServices.ConfiguredTaskAwaitable`1+ConfiguredTaskAwaiter[TResult].GetResult () [0x00000] in <9f46da0973534d9c8737c50aeebf5ecb>:0

at Emby.Server.Implementations.Library.MediaSourceManager+<GetLiveStreamMediaInfo>d__38.MoveNext () [0x00134] in <ea7199aea78e429a84e168a1e154646a>:0

--- End of stack trace from previous location where exception was thrown ---

at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw () [0x0000c] in <9f46da0973534d9c8737c50aeebf5ecb>:0

at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (System.Threading.Tasks.Task task) [0x0003e] in <9f46da0973534d9c8737c50aeebf5ecb>:0

at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Threading.Tasks.Task task) [0x00028] in <9f46da0973534d9c8737c50aeebf5ecb>:0

at System.Runtime.CompilerServices.TaskAwaiter.ValidateEnd (System.Threading.Tasks.Task task) [0x00008] in <9f46da0973534d9c8737c50aeebf5ecb>:0

at System.Runtime.CompilerServices.ConfiguredTaskAwaitable`1+ConfiguredTaskAwaiter[TResult].GetResult () [0x00000] in <9f46da0973534d9c8737c50aeebf5ecb>:0

at Emby.Server.MediaEncoding.Api.MediaInfoService+<Post>d__16.MoveNext () [0x00081] in <c2048c69b69a418e87e22afd7487c2f5>:0

--- End of stack trace from previous location where exception was thrown ---

at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw () [0x0000c] in <9f46da0973534d9c8737c50aeebf5ecb>:0

at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (System.Threading.Tasks.Task task) [0x0003e] in <9f46da0973534d9c8737c50aeebf5ecb>:0

at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Threading.Tasks.Task task) [0x00028] in <9f46da0973534d9c8737c50aeebf5ecb>:0

at System.Runtime.CompilerServices.TaskAwaiter.ValidateEnd (System.Threading.Tasks.Task task) [0x00008] in <9f46da0973534d9c8737c50aeebf5ecb>:0

at System.Runtime.CompilerServices.ConfiguredTaskAwaitable`1+ConfiguredTaskAwaiter[TResult].GetResult () [0x00000] in <9f46da0973534d9c8737c50aeebf5ecb>:0

at Emby.Server.Implementations.Services.ServiceExecGeneral+<GetTaskResult>d__3.MoveNext () [0x00081] in <ea7199aea78e429a84e168a1e154646a>:0

--- End of stack trace from previous location where exception was thrown ---

at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw () [0x0000c] in <9f46da0973534d9c8737c50aeebf5ecb>:0

at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (System.Threading.Tasks.Task task) [0x0003e] in <9f46da0973534d9c8737c50aeebf5ecb>:0

at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Threading.Tasks.Task task) [0x00028] in <9f46da0973534d9c8737c50aeebf5ecb>:0

at System.Runtime.CompilerServices.TaskAwaiter.ValidateEnd (System.Threading.Tasks.Task task) [0x00008] in <9f46da0973534d9c8737c50aeebf5ecb>:0

at System.Runtime.CompilerServices.ConfiguredTaskAwaitable`1+ConfiguredTaskAwaiter[TResult].GetResult () [0x00000] in <9f46da0973534d9c8737c50aeebf5ecb>:0

at Emby.Server.Implementations.Services.ServiceHandler+<ProcessRequestAsync>d__13.MoveNext () [0x001c2] in <ea7199aea78e429a84e168a1e154646a>:0

--- End of stack trace from previous location where exception was thrown ---

at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw () [0x0000c] in <9f46da0973534d9c8737c50aeebf5ecb>:0

at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (System.Threading.Tasks.Task task) [0x0003e] in <9f46da0973534d9c8737c50aeebf5ecb>:0

at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Threading.Tasks.Task task) [0x00028] in <9f46da0973534d9c8737c50aeebf5ecb>:0

at System.Runtime.CompilerServices.TaskAwaiter.ValidateEnd (System.Threading.Tasks.Task task) [0x00008] in <9f46da0973534d9c8737c50aeebf5ecb>:0

at System.Runtime.CompilerServices.ConfiguredTaskAwaitable+ConfiguredTaskAwaiter.GetResult () [0x00000] in <9f46da0973534d9c8737c50aeebf5ecb>:0

at Emby.Server.Implementations.HttpServer.HttpListenerHost+<RequestHandler>d__60.MoveNext () [0x009e8] in <ea7199aea78e429a84e168a1e154646a>:0

System.ComponentModel.Win32Exception