Emby Server has recently been crashing

[jp.server 0]

I've noticed that recently Emby has started to crash if I'm selecting somewhat rapidly between movies in my library. I'm by no means an Emby expert, but I have software experience so I checked the log file and found the below:

2017-01-23 07:14:06.4887 Error HttpServer: Error processing request
    *** Error Report ***
    Version: 3.1.2.0
    Command line: /usr/lib/emby-server/bin/MediaBrowser.Server.Mono.exe -programdata /var/lib/emby-server -ffmpeg /usr/bin/ffmpeg -ffprobe /usr/bin/ffprobe -restartpath /usr/lib/emby-server/restart.sh
    Operating system: Unix 4.4.0.59
    64-Bit OS: True
    64-Bit Process: True
    Mono: 4.6.2 (Stable 4.6.2.7/08fd525 Mon Nov 28 20:49:26 UTC 2016)
    Processor count: 8
    Program data path: /var/lib/emby-server
    Application directory: /usr/lib/emby-server/bin
    System.IO.IOException: Unable to write data to the transport connection: The socket has been shut down. ---> System.Net.Sockets.SocketException: The socket has been shut down
     at System.Net.Sockets.Socket.EndSend (System.IAsyncResult result) [0x00033] in <bd46d4d4f7964dfa9beea098499ab597>:0
     at System.Net.Sockets.NetworkStream.EndWrite (System.IAsyncResult asyncResult) [0x0005f] in <bd46d4d4f7964dfa9beea098499ab597>:0
     --- End of inner exception stack trace ---
     at Mono.Security.Protocol.Tls.SslStreamBase.EndWrite (System.IAsyncResult asyncResult) [0x00057] in <1d0bb82c94e7435eb09324cf5ef20e36>:0
     at Mono.Net.Security.Private.LegacySslStream.EndWrite (System.IAsyncResult asyncResult) [0x00006] in <bd46d4d4f7964dfa9beea098499ab597>:0
     at Mono.Net.Security.Private.MonoSslStreamImpl.EndWrite (System.IAsyncResult asyncResult) [0x00006] in <bd46d4d4f7964dfa9beea098499ab597>:0
     at System.Net.Security.SslStream.EndWrite (System.IAsyncResult asyncResult) [0x00006] in <bd46d4d4f7964dfa9beea098499ab597>:0
     at System.IO.Stream.<BeginEndWriteAsync>m__8 (System.IO.Stream stream, System.IAsyncResult asyncResult) [0x00000] in <8f2c484307284b51944a1a13a14c0266>:0
     at (wrapper delegate-invoke) System.Func`3[System.IO.Stream,System.IAsyncResult,System.Threading.Tasks.VoidTaskResult]:invoke_TResult_T1_T2 (System.IO.Stream,System.IAsyncResult)
     at System.Threading.Tasks.TaskFactory`1+FromAsyncTrimPromise`1[TResult,TInstance].Complete (TInstance thisRef, System.Func`3[T1,T2,TResult] endMethod, System.IAsyncResult asyncResult, System.Boolean requiresSynchronization) [0x00002] in <8f2c484307284b51944a1a13a14c0266>:0
    --- End of stack trace from previous location where exception was thrown ---
     at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw () [0x0000c] in <8f2c484307284b51944a1a13a14c0266>:0
     at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (System.Threading.Tasks.Task task) [0x0004e] in <8f2c484307284b51944a1a13a14c0266>:0
     at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Threading.Tasks.Task task) [0x0002e] in <8f2c484307284b51944a1a13a14c0266>:0

Is there anything that this error indicates that I could be doing differently to fix the situation? I've just started noticing this recently, and a quick restart of emby-server seems to fix it. But obviously, that is not desired to have to restart the service every 4 movies. 

 

Also, if there's anything else to check, please let me know and I'll post it.

 

I'm running Emby on Ubuntu Server 16.04 FYI - ASUS mobo w/ i7-6700k

 

Edit: From looking at other posts, it looks like mono sometimes affects stuff like this, but it looks like I'm running 4.6.2.7 which is not a version that folks have said causes problems. Checking the mono-project website I see 4.6.2.16 is released though, so my next step will be trying to update that.

$ mono --version
Mono JIT compiler version 4.6.2 (Stable 4.6.2.7/08fd525 Mon Nov 28 20:49:26 UTC 2016)

Edited January 23, 2017 by jp.server

[Luke 39422]

Hi there @jp.server, can you please attach the complete emby server log? thanks !

[jp.server 0]

Thanks for the quick reply! Attached are the two most recent log files. The file ending in ...2390 is from this morning, and shows the error messages from my post. The other file i'm just including in case, just because I saved it - it's the most recent log file from this evening. Emby seems to be running fine since the last restart this morning - I have not tried the same behavior since.

 

Please let me know if there's anything else I can provide or if you have anything for me to try. I can try to reproduce the issue again and capture the logs again if you would like. Thanks a ton for the help!!

server-63620752390.txt

server-63620752540.txt

[Luke 39422]

does this happen over the LAN with plain http?

[jp.server 0]

wow you're good. I just tried it using http on LAN and it seems like everything is working as expected. So that seems to indicate it's something about SSL/https. What do you think? Is there something you'd suggest for me to try? I'm currently using the self-signed certificate option

[Katia Rosado 0]

I'm having a similar problem...Everything was ok but yesterday i added a music folder and i'm back at getting stuck at 97.5% on Scan media library.

Tried to update my mono but i'm still at Stable 4.6.2.7 too.

I've attached my latest 2 logs

Log (1).txt

Log (2).txt

[Luke 39422]

Hi @@Katia Rosado, you're not stuck. Just give it time.

 

Additionally, go into each library in Emby library setup and uncheck the box to "Import missing episodes", and then that part of the scan will be faster. Let us know how it goes. thanks.

[Katia Rosado 0]

Hi @@Luke. For a week now every library scan stops at 97,5%.

Updated to the new version but the library thing remains so i just removed the trigger to do library scans and everyday im doing it manually but it never completes.

The other thing is this...When you say "import missing episodes" your refering to this? It has always been off by default in my case.

Thank you anyway.

 

[Luke 39422]

Hi @@Katia Rosado, I can pretty much promise it is not stuck, just be patient and allow it to complete.

 

If you just unchecked that checkbox it will be a little extra busy now removing data from the database, but going forward after that it will be faster. If you still think you're seeing an issue then please attach the complete emby server log. thanks !

[echoxxzz 22]

I have this same problem. Whenever I change rapidly between movies in my library emby crashes. I've tested this on Ubuntu 16.04, Ubuntu 16.10, Ubuntu 17.04, Centos 7 and Debian 9 and within 2 minutes of clicking between movies emby crashes. 

 

I tried updating to the latest Mono from the Mono website and that made zero difference.

 

I've switched back to Windows and so far so good no crashing.

 

The only real difference is that under Linux I was using NFS to access my movies on my NAS and under Windows I'm using SMB but plan to try NFS although I doubt this is the culprit since all you have to do is open the movie not actually play it to crash emby. 

[Luke 39422]

Hi there @@echoxxzz, we're very sorry to hear about your issue. We'd like to work with you to resolve it. Can you please discuss a specific example? Please provide the information requested in how to report a media playback issue. thanks !

[echoxxzz 22]

It's not really a media play back issue. It's a problem with the emby interface. Just keep clicking on a movie title and then go back to the library and click again. If you do it a dozen or so times the whole emby server crashes.

 

I've repeated my testing on a VMware virtual machine using Centos 7, Debian 9, Ubuntu 17.04 and 16,04 and I can replicate the problem with ease. If I create a Windows VM seems to be stable so far.

 

I have checked the emby logs and the syslog and there isn't even any info. I think the server crashes before it has a chance to log the error.

[Luke 39422]

Hi there @@echoxxzz, even though you think there is nothing useful in the server logs, can you please attach them anyway? thanks !

[Cerothen 95]

I had this problem for a long time also, my solution was to write the script below and run it with cron every minute, it creates a small log file which give info about the up time.

#!/bin/bash

# Settings
service="emby-server"
updateinterval=1440
logfile="/home/$service-status.log"

#Functions
getValue() {
	output=$(grep -oP "(?<=$1 = )\d+" $logfile)
	echo $output
}

incCounter() {
	var=$(( $(getValue "$1") + 1 ))
	sed -i -r "s/$1 = .*$/$1 = $var/" $logfile
	var=$(date)
	sed -i -r "s/$1 Last = .*$/$1 Last = $var/" $logfile
}

logDetails() {
	echo "$1" >> $logfile
}

# Details
servicestatus="$(service $service status)"

#Build File if Missing
if [ ! -f "$logfile" ]; then
    touch $logfile
	echo "Total = 0" >> $logfile
	echo "Total Last = Never" >> $logfile
	echo "Up = 0" >> $logfile
	echo "Up Last = Never" >> $logfile
	echo "Down = 0" >> $logfile
	echo "Down Last = Never" >> $logfile
	echo "Update Check = 0" >> $logfile
	echo "Update Check Last = Never" >> $logfile
	echo "Update Apply = 0" >> $logfile
	echo "Update Apply Last = Never" >> $logfile
	echo "" >> $logfile
	echo "Interesting Events:" >> $logfile
fi

#Increment Total
incCounter "Total"

#Check if running
if [[ $servicestatus == *"is not running"* ]]; then
	incCounter "Down"
	logDetails "[$(date)] $servicestatus"
	logDetails "[$(date)] $(service $service start)"
	logDetails "[$(date)] $(service $service status)"
else
	incCounter "Up"
	
	#Check if we need to do an update
	total=$(getValue 'Total')
	checkupdate=$(($total % $updateinterval))
	
	if [ $checkupdate -eq 0 ]; then
		incCounter "Update Check"
		
		#Get new package info
		apt-get update
		upgradedata="$(apt-get upgrade -y)"
	
		if [[ $upgradedata == *"The following packages will be upgraded:"*"$service"*"upgraded,"*"newly installed,"*"to remove and"*"not upgraded."* ]]; then
			incCounter "Update Apply"
			logDetails "[$(date)] $service was upgraded to the latest version."
		fi
	fi
fi

[Luke 39422]

@@Cerothen what problem have you had exactly?

[echoxxzz 22]

I did three attempts to replicate the problem and I was able to get the server to crash within 10-20 clicks. I have attached the three log files.

server-63634254753.txt

server-63634274213.txt

server-63634274995.txt

[Luke 39422]

Thanks for the info, we'll take a look.

[Cerothen 95]

@@Luke, it used to happen more frequently (a year or so ago) it hasn't really happened lately. I just keep it running since it handles updates a little smoother, I never found that Emby would restart (or start back up if manually restarted in the web gui).

 

Its really not an issue for me. I recently updated the script though since Ombi was also being disagreeable to staying running (same script is used for each) so I cleared out all the history.

[Luke 39422]

Thanks for the feedback.