Jump to content
Sign in to follow this  
thefirstofthe300

Summary of the random crashing problem on Ubuntu and Debian

Recommended Posts

thefirstofthe300
If you are running an Ubuntu or Debian server and mono version 3.10 or older or 4.0 or newer and are experiencing random crashing of Emby with no errors in your logs, you might be experiencing an issue related to a bug in the kernel.
 
Finally one of the Sonarr devs popped in and helped us figure out a possible solution to the problem.
 

Hey guys, I'm on of the devs over at Sonarr. We (probably) ran into the same problem. Better part is that we isolated the problem in the linux kernel.

Would cause all kinds of native SIGSEGV and NullReferenceExceptions, took me almost a month and 100+ hours to isolate it.

 

Anyway, the problem I found only occurs in a virtual machine, afaik anyway, so if you're not running a vm then you can probably stop reading.

 

The fix for ubuntu is committed in their repository, but we'll have to wait patiently for it to get released.

Or more precisely, the commits needed were backported from linux 4.0.

 
Happened to all mono versions tested, and started with Ubuntu kernel 3.13.0-48. At first we thought it was user-specific, but as more and more reports started to accrue we had to investigate.
In hindsight it was rather obvious, slowly users started to install and boot into 3.13.0-48 in late March explaining the slow increase in affected users.
 

Some links:

https://forums.sonarr.tv/t/native-mono-crashes/4985

https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1450584

 

I haven't reported it yet to the debian team.

 

I kindly ask that someone over at Emby coordinates this across the forum threads and github issues.

 

Of course one of you will have to test if the proposed fix actually solves the problem for Emby, which involves compiling a kernel.

 

 

Cheers guys and have fun!

 
 
Once users started upgrading their kernels to as new as 4.0, they reported that the stability of Emby has greatly improved.
 
Here are some instructions on how to upgrade your kernel without having to do a distro upgrade: http://askubuntu.com/questions/119080/how-to-update-kernel-to-the-latest-mainline-version-without-any-distro-upgrade
 
Also, a few users reported that they were still experiencing crashed even though they weren't running Emby in a VM. Many of them also reported that a kernel upgrade actually fixed their problem.
 
If you are still having issues with Emby crashing after a kernel upgrade and you don't know why, please let us know.
Edited by DaBungalow
  • Like 4

Share this post


Link to post
Share on other sites
Luke

Great stuff!

  • Like 1

Share this post


Link to post
Share on other sites
thefirstofthe300

Since I don't have the ability to pin this topic, would someone else please do it. I think this would be a good thing to have pinned for a while.

Share this post


Link to post
Share on other sites
ryantiger658

Thank you for the summary!

I just updated to a newer kernel and I am still having the same problem... I am running Ubuntu Trusty. 

 

My (complete) logs are as follows:

 

 

2015-05-26 19:10:45.6302 Info - Main: Emby

Command line: /opt/emby/MediaBrowser.Server.Mono.exe -programdata /var/lib/emby
Operating system: Unix 4.0.4.40004
Processor count: 2
64-Bit OS: True
64-Bit Process: True
Program data path: /var/lib/emby
Mono: 4.0.1 (tarball Tue May 12 18:04:15 UTC 2015)
Application Path: /opt/emby/MediaBrowser.Server.Mono.exe
2015-05-26 19:10:45.8506 Info - App: Application version: 3.0.5607.2
2015-05-26 19:10:45.9281 Info - App: Application configuration:
{"EnableUPnP":true,"PublicPort":8096,"PublicHttpsPort":8920,"HttpServerPortNumber":8096,"HttpsPortNumber":8920,"EnableHttps":false,"EnableUserSpecificUserViews":true,"EnableInternetProviders":true,"IsPortAuthorized":true,"MetadataPath":"/MBData/Metadata","SeasonZeroDisplayName":"Specials","SaveLocalMeta":false,"EnableLocalizedGuids":true,"EnableLibraryMetadataSubFolder":true,"PreferredMetadataLanguage":"en","MetadataCountryCode":"US","SortReplaceCharacters":[".","+","%"],"SortRemoveCharacters":[",","&","-","{","}","'"],"SortRemoveWords":["the","a","an"],"MinResumePct":5,"MaxResumePct":90,"MinResumeDurationSeconds":300,"RealtimeLibraryMonitorDelay":40,"EnableDashboardResponseCaching":true,"EnableDashboardResourceMinification":true,"DashboardSourcePath":"","MergeMetadataAndImagesByName":true,"EnableStandaloneMetadata":true,"ImageSavingConvention":"Compatible","MetadataOptions":[{"ItemType":"Book","ImageOptions":[{"Type":"Backdrop","Limit":1,"MinWidth":1280}],"DisabledMetadataSavers":[],"LocalMetadataReaderOrder":[],"DisabledMetadataFetchers":[],"MetadataFetcherOrder":[],"DisabledImageFetchers":[],"ImageFetcherOrder":[]},{"ItemType":"Movie","ImageOptions":[{"Type":"Backdrop","Limit":3,"MinWidth":1280},{"Type":"Art","Limit":0,"MinWidth":0},{"Type":"Disc","Limit":0,"MinWidth":0},{"Type":"Primary","Limit":1,"MinWidth":0},{"Type":"Banner","Limit":1,"MinWidth":0},{"Type":"Thumb","Limit":1,"MinWidth":0},{"Type":"Logo","Limit":1,"MinWidth":0}],"DisabledMetadataSavers":["Emby Xml"],"LocalMetadataReaderOrder":[],"DisabledMetadataFetchers":[],"MetadataFetcherOrder":[],"DisabledImageFetchers":[],"ImageFetcherOrder":[]},{"ItemType":"Series","ImageOptions":[{"Type":"Backdrop","Limit":2,"MinWidth":1280},{"Type":"Art","Limit":0,"MinWidth":0},{"Type":"Primary","Limit":1,"MinWidth":0},{"Type":"Banner","Limit":1,"MinWidth":0},{"Type":"Thumb","Limit":1,"MinWidth":0},{"Type":"Logo","Limit":1,"MinWidth":0}],"DisabledMetadataSavers":["Emby Xml"],"LocalMetadataReaderOrder":[],"DisabledMetadataFetchers":[],"MetadataFetcherOrder":[],"DisabledImageFetchers":[],"ImageFetcherOrder":[]},{"ItemType":"MusicAlbum","ImageOptions":[{"Type":"Backdrop","Limit":1,"MinWidth":1280},{"Type":"Disc","Limit":0,"MinWidth":0}],"DisabledMetadataSavers":["Emby Xml"],"LocalMetadataReaderOrder":[],"DisabledMetadataFetchers":[],"MetadataFetcherOrder":[],"DisabledImageFetchers":[],"ImageFetcherOrder":[]},{"ItemType":"MusicArtist","ImageOptions":[{"Type":"Backdrop","Limit":1,"MinWidth":1280},{"Type":"Banner","Limit":0,"MinWidth":0},{"Type":"Art","Limit":0,"MinWidth":0}],"DisabledMetadataSavers":["Emby Xml"],"LocalMetadataReaderOrder":[],"DisabledMetadataFetchers":[],"MetadataFetcherOrder":[],"DisabledImageFetchers":[],"ImageFetcherOrder":[]},{"ItemType":"Season","ImageOptions":[{"Type":"Backdrop","Limit":0,"MinWidth":1280}],"DisabledMetadataSavers":["Emby Xml"],"LocalMetadataReaderOrder":[],"DisabledMetadataFetchers":[],"MetadataFetcherOrder":[],"DisabledImageFetchers":[],"ImageFetcherOrder":[]},{"ItemType":"Episode","ImageOptions":[{"Type":"Backdrop","Limit":3,"MinWidth":1280}],"DisabledMetadataSavers":["Emby Xml"],"LocalMetadataReaderOrder":[],"DisabledMetadataFetchers":[],"MetadataFetcherOrder":[],"DisabledImageFetchers":[],"ImageFetcherOrder":[]},{"ItemType":"MusicVideo","ImageOptions":[{"Type":"Backdrop","Limit":3,"MinWidth":1280}],"DisabledMetadataSavers":["Emby Xml"],"LocalMetadataReaderOrder":[],"DisabledMetadataFetchers":[],"MetadataFetcherOrder":[],"DisabledImageFetchers":[],"ImageFetcherOrder":[]},{"ItemType":"Video","ImageOptions":[{"Type":"Backdrop","Limit":3,"MinWidth":1280}],"DisabledMetadataSavers":["Emby Xml"],"LocalMetadataReaderOrder":[],"DisabledMetadataFetchers":[],"MetadataFetcherOrder":[],"DisabledImageFetchers":[],"ImageFetcherOrder":[]}],"EnableAutomaticRestart":true,"EnableRealtimeMonitor":true,"PathSubstitutions":[],"ServerName":"","UICulture":"en-us","PeopleMetadataOptions":{"DownloadActorMetadata":true,"DownloadDirectorMetadata":true,"DownloadProducerMetadata":true,"DownloadWriterMetadata":true,"DownloadComposerMetadata":false,"DownloadOtherPeopleMetadata":false,"DownloadGuestStarMetadata":true},"FindInternetTrailers":true,"InsecureApps9":["Chromecast","iOS","Unknown app","iPad","iPhone","Windows Phone"],"SaveMetadataHidden":true,"ContentTypes":[],"EnableAudioArchiveFiles":false,"EnableVideoArchiveFiles":false,"RemoteClientBitrateLimit":0,"EnableDebugLevelLogging":false,"EnableAutoUpdate":true,"SystemUpdateLevel":"Beta","LogFileRetentionDays":3,"RunAtStartup":false,"IsStartupWizardCompleted":true,"CachePath":"/MBData/Cache","EnableCustomPathSubFolders":true}
2015-05-26 19:10:45.9482 Info - App: Loading EmbyTV, Version=0.10.2.0, Culture=neutral, PublicKeyToken=null
2015-05-26 19:10:45.9482 Info - App: Loading GenreCleaner, Version=1.0.1.5, Culture=neutral, PublicKeyToken=null
2015-05-26 19:10:45.9482 Info - App: Loading MBBackup, Version=1.0.0.4, Culture=neutral, PublicKeyToken=null
2015-05-26 19:10:45.9482 Info - App: Loading MediaBrowser.Plugins.PushBulletNotifications, Version=3.0.5491.36573, Culture=neutral, PublicKeyToken=null
2015-05-26 19:10:45.9482 Info - App: Loading MediaBrowser.Plugins.RottenTomatoes, Version=1.0.5385.34041, Culture=neutral, PublicKeyToken=null
2015-05-26 19:10:45.9482 Info - App: Loading MediaBrowser.Plugins.Trailers, Version=1.0.5573.39176, Culture=neutral, PublicKeyToken=null
2015-05-26 19:10:45.9482 Info - App: Loading PodCasts, Version=1.0.5568.28062, Culture=neutral, PublicKeyToken=null
2015-05-26 19:10:45.9482 Info - App: Loading MediaBrowser.Plugins.StudioCleaner, Version=3.0.5519.24131, Culture=neutral, PublicKeyToken=null
2015-05-26 19:10:45.9482 Info - App: Loading Trakt, Version=3.0.5584.22492, Culture=neutral, PublicKeyToken=null
2015-05-26 19:10:45.9482 Info - App: Loading MediaBrowser.Api, Version=3.0.5607.2, Culture=neutral, PublicKeyToken=null
2015-05-26 19:10:45.9482 Info - App: Loading MediaBrowser.WebDashboard, Version=3.0.5607.2, Culture=neutral, PublicKeyToken=null
2015-05-26 19:10:45.9482 Info - App: Loading MediaBrowser.Model, Version=3.0.5607.2, Culture=neutral, PublicKeyToken=null
2015-05-26 19:10:45.9482 Info - App: Loading MediaBrowser.Common, Version=3.0.5607.2, Culture=neutral, PublicKeyToken=null
2015-05-26 19:10:45.9482 Info - App: Loading MediaBrowser.Controller, Version=3.0.5607.2, Culture=neutral, PublicKeyToken=null
2015-05-26 19:10:45.9482 Info - App: Loading MediaBrowser.Providers, Version=3.0.5607.2, Culture=neutral, PublicKeyToken=null
2015-05-26 19:10:45.9482 Info - App: Loading MediaBrowser.Common.Implementations, Version=3.0.5607.2, Culture=neutral, PublicKeyToken=null
2015-05-26 19:10:45.9482 Info - App: Loading MediaBrowser.Server.Implementations, Version=3.0.5607.2, Culture=neutral, PublicKeyToken=null
2015-05-26 19:10:45.9482 Info - App: Loading MediaBrowser.MediaEncoding, Version=3.0.5607.2, Culture=neutral, PublicKeyToken=null
2015-05-26 19:10:45.9482 Info - App: Loading MediaBrowser.Dlna, Version=3.0.5607.2, Culture=neutral, PublicKeyToken=null
2015-05-26 19:10:45.9482 Info - App: Loading MediaBrowser.LocalMetadata, Version=3.0.5607.2, Culture=neutral, PublicKeyToken=null
2015-05-26 19:10:45.9482 Info - App: Loading MediaBrowser.XbmcMetadata, Version=3.0.5607.2, Culture=neutral, PublicKeyToken=null
2015-05-26 19:10:45.9482 Info - App: Loading MediaBrowser.IsoMounting.Linux, Version=1.0.5131.24779, Culture=neutral, PublicKeyToken=null
2015-05-26 19:10:45.9482 Info - App: Loading MediaBrowser.Server.Mono, Version=3.0.5607.2, Culture=neutral, PublicKeyToken=null
2015-05-26 19:10:45.9482 Info - App: Loading MediaBrowser.Server.Startup.Common, Version=3.0.5607.2, Culture=neutral, PublicKeyToken=null
2015-05-26 19:10:46.1011 Info - SqliteUserRepository: Sqlite 3.8.2 opening /var/lib/emby/data/users.db
2015-05-26 19:10:46.1651 Info - SqliteItemRepository: Sqlite 3.8.2 opening /var/lib/emby/data/chapters.db
2015-05-26 19:10:46.1701 Info - SqliteItemRepository: Sqlite 3.8.2 opening /var/lib/emby/data/mediainfo.db
2015-05-26 19:10:46.1803 Info - SqliteFileOrganizationRepository: Sqlite 3.8.2 opening /var/lib/emby/data/fileorganization.db
2015-05-26 19:10:46.1898 Info - AuthenticationRepository: Sqlite 3.8.2 opening /var/lib/emby/data/authentication.db
2015-05-26 19:10:46.2032 Info - SyncRepository: Sqlite 3.8.2 opening /var/lib/emby/data/sync14.db
2015-05-26 19:10:46.2619 Info - ImageMagick: ImageMagick version: ImageMagick 6.9.0-10 Q8 x86_64 2015-05-11 http://www.imagemagick.org
2015-05-26 19:10:46.3008 Info - ImageProcessor: ImageProcessor started with 4 max concurrent image processes
2015-05-26 19:10:46.3577 Info - App: FFMpeg: /var/lib/emby/ffmpeg/20150331/ffmpeg
2015-05-26 19:10:46.3577 Info - App: FFProbe: /var/lib/emby/ffmpeg/20150331/ffprobe
2015-05-26 19:10:46.3720 Info - ActivityRepository: Sqlite 3.8.2 opening /var/lib/emby/data/activitylog.db
2015-05-26 19:10:46.3802 Info - SqliteDisplayPreferencesRepository: Sqlite 3.8.2 opening /var/lib/emby/data/displaypreferences.db
2015-05-26 19:10:46.3865 Info - SqliteItemRepository: Sqlite 3.8.2 opening /var/lib/emby/data/library.db
2015-05-26 19:10:46.3935 Info - SqliteProviderInfoRepository: Sqlite 3.8.2 opening /var/lib/emby/data/refreshinfo.db
2015-05-26 19:10:46.4011 Info - SqliteUserDataRepository: Sqlite 3.8.2 opening /var/lib/emby/data/userdata_v2.db
2015-05-26 19:10:46.4073 Info - SqliteNotificationsRepository: Sqlite 3.8.2 opening /var/lib/emby/data/notifications.db
2015-05-26 19:10:46.6625 Info - HttpServer: Calling ServiceStack AppHost.Init
2015-05-26 19:10:49.1978 Info - ServiceStackHost: Initializing Application took 2974.309ms
2015-05-26 19:10:49.2164 Info - ServerManager: Loading Http Server
2015-05-26 19:10:49.2189 Info - HttpServer: Adding HttpListener prefix http://+:8096/
2015-05-26 19:10:49.2189 Info - HttpServer: Adding HttpListener prefix https://+:8920/
2015-05-26 19:10:49.2297 Info - HttpServer: attempting to load pfx: /var/lib/emby/ssl/cert_9c31b7884ea5475c8687970fc5996297.pfx
2015-05-26 19:10:49.5625 Info - EmbyTV: Directory is: /var/lib/emby/plugins/EmbyTV
2015-05-26 19:10:49.5685 Info - EmbyTV: Creating a TunerHost list: 1
2015-05-26 19:10:49.5692 Info - EmbyTV: Creating a TunerHost of type: HdHomeRun
2015-05-26 19:10:49.5702 Info - EmbyTV: Adding variable: Url with value of 10.248.54.102
2015-05-26 19:10:49.5702 Info - EmbyTV: Added: Url with value of 10.248.54.102
2015-05-26 19:10:49.5702 Info - EmbyTV: Adding variable: OnlyFavorites with value of false
2015-05-26 19:10:49.5702 Info - EmbyTV: Added: OnlyFavorites with value of false
2015-05-26 19:10:49.5702 Info - EmbyTV: Done Creating Tuner
2015-05-26 19:10:49.5891 Info - App: HttpClientManager GET: http://10.248.54.102/
2015-05-26 19:10:49.6601 Error - Main: UnhandledException
*** Error Report ***
Version: 3.0.5607.2
Command line: /opt/emby/MediaBrowser.Server.Mono.exe -programdata /var/lib/emby
Operating system: Unix 4.0.4.40004
Processor count: 2
64-Bit OS: True
64-Bit Process: True
Program data path: /var/lib/emby
Mono: 4.0.1 (tarball Tue May 12 18:04:15 UTC 2015)
Application Path: /opt/emby/MediaBrowser.Server.Mono.exe
Document element did not appear.  Line 1, position 1.
System.Xml.XmlException
 at Mono.Xml2.XmlTextReader.Read () [0x00000] in <filename unknown>:0 
 at System.Xml.XmlTextReader.Read () [0x00000] in <filename unknown>:0 
 at System.Xml.XmlReader.MoveToContent () [0x00000] in <filename unknown>:0 
 at System.Xml.Serialization.XmlSerializationReaderInterpreter.ReadRoot () [0x00000] in <filename unknown>:0 
 at System.Xml.Serialization.XmlSerializer.Deserialize (System.Xml.Serialization.XmlSerializationReader reader) [0x00000] in <filename unknown>:0 
--- End of stack trace from previous location where exception was thrown ---
 at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw () [0x00000] in <filename unknown>:0 
 at System.Runtime.CompilerServices.AsyncMethodBuilderCore.<ThrowAsync>m__1 (System.Object state) [0x00000] in <filename unknown>:0 
 
2015-05-26 19:10:49.6991 Info - App: Core startup complete

 

Any thoughts?

 

Thanks!

Share this post


Link to post
Share on other sites
Luke

possibly plugin related. remove all plugin dll's. this is not the same issue as what this topic is about.

Share this post


Link to post
Share on other sites
ryantiger658

That did the trick! Thanks Luke! What was it about the logs that told you that it was plugin related? I am just curious so that I can better debug in the future.

 

Edit-Update: I removed the dll's, then reinstalled the plugins and it is crashing again with the same error. Aside from removing one dll at a time is there any way to know which plugin is causing the problem?

 

Edit 2: It is a problem with the EmbyTV plugin, I will move over there and see if I can help them with debugging.

 

Thanks!

Edited by ryantiger658

Share this post


Link to post
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now
Sign in to follow this  

×
×
  • Create New...