Jump to content

CentOS Crashing on 3.0.5785.0


djotaku

Recommended Posts

Sorry to seem to be spamming the topic, but I just noticed something else:

 

I couldn't ever see anything in the most recent log that revealed the reason for the crash, but if I look in the penultimate log....I see this at the end (I can't attach the whole log as it's 40MB)

2016-01-10 17:37:53.8991 Info MediaEncoder: Killing ffmpeg process
2016-01-10 17:37:53.8991 Error MediaEncoder: Error sending q command to process
2016-01-10 17:37:53.8991 Error MediaEncoder: Error in WaitForExit
2016-01-10 17:37:53.8991 Error MediaEncoder: Error killing process
    *** Error Report ***
    Version: 3.0.5785.0
    Command line: /usr/lib/emby-server/bin/MediaBrowser.Server.Mono.exe -programdata /var/lib/emby-server -ffmpeg /bin/ffmpeg -ffprobe /bin/ffprobe -restartpath /usr/lib/emby-server/restart.sh
    Operating system: Unix 3.10.0.229
    Processor count: 1
    64-Bit OS: True
    64-Bit Process: True
    Program data path: /var/lib/emby-server
    Mono: 4.2.1 (Stable 4.2.1.102/6dd2d0d Mon Dec 28 00:10:17 UTC 2015)
    Application Path: /usr/lib/emby-server/bin/MediaBrowser.Server.Mono.exe
    No process to kill.
    System.SystemException
      at System.Diagnostics.Process.Close (Int32 signo) <0x41494790 + 0x000ab> in <filename unknown>:0
      at System.Diagnostics.Process.Kill () <0x41494760 + 0x00013> in <filename unknown>:0
      at (wrapper remoting-invoke-with-check) System.Diagnostics.Process:Kill ()
      at MediaBrowser.MediaEncoding.Encoder.MediaEncoder.StopProcess (MediaBrowser.MediaEncoding.Encoder.ProcessWrapper process, Int32 waitTimeMs, Boolean enableForceKill) <0x41494000 + 0x001ff> in <filename unknown>:0
    
2016-01-10 17:37:53.8991 Info MediaEncoder: Killing ffmpeg process
2016-01-10 17:37:53.8991 Error MediaEncoder: Error sending q command to process
2016-01-10 17:37:53.8991 Error MediaEncoder: Error in WaitForExit
2016-01-10 17:37:53.8991 Error MediaEncoder: Error killing process
    *** Error Report ***
    Version: 3.0.5785.0
    Command line: /usr/lib/emby-server/bin/MediaBrowser.Server.Mono.exe -programdata /var/lib/emby-server -ffmpeg /bin/ffmpeg -ffprobe /bin/ffprobe -restartpath /usr/lib/emby-server/restart.sh
    Operating system: Unix 3.10.0.229
    Processor count: 1
    64-Bit OS: True
    64-Bit Process: True
    Program data path: /var/lib/emby-server
    Mono: 4.2.1 (Stable 4.2.1.102/6dd2d0d Mon Dec 28 00:10:17 UTC 2015)
    Application Path: /usr/lib/emby-server/bin/MediaBrowser.Server.Mono.exe
    No process to kill.
    System.SystemException
      at System.Diagnostics.Process.Close (Int32 signo) <0x41494790 + 0x000ab> in <filename unknown>:0
      at System.Diagnostics.Process.Kill () <0x41494760 + 0x00013> in <filename unknown>:0
      at (wrapper remoting-invoke-with-check) System.Diagnostics.Process:Kill ()
      at MediaBrowser.MediaEncoding.Encoder.MediaEncoder.StopProcess (MediaBrowser.MediaEncoding.Encoder.ProcessWrapper process, Int32 waitTimeMs, Boolean enableForceKill) <0x41494000 + 0x001ff> in <filename unknown>:0
    
2016-01-10 17:37:53.8991 Info App: Disposing SsdpHandler
2016-01-10 17:37:53.9021 Info App: Disposing ActivityRepository
2016-01-10 17:37:53.9025 Info App: Disposing MovieDbProvider
2016-01-10 17:37:53.9025 Info App: Disposing EmbyTV
2016-01-10 17:37:53.9060 Info App: Disposing OpenSubtitleDownloader
2016-01-10 17:37:53.9066 Info App: Disposing LinuxIsoManager
2016-01-10 17:37:53.9066 Info App: Disposing LinuxMount
2016-01-10 17:37:53.9066 Info App: Disposing ApiEntryPoint
2016-01-10 17:37:53.9076 Info App: Disposing ServerEntryPoint
2016-01-10 17:37:53.9086 Info App: Disposing ActivityLogEntryPoint
2016-01-10 17:37:53.9141 Info App: Disposing AutomaticRestartEntryPoint
2016-01-10 17:37:53.9141 Info App: Disposing ExternalPortForwarding
2016-01-10 17:37:53.9177 Info App: Disposing LibraryChangedNotifier
2016-01-10 17:37:53.9181 Info App: Disposing LoadRegistrations
2016-01-10 17:37:53.9181 Info App: Disposing Notifications
2016-01-10 17:37:53.9194 Info App: Disposing WebSocketNotifier
2016-01-10 17:37:53.9194 Info App: Disposing RefreshUsersMetadata
2016-01-10 17:37:53.9194 Info App: Disposing UsageEntryPoint
2016-01-10 17:37:53.9194 Info App: Disposing ConnectEntryPoint
2016-01-10 17:37:53.9194 Info App: Disposing UdpServerEntryPoint
2016-01-10 17:37:53.9210 Info App: Disposing ServerEventNotifier
2016-01-10 17:37:53.9249 Info App: Disposing UserDataChangeNotifier
2016-01-10 17:37:53.9253 Info App: Disposing LibraryMonitorStartup
2016-01-10 17:37:53.9253 Info App: Disposing EntryPoint
2016-01-10 17:37:53.9253 Info App: Disposing HdHomerunDiscovery
2016-01-10 17:37:53.9253 Info App: Disposing NewsEntryPoint
2016-01-10 17:37:53.9253 Info App: Disposing DlnaProfileEntryPoint
2016-01-10 17:37:53.9253 Info App: Disposing DlnaEntryPoint
2016-01-10 17:37:53.9302 Info App: Disposing EntryPoint
2016-01-10 17:37:53.9302 Info App: Disposing KeepServerAwake
2016-01-10 17:37:53.9302 Info App: Disposing StartupWizard

This would explain why we seem to be able to watch a movie and then it crashes.

 

We need the full log. Is it 40MB before or after compression? It's a text file, so it should compress really well.

Link to comment
Share on other sites

Later today I'll compress it & upload it - what do you prefer? tgz or zip?

 

As an aside to anyone else following this topic and reading this far - the change I made to the systemd file to make it restart is working well as a nice workaround until we figure out the reason for the crash. It allows my wife and kid to always find a working emby.

Link to comment
Share on other sites

Later today I'll compress it & upload it - what do you prefer? tgz or zip?

 

As an aside to anyone else following this topic and reading this far - the change I made to the systemd file to make it restart is working well as a nice workaround until we figure out the reason for the crash. It allows my wife and kid to always find a working emby.

 

Any of those format will be the same for me.

Link to comment
Share on other sites

After reviewing your log file I would try to disable the automatic subtitle fetching from OpenSubtitles.

 

I didn't have the time to review the log deeply but it seems that you are hitting the limit of subtitles that OpenSubtitles allows you to download, so the Emby task that grabs the subs starts to fail over and over again.

2016-01-10 17:32:34.3796 Info App: HttpClientManager POST: http://api.opensubtitles.org/xml-rpc
2016-01-10 17:32:34.5083 Error App: Error downloading subtitles
    *** Error Report ***
    Version: 3.0.5785.0
    Command line: /usr/lib/emby-server/bin/MediaBrowser.Server.Mono.exe -programdata /var/lib/emby-server -ffmpeg /bin/ffmpeg -ffprobe /bin/ffprobe -restartpath /usr/lib/emby-server/restart.sh
    Operating system: Unix 3.10.0.229
    Processor count: 1
    64-Bit OS: True
    64-Bit Process: True
    Program data path: /var/lib/emby-server
    Mono: 4.2.1 (Stable 4.2.1.102/6dd2d0d Mon Dec 28 00:10:17 UTC 2015)
    Application Path: /usr/lib/emby-server/bin/MediaBrowser.Server.Mono.exe
    Subtitle with Id 1953632029 was not found. Name: SubtitleDownload method response. Status: SubtitleDownload method response hold all expected values from server.. Message: 407 Download limit reached
    MediaBrowser.Common.Extensions.ResourceNotFoundException
      at MediaBrowser.Providers.Subtitles.OpenSubtitleDownloader+<GetSubtitlesInternal>c__async0.MoveNext () <0x412047f0 + 0x009e7> in <filename unknown>:0
    --- End of stack trace from previous location where exception was thrown ---
      at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw () <0x411a86b0 + 0x0002b> in <filename unknown>:0
      at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (System.Threading.Tasks.Task task) <0x411a8490 + 0x000c7> in <filename unknown>:0
      at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Threading.Tasks.Task task) <0x411a8410 + 0x0006b> in <filename unknown>:0
      at System.Runtime.CompilerServices.TaskAwaiter.ValidateEnd (System.Threading.Tasks.Task task) <0x40c35410 + 0x0003b> in <filename unknown>:0
      at System.Runtime.CompilerServices.ConfiguredTaskAwaitable`1+ConfiguredTaskAwaiter[TResult].GetResult () <0x40c6d0e0 + 0x00017> in <filename unknown>:0
      at MediaBrowser.Providers.Subtitles.SubtitleManager+<DownloadSubtitles>c__async1.MoveNext () <0x41202c30 + 0x00363> in <filename unknown>:0
    --- End of stack trace from previous location where exception was thrown ---
      at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw () <0x411a86b0 + 0x0002b> in <filename unknown>:0
      at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (System.Threading.Tasks.Task task) <0x411a8490 + 0x000c7> in <filename unknown>:0
      at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Threading.Tasks.Task task) <0x411a8410 + 0x0006b> in <filename unknown>:0
      at System.Runtime.CompilerServices.TaskAwaiter.ValidateEnd (System.Threading.Tasks.Task task) <0x40c35410 + 0x0003b> in <filename unknown>:0
      at System.Runtime.CompilerServices.ConfiguredTaskAwaitable+ConfiguredTaskAwaiter.GetResult () <0x40c353e0 + 0x00013> in <filename unknown>:0
      at MediaBrowser.Providers.MediaInfo.SubtitleDownloader+<DownloadSubtitles>c__async1.MoveNext () <0x411dfe50 + 0x01043> in <filename unknown>:0

Note the: Message: 407 Download limit reached

 

 

The server try to cancel the task but it can't and decides to shutdown itself.

2016-01-10 17:37:48.7113 Info App: Download missing subtitles: Timed out waiting for task to stop

To summarize: try to disable scheduled task that grabs the subs from OpenSubtitles and see if the problem goes away. If it does, then consider checking with OpenSubtitles how can you raise the limit of subtitles you can download per day (probably you will need a premium account or something).

Edited by fc7
Link to comment
Share on other sites

I'll turn it off, but it's a red herring. It was happening before I started grabbing from opensubtitles.

 

Ok. So start by turning that off. This should give us a much cleaner log (without all the subtitles errors) and once the problem happens again upload the new log and I will check it again. What do you think?

Link to comment
Share on other sites

Yup. Just turned it off. I'll start with the 542PM log for today and submit any more logs that are made today before I end my day.

Link to comment
Share on other sites

I'm attaching my logs. If you know naming system that lets you know which are the most recent, you can filter that out. If not, I'll have to upload the just most recent ones tomorrow.attachicon.gifserver-63587925593.txt.tar.gz

 

Did Emby crash again?

 

If yes, please post the time it crashed, at least approximately.

 

Edit: sorry but I can't review so many log files to find the latests logs. Please only upload the latest log, after the server is crashing otherwise I will not analyze them. Thanks.

Edited by fc7
Link to comment
Share on other sites

Sorry about. Yes, it crashed about 5-7 times. That's why there were so many logs. I'll try again when I get home today to grab the last two.

Link to comment
Share on other sites

Ok, so there was a crash because this is what I get when I interrogate the status:

# systemctl status emby-server
● emby-server.service - Emby Media Server
   Loaded: loaded (/usr/lib/systemd/system/emby-server.service; disabled; vendor preset: disabled)
   Active: active (running) since Wed 2016-01-13 16:12:49 EST; 1h 4min ago
  Process: 2117 ExecStopPost=/usr/lib/emby-server/emby-server.sh clear (code=exited, status=0/SUCCESS)
 Main PID: 2131 (su)
   CGroup: /system.slice/emby-server.service
           ‣ 2131 su -s /bin/sh -c exec "$0" "$@" emby -- env MONO_THREADS_PER_CPU=500 MONO_GC_PARAMS=nursery-size=128m LD_LIBRARY_PATH=/usr/lib/emby-server/lib64 /usr/bin/mono-sgen --optimize=all /usr/lib/emby...

Jan 13 16:12:49 babyluigi.mushroomkingdom systemd[1]: Started Emby Media Server.
Jan 13 16:12:49 babyluigi.mushroomkingdom systemd[1]: Starting Emby Media Server...
Jan 13 16:12:49 babyluigi.mushroomkingdom su[2131]: (to emby) root on none

And it should have been running for days now.

 

Here are the last two logs.


server-63588293647.txtserver-63588298370.txt

Link to comment
Share on other sites

Ok, so there was a crash because this is what I get when I interrogate the status:

# systemctl status emby-server
● emby-server.service - Emby Media Server
   Loaded: loaded (/usr/lib/systemd/system/emby-server.service; disabled; vendor preset: disabled)
   Active: active (running) since Wed 2016-01-13 16:12:49 EST; 1h 4min ago
  Process: 2117 ExecStopPost=/usr/lib/emby-server/emby-server.sh clear (code=exited, status=0/SUCCESS)
 Main PID: 2131 (su)
   CGroup: /system.slice/emby-server.service
           ‣ 2131 su -s /bin/sh -c exec "$0" "$@" emby -- env MONO_THREADS_PER_CPU=500 MONO_GC_PARAMS=nursery-size=128m LD_LIBRARY_PATH=/usr/lib/emby-server/lib64 /usr/bin/mono-sgen --optimize=all /usr/lib/emby...

Jan 13 16:12:49 babyluigi.mushroomkingdom systemd[1]: Started Emby Media Server.
Jan 13 16:12:49 babyluigi.mushroomkingdom systemd[1]: Starting Emby Media Server...
Jan 13 16:12:49 babyluigi.mushroomkingdom su[2131]: (to emby) root on none

And it should have been running for days now.

 

Here are the last two logs.

attachicon.gifserver-63588293647.txtattachicon.gifserver-63588298370.txt

 

I checked both logs but I can't find any evidence that suggests that the server crashed or had any important problem. Having said that, maybe these are not the correct logs (I think these are the logfiles after the server last restart but we need the previous ones that may include the problem).

 

What I would suggest you is to:

  1. Stop the Emby server service
  2. Rollback the modification you made to the service file to automatically restart the service
  3. Start the Emby server service. This time write down the date and time when the server is starting.

If it crashes again, then grab all the log files between the date the server started until the last one available (that should include the crash).

 

Once we have this information, post it here and you can reconfigure the service again to autorestart if you wish.

Edited by fc7
Link to comment
Share on other sites

OK, last night at 8:41PM I started the server. This was the status before I retired for the night:

 

systemctl status emby-server

● emby-server.service - Emby Media Server
   Loaded: loaded (/usr/lib/systemd/system/emby-server.service; disabled; vendor preset: disabled)
   Active: active (running) since Fri 2016-01-15 20:41:41 EST; 3h 6min ago
  Process: 9443 ExecStopPost=/usr/lib/emby-server/emby-server.sh clear (code=exited, status=0/SUCCESS)
 Main PID: 9463 (su)
   CGroup: /system.slice/emby-server.service
           ‣ 9463 su -s /bin/sh -c exec "$0" "$@" emby -- env MONO_THREADS_PER_CPU=500 MONO_GC_PARAMS=nursery-size=128m LD_LIBRARY_PATH=/usr/lib/emby-server/lib64 /usr/bin/mono-sgen --optimize=all /usr/lib/emby...

Jan 15 20:41:41 babyluigi.mushroomkingdom systemd[1]: Started Emby Media Server.
Jan 15 20:41:41 babyluigi.mushroomkingdom systemd[1]: Starting Emby Media Server...
Jan 15 20:41:41 babyluigi.mushroomkingdom su[9463]: (to emby) root on none

This morning, around 730AM my wife tried to watch on Kodi and it wouldn't connect. My android app would not connect.

 

Here is the current status:

● emby-server.service - Emby Media Server
   Loaded: loaded (/usr/lib/systemd/system/emby-server.service; disabled; vendor preset: disabled)
   Active: failed (Result: exit-code) since Sat 2016-01-16 00:19:36 EST; 7h ago
  Process: 10059 ExecStopPost=/usr/lib/emby-server/emby-server.sh clear (code=exited, status=0/SUCCESS)
  Process: 9463 ExecStart=/usr/lib/emby-server/emby-server.sh start (code=exited, status=134)
 Main PID: 9463 (code=exited, status=134)

Jan 15 20:41:41 babyluigi.mushroomkingdom systemd[1]: Started Emby Media Server.
Jan 15 20:41:41 babyluigi.mushroomkingdom systemd[1]: Starting Emby Media Server...
Jan 15 20:41:41 babyluigi.mushroomkingdom su[9463]: (to emby) root on none
Jan 16 00:19:36 babyluigi.mushroomkingdom systemd[1]: emby-server.service: main process exited, code=exited, status=134/n/a
Jan 16 00:19:36 babyluigi.mushroomkingdom systemd[1]: Unit emby-server.service entered failed state.
Jan 16 00:19:36 babyluigi.mushroomkingdom systemd[1]: emby-server.service failed.

So it died about 45 minutes after I stopped checking if it was going to die.

 

Here are the logs.

server-63588482997.txtserver-63588487302.txtserver-63588499200.txttranscode-4f44c926-b073-4089-a9f2-0f85507d1d64.txttranscode-86cfa2dd-e5cc-4555-9203-fae3f83017bd.txt

 

This is the order they go in:

-rw-r--r--. 1 emby   250246 Jan 15 20:40 server-63588482997.txt
-rw-r--r--. 1 emby    30550 Jan 15 20:47 transcode-4f44c926-b073-4089-a9f2-0f85507d1d64.txt
-rw-r--r--. 1 emby    28325 Jan 15 21:14 transcode-86cfa2dd-e5cc-4555-9203-fae3f83017bd.txt
-rw-r--r--. 1 emby  1319735 Jan 16 00:00 server-63588487302.txt
-rw-r--r--. 1 emby    74745 Jan 16 00:19 server-63588499200.txt

 

 

 

 

 

Link to comment
Share on other sites

there's a lot of dlna traffic, so you could try turning off the dlna server and play to features in the server web interface under Dlna

Link to comment
Share on other sites

If there were any question about whether it had to do with the RAM, I boosted the RAM to 4GB today. Still crashed. I'm now going to turn off the DLNA - would I need to restart the server after that ? Or is it immediate?

Link to comment
Share on other sites

  • 2 weeks later...
drewbuchman

I am having this same issue on the same version. It only does this during idle not while playing items. My system specs are below

 

Centos7

Dell 1950 power edge

2 x 2.3 quad core xeon

64 gb ram

4991 tv episodes

1089 Movies

20000 songs 

Link to comment
Share on other sites

I am having this same issue on the same version. It only does this during idle not while playing items. My system specs are below

 

Centos7

Dell 1950 power edge

2 x 2.3 quad core xeon

64 gb ram

4991 tv episodes

1089 Movies

20000 songs 

 

Hi, in order to best help you, please see how to report a problem. Thanks.

  • Like 1
Link to comment
Share on other sites

I am having this same issue on the same version. It only does this during idle not while playing items. My system specs are below

 

Centos7

Dell 1950 power edge

2 x 2.3 quad core xeon

64 gb ram

4991 tv episodes

1089 Movies

20000 songs 

I guess now I don't feel like it's my fauilt for having a low-powered machine.

  • Like 1
Link to comment
Share on other sites

FingerlessGlovs

I've got the same issue. Fresh install and i just moved from using Ubuntu server.

 

Specs

6 core amd 1090t

12gb ram

Centos7

Kernel 4.4

200+ movies

2000+ TV epoisodes.

Latest emby from repo stable.

 

My dlna is also getting spamed in my logs. Mono is still running emby when it crashes/becomes unrespondise. Some times mono gets killed/terminated when it crashes

Edited by MrJonny
Link to comment
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now
×
×
  • Create New...