Server constantly crashing or hanging

[imagineverything 0]

Hi, I've been struggling with getting my emby system setup properly. I had a good setup with mediabrowser but since the emby branding, my system has become increasingly unstable. When I enter my server address remotely, more often than not, I receive no response and need to logon remotely via teamviewer to reboot emby.

 

I have a feeling it is related to the size of the library, since it keeps trying to rescan the library, which spans two drives and 6 TB.

 

I have uploaded all of my recently recorded emby server logs to google drive.

 

Maybe someone here can take a look and hopefully identify something simple that I am doing wrong.

 

https://drive.google.com/folderview?id=0B1LHQsfF7SllS3dqS2pqVkU2ZUk&usp=sharing

 

Thanks everyone for all the hard work and time invested in this project. It is a great product so far.

[spoopers 5]

Having the same problem, tried a fresh install but still happening. Getting the below just before crashing:

 

Access token is invalid or expired.

MediaBrowser.Controller.Net.SecurityException

  at MediaBrowser.Server.Implementations.HttpServer.Security.AuthService.ValidateSecurityToken(IServiceRequest request, String token)

  at MediaBrowser.Server.Implementations.HttpServer.Security.AuthService.ValidateUser(IServiceRequest request, IAuthenticationAttributes authAttribtues)

  at MediaBrowser.Server.Implementations.HttpServer.Security.AuthService.Authenticate(IServiceRequest request, IAuthenticationAttributes authAttribtues)

  at MediaBrowser.Controller.Net.AuthenticatedAttribute.RequestFilter(IRequest request, IResponse response, Object requestDto)

  at ServiceStack.ServiceStackHost.ApplyRequestFiltersSingle(IRequest req, IResponse res, Object requestDto)

  at ServiceStack.ServiceStackHost.ApplyRequestFilters(IRequest req, IResponse res, Object requestDto)

  at ServiceStack.Host.RestHandler.ProcessRequestAsync(IRequest httpReq, IResponse httpRes, String operationName)

 

2015-12-07 08:08:46.0219 Info - HttpServer: HTTP Response 401 to 78.17.232.208. Time: 21ms. http://xxxxxxxxxxx/mediabrowser/Users/xxxxxxxxxx

2015-12-07 08:08:46.2063 Info - HttpServer: HTTP GET http://1xxxxxxxx/mediabrowser/Users/Public?format=json. UserAgent: python-requests/2.8.1

2015-12-07 08:08:46.2220 Info - HttpServer: HTTP Response 200 to 78.17.232.208. Time: 16ms. http://xxxxxxxxxxxx/mediabrowser/Users/Public?format=json

 

Edit:

 

Scratch that, I have restarted it a few times and each time it crashes with a different error, crash can happen within a few seconds or even up to a few hours. I'd prefer not post full server logs here but can zip them to you if required.

 

Any way to roll back to a previous release until this is resolved?

Edited December 7, 2015 by shaneypoopers

[Ian-Highlander 145]

Was going to raise an issue this morning with the same problem.

 

Emby works fine for an unspecified random amount of time, can be minutes or hours although while in use always seems perfect, we can watch things for hours on end with no issues at all.

 

It seems to be when it's left to idle for any length of time (although weirdly it's not every single time it's left to idle) it just starts consuming processor usage. Every time when this happens, the processor is sitting on between 13% and 27% usage with no scheduled tasks running and at that point you can hit the management interface, but none of the clients can get anything including the web client.

 

Log file from the last time this happened (Sunday morning) attached to this post.

server-63584956799.zip

[Tranquil 93]

@Ian-Highlander:

 

Pls try to disable your TV Theme Songs Addon. Regarding to the logs, there is a problem with it. Maybe it is causing your crashes.

Last days there was an Update of some of these plugins, maybe something went wrong.

 

@emagineverything:

 

It seems that your system is running on low memory, can it be? There are "out of memory" messages coming up in for logfile.

Furthermore, there are messages in it that let me assume, that there is some kind of firewall or other software that touches your network sockets. 

[spoopers 5]

Hi,
 
Switched on debug logging, see link.

 

https://drive.google.com/file/d/0Bw5xzi4uglNfc3EwYmo4MWJ0VWM/view?usp=sharing

Edited December 7, 2015 by shaneypoopers

[Ian-Highlander 145]

@Ian-Highlander:

 

Pls try to disable your TV Theme Songs Addon. Regarding to the logs, there is a problem with it. Maybe it is causing your crashes.

Last days there was an Update of some of these plugins, maybe something went wrong.

 

@emagineverything:

 

It seems that your system is running on low memory, can it be? There are "out of memory" messages coming up in for logfile.

Furthermore, there are messages in it that let me assume, that there is some kind of firewall or other software that touches your network sockets. 

Hi @@Tranquil

 

Thanks for looking, but I don't have the TV Theme Songs Addon plugin installed and never have. The ONLY plugin I EVER install is the Auto box sets one. I did once install ServerWMC on one previous install, but this is a clean build I re-did from scratch a month or so ago and I can categorically say I've never installed any plugins on it other than the auto box sets.

Edited December 7, 2015 by Ian-Highlander

[Ian-Highlander 145]

It's actually getting worse (or maybe I'm just looking at it at a worse point), just did it again and it peaked at 70% processor usage on an 8 core i7 processor before I gave up and stopped the service again.

 

Screenshot shows nothing running on the scheduled tasks, but 63% processor usage. Nothing available on any of the clients, EMC just hangs trying to load and eventually times out and loads with the default "No Items", the web client doesn't even get that far, I just get the home bar across the top and nothing else. Restart the service and all is well again until next time which could be minutes or hours.

 

New log attached.

 

server-63585043199.zip

Edited December 7, 2015 by Ian-Highlander

[spoopers 5]

Crashing every 5 minutes now

[Luke 38818]

Try removing installed plugins and see if that makes a difference.

[Ian-Highlander 145]

Try removing installed plugins and see if that makes a difference.

The only plugin I have ever installed on mine is the Auto box sets one. Is it likely that's the cause of the problem given the errors in the logs?

Edited December 7, 2015 by Ian-Highlander

[Tranquil 93]

There are so many exceptions in your logfiles, I cant figure out what really is happening. Many timeouts of TheMovieDB and NULL IDs for some elements.

I would suggest to deinstall all addons and, if it did not work for you, remove and readd your library folders so that the complete index is recreated. 

[Ian-Highlander 145]

There are so many exceptions in your logfiles, I cant figure out what really is happening. Many timeouts of TheMovieDB and NULL IDs for some elements.

I would suggest to deinstall all addons and, if it did not work for you, remove and readd your library folders so that the complete index is recreated. 

So start from scratch effectively again having only just done it for the umpteenth time? I only re-built this a few weeks ago from scratch, complete clean install. All I've done is set the images setting to legacy, install the Auto box sets plugin and add my own custom icons, other than that it's completely out of the box stock, that's really enough to break the system is it? (I'm sure it can't be).

 

Whilst I am happy (ok not happy but prepared) to re-scan my media library in again for the gawd knows how many'th time, surely there has to be a better answer than that?

Edited December 7, 2015 by Ian-Highlander

[speechles 1976]

@@Ian-Highlander

 

It is people causing your issue. Look at your log first posted. The server fires off 40 or so people requests in async. Right away comes the error after this. The error given is 500 (internal server error). So maybe tmdb was having issues? Maybe tmdb isnt issuing 429 (too many requests) anymore? So emby doesnt know to end the task, and stop hammering their server? It sees the 500 and assumes it can retry. Not a 429 telling it give up. Has this happened any other time that you have logs from?

Edited December 7, 2015 by speechles

[Ian-Highlander 145]

@@Ian-Highlander

 

It is people causing your issue. Look at your log first posted. The server fires off 40 or so people requests in async. Right away comes the error after this. The error given is 500 (internal server error). So maybe tmdb was having issues? Maybe tmdb isnt issuing 429 (too many requests) anymore? So emby doesnt know to end the task, and stop hammering their server? It sees the 500 and assumes it can retry. Not a 429 telling it give up. Has this happened any other time that you have logs from?

Thanks @@speechles funnily enough the people issue is another thread on the forums I've been taking part in seperately too.

 

There are two different logs of mine already in this thread, but for more completeness, I've attached the last five log files from my system (including one that's 128MB when it's uncompressed  ).

 

This has been going on for a while now and I can use the tmdb website normally with no issues at all. I assume Emby uses some sort of api connection to a different link though?

logs.zip

[speechles 1976]

2015-12-07 00:00:08.2955 Info - HttpServer: HTTP GET http://192.168.1.103:8096/mediabrowser/Users/9ee9d25b-37e6-4b2b-8c77-00420fec5c2e/Items?ParentId=7feaaec39451885c7ddd676f7c588291&sortBy=&CollapseBoxSetItems=False&fields=Overview,Path,ParentId,DisplayPreferencesId,DateCreated,DateLastMediaAdded,Metascore,MediaStreams,SortName,Taglines,MediaSources,SpecialEpisodeNumbers&ImageTypes=&recursive=False&genres=&Ids=&studioIds=&ExcludeItemTypes=&IncludeItemTypes=&ExcludeLocationTypes=&personIds=&personTypes=&years=&format=Json. UserAgent: 2015-12-07 00:00:08.2955 Error - ServerManager: SessionWebSocketListener failed processing WebSocket message Context 	*** Error Report *** 	Version: 3.0.5781.4 	Command line: C:\Users\iana\AppData\Roaming\Emby-Server\system\MediaBrowser.ServerApplication.exe -service 	Operating system: Microsoft Windows NT 6.1.7601 Service Pack 1 	Processor count: 8 	64-Bit OS: True 	64-Bit Process: False 	Program data path: C:\Users\iana\AppData\Roaming\Emby-Server 	Application Path: C:\Users\iana\AppData\Roaming\Emby-Server\system\MediaBrowser.ServerApplication.exe 	Value cannot be null. 	Parameter name: id 	System.ArgumentNullException 	 at MediaBrowser.Server.Implementations.Library.LibraryManager.GetItemById(Guid id) 	 at MediaBrowser.Server.Implementations.Session.SessionManager.ReportNowViewingItem(String sessionId, String itemId) 	 at MediaBrowser.Server.Implementations.Session.SessionWebSocketListener.ProcessContextMessage(WebSocketMessageInfo message) 	 at MediaBrowser.Server.Implementations.Session.SessionWebSocketListener.ProcessMessage(WebSocketMessageInfo message) 	 at MediaBrowser.Server.Implementations.ServerManager.ServerManager.<>c__DisplayClass3.<>c__DisplayClass5.<<ProcessWebSocketMessageReceived>b__2>d__7.MoveNext() 

Besides people, which is the main problem with your library, there is this. Do you know what item belongs to this guid ( 7feaaec39451885c7ddd676f7c588291 )? This is the one causing your initial errors after every load.

 

Sent from my Nexus 7 using Tapatalk

[Ian-Highlander 145]

2015-12-07 00:00:08.2955 Info - HttpServer: HTTP GET http://192.168.1.103:8096/mediabrowser/Users/9ee9d25b-37e6-4b2b-8c77-00420fec5c2e/Items?ParentId=7feaaec39451885c7ddd676f7c588291&sortBy=&CollapseBoxSetItems=False&fields=Overview,Path,ParentId,DisplayPreferencesId,DateCreated,DateLastMediaAdded,Metascore,MediaStreams,SortName,Taglines,MediaSources,SpecialEpisodeNumbers&ImageTypes=&recursive=False&genres=&Ids=&studioIds=&ExcludeItemTypes=&IncludeItemTypes=&ExcludeLocationTypes=&personIds=&personTypes=&years=&format=Json. UserAgent: 2015-12-07 00:00:08.2955 Error - ServerManager: SessionWebSocketListener failed processing WebSocket message Context 	*** Error Report *** 	Version: 3.0.5781.4 	Command line: C:\Users\iana\AppData\Roaming\Emby-Server\system\MediaBrowser.ServerApplication.exe -service 	Operating system: Microsoft Windows NT 6.1.7601 Service Pack 1 	Processor count: 8 	64-Bit OS: True 	64-Bit Process: False 	Program data path: C:\Users\iana\AppData\Roaming\Emby-Server 	Application Path: C:\Users\iana\AppData\Roaming\Emby-Server\system\MediaBrowser.ServerApplication.exe 	Value cannot be null. 	Parameter name: id 	System.ArgumentNullException 	 at MediaBrowser.Server.Implementations.Library.LibraryManager.GetItemById(Guid id) 	 at MediaBrowser.Server.Implementations.Session.SessionManager.ReportNowViewingItem(String sessionId, String itemId) 	 at MediaBrowser.Server.Implementations.Session.SessionWebSocketListener.ProcessContextMessage(WebSocketMessageInfo message) 	 at MediaBrowser.Server.Implementations.Session.SessionWebSocketListener.ProcessMessage(WebSocketMessageInfo message) 	 at MediaBrowser.Server.Implementations.ServerManager.ServerManager.<>c__DisplayClass3.<>c__DisplayClass5.<<ProcessWebSocketMessageReceived>b__2>d__7.MoveNext() 

Besides people, which is the main problem with your library, there is this. Do you know what item belongs to this guid ( 7feaaec39451885c7ddd676f7c588291 )? This is the one causing your initial errors after every load.

 

Sent from my Nexus 7 using Tapatalk

 

That's interesting, that's "Weird Al" Yankovic and only yesterday I noticed that in the album artists section in EMC the folder and metadata was there but it was empty when you went into it (none of the music was there) when all others were populated although interestingly the web client shows the media fine. I wonder if the quotation marks are causing the issue?

 

I'll dig around a bit round that artist and see if there's anything obvious causing the issue.

 

That's definitely not all the problems as you say, but at least I know where to start looking for one of the issues. Thanks for your help so far

 

 

Edit: I've now edited everything to do with "Weird Al" Yankovic to remove the quotation marks (even though that is how he has it), have made sure the correct musicbrainz IDs are assigned to everything (most were) and have locked all metadata for him, see if that gets rid of that one particular error. Doesn't solve the main problem, but it's a step in the right direction.

Edited December 7, 2015 by Ian-Highlander

[Ian-Highlander 145]

Definitely looks like it's the people scan that's causing the issue. I triggered it manually to see what would happen, scan finished in the management interface in 47 seconds, however the processor maxed out (and I mean maxed out) and stayed there until eventually I gave up and restarted the service again.

 

That's maxing out all 8 cores on an i7 processor after only being restarted earlier today.

 

Log file filling up with thousands of these:

 

2015-12-07 19:18:14.3745 Error - App: Error in TheMovieDb
*** Error Report ***
Version: 3.0.5781.4
Command line: C:\Users\iana\AppData\Roaming\Emby-Server\system\MediaBrowser.ServerApplication.exe -service
Operating system: Microsoft Windows NT 6.1.7601 Service Pack 1
Processor count: 8
64-Bit OS: True
64-Bit Process: False
Program data path: C:\Users\iana\AppData\Roaming\Emby-Server
Application Path: C:\Users\iana\AppData\Roaming\Emby-Server\system\MediaBrowser.ServerApplication.exe
Cancelling connection to http://api.themoviedb.org/3/search/person?api_key=f6bd687ffa63cd282b6ff2c6877f2669&query=Caroline+Stefanie+Clay due to a previous timeout.
MediaBrowser.Model.Net.HttpException
  at MediaBrowser.Common.Implementations.HttpClientManager.HttpClientManager.<SendAsyncInternal>d__1b.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
  at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
  at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
  at MediaBrowser.Common.Implementations.HttpClientManager.HttpClientManager.<SendAsync>d__5.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
  at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
  at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
  at MediaBrowser.Common.Implementations.HttpClientManager.HttpClientManager.<Get>d__0.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
  at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
  at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
  at System.Runtime.CompilerServices.TaskAwaiter.ValidateEnd(Task task)
  at MediaBrowser.Providers.People.MovieDbPersonProvider.<GetSearchResults>d__6.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
  at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
  at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
  at MediaBrowser.Providers.People.MovieDbPersonProvider.<GetTmdbId>d__20.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
  at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
  at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
  at MediaBrowser.Providers.People.MovieDbPersonProvider.<GetMetadata>d__14.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
  at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
  at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
  at MediaBrowser.Providers.Manager.MetadataService`2.<ExecuteRemoteProviders>d__55.MoveNext()

Edited December 7, 2015 by Ian-Highlander

[imagineverything 0]

thanks for all the responses. I have 16GB of ram, with almost nothing else running...  here are updated server logs, with "people" turned off.

 

https://www.dropbox.com/sh/tlzw7ph1dssrye8/AAD4HqOHcw-HJ0NuGdKVWWJpa?dl=0 

[Luke 38818]

is there anything we should be looking for?

[Happy2Play 9140]

thanks for all the responses. I have 16GB of ram, with almost nothing else running...  here are updated server logs, with "people" turned off.

 

https://www.dropbox.com/sh/tlzw7ph1dssrye8/AAD4HqOHcw-HJ0NuGdKVWWJpa?dl=0 

How did you turn "People" off?

 

 Line 19990: 2015-12-08 03:00:00.0071 Info App: DailyTrigger fired for task: Refresh people

 Line 19992: 2015-12-08 03:00:00.0071 Info App: Executing Refresh people

 Line 631509: 2015-12-08 03:30:25.5325 Info App: Refresh people Completed after 30 minute(s) and 25 seconds

[imagineverything 0]

How did you turn "People" off?

 

 Line 19990: 2015-12-08 03:00:00.0071 Info App: DailyTrigger fired for task: Refresh people

 Line 19992: 2015-12-08 03:00:00.0071 Info App: Executing Refresh people

 Line 631509: 2015-12-08 03:30:25.5325 Info App: Refresh people Completed after 30 minute(s) and 25 seconds

I turned it off on friday. Metadata seems to have an issue downloading and the library scan seems to get stuck still, at 18.7%

[Happy2Play 9140]

Only thing that stands out is 'System.OutOfMemoryException' thrown during your Refresh Guide task.

[JeremyFr79 228]

I was having all kinds of issues similar to those posted.  Turns out my metadata had become corrupted.  I wiped EMBY completely off my server, did a fresh install and rebuilt my Libraries, after less than 24 hours everything was back and haven't had an issue since.

[imagineverything 0]

 id rather not go down the wipe all metadata route.....

[JeremyFr79 228]

 id rather not go down the wipe all metadata route.....

 

Honestly it's not THAT bad, I have roughly 1500 Movies, and a Ton of T.V. took less than 4 hours to rebuild the entire library.