Jump to content

Server constantly crashing or hanging


imagineverything

Recommended Posts

imagineverything

Hi, I've been struggling with getting my emby system setup properly. I had a good setup with mediabrowser but since the emby branding, my system has become increasingly unstable. When I enter my server address remotely, more often than not, I receive no response and need to logon remotely via teamviewer to reboot emby.

 

I have a feeling it is related to the size of the library, since it keeps trying to rescan the library, which spans two drives and 6 TB.

 

I have uploaded all of my recently recorded emby server logs to google drive.

 

Maybe someone here can take a look and hopefully identify something simple that I am doing wrong.

 

https://drive.google.com/folderview?id=0B1LHQsfF7SllS3dqS2pqVkU2ZUk&usp=sharing

 

Thanks everyone for all the hard work and time invested in this project. It is a great product so far.

Link to post
Share on other sites

Having the same problem, tried a fresh install but still happening. Getting the below just before crashing:

 

Access token is invalid or expired.
MediaBrowser.Controller.Net.SecurityException
  at MediaBrowser.Server.Implementations.HttpServer.Security.AuthService.ValidateSecurityToken(IServiceRequest request, String token)
  at MediaBrowser.Server.Implementations.HttpServer.Security.AuthService.ValidateUser(IServiceRequest request, IAuthenticationAttributes authAttribtues)
  at MediaBrowser.Server.Implementations.HttpServer.Security.AuthService.Authenticate(IServiceRequest request, IAuthenticationAttributes authAttribtues)
  at MediaBrowser.Controller.Net.AuthenticatedAttribute.RequestFilter(IRequest request, IResponse response, Object requestDto)
  at ServiceStack.ServiceStackHost.ApplyRequestFiltersSingle(IRequest req, IResponse res, Object requestDto)
  at ServiceStack.ServiceStackHost.ApplyRequestFilters(IRequest req, IResponse res, Object requestDto)
  at ServiceStack.Host.RestHandler.ProcessRequestAsync(IRequest httpReq, IResponse httpRes, String operationName)
 
2015-12-07 08:08:46.0219 Info - HttpServer: HTTP Response 401 to 78.17.232.208. Time: 21ms. http://xxxxxxxxxxx/mediabrowser/Users/xxxxxxxxxx
2015-12-07 08:08:46.2063 Info - HttpServer: HTTP GET http://1xxxxxxxx/mediabrowser/Users/Public?format=json. UserAgent: python-requests/2.8.1
2015-12-07 08:08:46.2220 Info - HttpServer: HTTP Response 200 to 78.17.232.208. Time: 16ms. http://xxxxxxxxxxxx/mediabrowser/Users/Public?format=json
 
Edit:
 
Scratch that, I have restarted it a few times and each time it crashes with a different error, crash can happen within a few seconds or even up to a few hours. I'd prefer not post full server logs here but can zip them to you if required.
 
Any way to roll back to a previous release until this is resolved?
Edited by shaneypoopers
Link to post
Share on other sites
Ian-Highlander

Was going to raise an issue this morning with the same problem.

 

Emby works fine for an unspecified random amount of time, can be minutes or hours although while in use always seems perfect, we can watch things for hours on end with no issues at all.

 

It seems to be when it's left to idle for any length of time (although weirdly it's not every single time it's left to idle) it just starts consuming processor usage. Every time when this happens, the processor is sitting on between 13% and 27% usage with no scheduled tasks running and at that point you can hit the management interface, but none of the clients can get anything including the web client.

 

Log file from the last time this happened (Sunday morning) attached to this post.

server-63584956799.zip

Link to post
Share on other sites

@Ian-Highlander:

 

Pls try to disable your TV Theme Songs Addon. Regarding to the logs, there is a problem with it. Maybe it is causing your crashes.

Last days there was an Update of some of these plugins, maybe something went wrong.

 

@emagineverything:

 

It seems that your system is running on low memory, can it be? There are "out of memory" messages coming up in for logfile.

Furthermore, there are messages in it that let me assume, that there is some kind of firewall or other software that touches your network sockets. 

Link to post
Share on other sites
Ian-Highlander

@Ian-Highlander:

 

Pls try to disable your TV Theme Songs Addon. Regarding to the logs, there is a problem with it. Maybe it is causing your crashes.

Last days there was an Update of some of these plugins, maybe something went wrong.

 

@emagineverything:

 

It seems that your system is running on low memory, can it be? There are "out of memory" messages coming up in for logfile.

Furthermore, there are messages in it that let me assume, that there is some kind of firewall or other software that touches your network sockets. 

Hi @@Tranquil

 

Thanks for looking, but I don't have the TV Theme Songs Addon plugin installed and never have. The ONLY plugin I EVER install is the Auto box sets one. I did once install ServerWMC on one previous install, but this is a clean build I re-did from scratch a month or so ago and I can categorically say I've never installed any plugins on it other than the auto box sets.

Edited by Ian-Highlander
Link to post
Share on other sites
Ian-Highlander

It's actually getting worse (or maybe I'm just looking at it at a worse point), just did it again and it peaked at 70% processor usage on an 8 core i7 processor before I gave up and stopped the service again.

 

Screenshot shows nothing running on the scheduled tasks, but 63% processor usage. Nothing available on any of the clients, EMC just hangs trying to load and eventually times out and loads with the default "No Items", the web client doesn't even get that far, I just get the home bar across the top and nothing else. Restart the service and all is well again until next time which could be minutes or hours.

 

New log attached.

 

5665716eef470_Untitled.jpg

server-63585043199.zip

Edited by Ian-Highlander
Link to post
Share on other sites
Ian-Highlander

Try removing installed plugins and see if that makes a difference.

The only plugin I have ever installed on mine is the Auto box sets one. Is it likely that's the cause of the problem given the errors in the logs?

Edited by Ian-Highlander
Link to post
Share on other sites

There are so many exceptions in your logfiles, I cant figure out what really is happening. Many timeouts of TheMovieDB and NULL IDs for some elements.

I would suggest to deinstall all addons and, if it did not work for you, remove and readd your library folders so that the complete index is recreated. 

Link to post
Share on other sites
Ian-Highlander

There are so many exceptions in your logfiles, I cant figure out what really is happening. Many timeouts of TheMovieDB and NULL IDs for some elements.

I would suggest to deinstall all addons and, if it did not work for you, remove and readd your library folders so that the complete index is recreated. 

So start from scratch effectively again having only just done it for the umpteenth time? I only re-built this a few weeks ago from scratch, complete clean install. All I've done is set the images setting to legacy, install the Auto box sets plugin and add my own custom icons, other than that it's completely out of the box stock, that's really enough to break the system is it? (I'm sure it can't be).

 

Whilst I am happy (ok not happy but prepared) to re-scan my media library in again for the gawd knows how many'th time, surely there has to be a better answer than that?

Edited by Ian-Highlander
Link to post
Share on other sites
speechles

@@Ian-Highlander

 

It is people causing your issue. Look at your log first posted. The server fires off 40 or so people requests in async. Right away comes the error after this. The error given is 500 (internal server error). So maybe tmdb was having issues? Maybe tmdb isnt issuing 429 (too many requests) anymore? So emby doesnt know to end the task, and stop hammering their server? It sees the 500 and assumes it can retry. Not a 429 telling it give up. Has this happened any other time that you have logs from?

Edited by speechles
Link to post
Share on other sites
Ian-Highlander

@@Ian-Highlander

 

It is people causing your issue. Look at your log first posted. The server fires off 40 or so people requests in async. Right away comes the error after this. The error given is 500 (internal server error). So maybe tmdb was having issues? Maybe tmdb isnt issuing 429 (too many requests) anymore? So emby doesnt know to end the task, and stop hammering their server? It sees the 500 and assumes it can retry. Not a 429 telling it give up. Has this happened any other time that you have logs from?

Thanks @@speechles funnily enough the people issue is another thread on the forums I've been taking part in seperately too.

 

There are two different logs of mine already in this thread, but for more completeness, I've attached the last five log files from my system (including one that's 128MB when it's uncompressed  :blink: ).

 

This has been going on for a while now and I can use the tmdb website normally with no issues at all. I assume Emby uses some sort of api connection to a different link though?

logs.zip

Link to post
Share on other sites
speechles

2015-12-07 00:00:08.2955 Info - HttpServer: HTTP GET http://192.168.1.103:8096/mediabrowser/Users/9ee9d25b-37e6-4b2b-8c77-00420fec5c2e/Items?ParentId=7feaaec39451885c7ddd676f7c588291&sortBy=&CollapseBoxSetItems=False&fields=Overview,Path,ParentId,DisplayPreferencesId,DateCreated,DateLastMediaAdded,Metascore,MediaStreams,SortName,Taglines,MediaSources,SpecialEpisodeNumbers&ImageTypes=&recursive=False&genres=&Ids=&studioIds=&ExcludeItemTypes=&IncludeItemTypes=&ExcludeLocationTypes=&personIds=&personTypes=&years=&format=Json. UserAgent: 2015-12-07 00:00:08.2955 Error - ServerManager: SessionWebSocketListener failed processing WebSocket message Context 	*** Error Report *** 	Version: 3.0.5781.4 	Command line: C:\Users\iana\AppData\Roaming\Emby-Server\system\MediaBrowser.ServerApplication.exe -service 	Operating system: Microsoft Windows NT 6.1.7601 Service Pack 1 	Processor count: 8 	64-Bit OS: True 	64-Bit Process: False 	Program data path: C:\Users\iana\AppData\Roaming\Emby-Server 	Application Path: C:\Users\iana\AppData\Roaming\Emby-Server\system\MediaBrowser.ServerApplication.exe 	Value cannot be null. 	Parameter name: id 	System.ArgumentNullException 	 at MediaBrowser.Server.Implementations.Library.LibraryManager.GetItemById(Guid id) 	 at MediaBrowser.Server.Implementations.Session.SessionManager.ReportNowViewingItem(String sessionId, String itemId) 	 at MediaBrowser.Server.Implementations.Session.SessionWebSocketListener.ProcessContextMessage(WebSocketMessageInfo message) 	 at MediaBrowser.Server.Implementations.Session.SessionWebSocketListener.ProcessMessage(WebSocketMessageInfo message) 	 at MediaBrowser.Server.Implementations.ServerManager.ServerManager.<>c__DisplayClass3.<>c__DisplayClass5.<<ProcessWebSocketMessageReceived>b__2>d__7.MoveNext() 
Besides people, which is the main problem with your library, there is this. Do you know what item belongs to this guid ( 7feaaec39451885c7ddd676f7c588291 )? This is the one causing your initial errors after every load.

 

Sent from my Nexus 7 using Tapatalk

Link to post
Share on other sites
Ian-Highlander
2015-12-07 00:00:08.2955 Info - HttpServer: HTTP GET http://192.168.1.103:8096/mediabrowser/Users/9ee9d25b-37e6-4b2b-8c77-00420fec5c2e/Items?ParentId=7feaaec39451885c7ddd676f7c588291&sortBy=&CollapseBoxSetItems=False&fields=Overview,Path,ParentId,DisplayPreferencesId,DateCreated,DateLastMediaAdded,Metascore,MediaStreams,SortName,Taglines,MediaSources,SpecialEpisodeNumbers&ImageTypes=&recursive=False&genres=&Ids=&studioIds=&ExcludeItemTypes=&IncludeItemTypes=&ExcludeLocationTypes=&personIds=&personTypes=&years=&format=Json. UserAgent: 2015-12-07 00:00:08.2955 Error - ServerManager: SessionWebSocketListener failed processing WebSocket message Context 	*** Error Report *** 	Version: 3.0.5781.4 	Command line: C:\Users\iana\AppData\Roaming\Emby-Server\system\MediaBrowser.ServerApplication.exe -service 	Operating system: Microsoft Windows NT 6.1.7601 Service Pack 1 	Processor count: 8 	64-Bit OS: True 	64-Bit Process: False 	Program data path: C:\Users\iana\AppData\Roaming\Emby-Server 	Application Path: C:\Users\iana\AppData\Roaming\Emby-Server\system\MediaBrowser.ServerApplication.exe 	Value cannot be null. 	Parameter name: id 	System.ArgumentNullException 	 at MediaBrowser.Server.Implementations.Library.LibraryManager.GetItemById(Guid id) 	 at MediaBrowser.Server.Implementations.Session.SessionManager.ReportNowViewingItem(String sessionId, String itemId) 	 at MediaBrowser.Server.Implementations.Session.SessionWebSocketListener.ProcessContextMessage(WebSocketMessageInfo message) 	 at MediaBrowser.Server.Implementations.Session.SessionWebSocketListener.ProcessMessage(WebSocketMessageInfo message) 	 at MediaBrowser.Server.Implementations.ServerManager.ServerManager.<>c__DisplayClass3.<>c__DisplayClass5.<<ProcessWebSocketMessageReceived>b__2>d__7.MoveNext() 
Besides people, which is the main problem with your library, there is this. Do you know what item belongs to this guid ( 7feaaec39451885c7ddd676f7c588291 )? This is the one causing your initial errors after every load.

 

Sent from my Nexus 7 using Tapatalk

 

That's interesting, that's "Weird Al" Yankovic and only yesterday I noticed that in the album artists section in EMC the folder and metadata was there but it was empty when you went into it (none of the music was there) when all others were populated although interestingly the web client shows the media fine. I wonder if the quotation marks are causing the issue?

 

I'll dig around a bit round that artist and see if there's anything obvious causing the issue.

 

That's definitely not all the problems as you say, but at least I know where to start looking for one of the issues. Thanks for your help so far :)

 

 

Edit: I've now edited everything to do with "Weird Al" Yankovic to remove the quotation marks (even though that is how he has it), have made sure the correct musicbrainz IDs are assigned to everything (most were) and have locked all metadata for him, see if that gets rid of that one particular error. Doesn't solve the main problem, but it's a step in the right direction.

Edited by Ian-Highlander
Link to post
Share on other sites
Ian-Highlander

Definitely looks like it's the people scan that's causing the issue. I triggered it manually to see what would happen, scan finished in the management interface in 47 seconds, however the processor maxed out (and I mean maxed out) and stayed there until eventually I gave up and restarted the service again.

 

That's maxing out all 8 cores on an i7 processor after only being restarted earlier today.

 

Log file filling up with thousands of these:

 

2015-12-07 19:18:14.3745 Error - App: Error in TheMovieDb
*** Error Report ***
Version: 3.0.5781.4
Command line: C:\Users\iana\AppData\Roaming\Emby-Server\system\MediaBrowser.ServerApplication.exe -service
Operating system: Microsoft Windows NT 6.1.7601 Service Pack 1
Processor count: 8
64-Bit OS: True
64-Bit Process: False
Program data path: C:\Users\iana\AppData\Roaming\Emby-Server
Application Path: C:\Users\iana\AppData\Roaming\Emby-Server\system\MediaBrowser.ServerApplication.exe
Cancelling connection to http://api.themoviedb.org/3/search/person?api_key=f6bd687ffa63cd282b6ff2c6877f2669&query=Caroline+Stefanie+Clay due to a previous timeout.
MediaBrowser.Model.Net.HttpException
  at MediaBrowser.Common.Implementations.HttpClientManager.HttpClientManager.<SendAsyncInternal>d__1b.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
  at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
  at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
  at MediaBrowser.Common.Implementations.HttpClientManager.HttpClientManager.<SendAsync>d__5.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
  at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
  at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
  at MediaBrowser.Common.Implementations.HttpClientManager.HttpClientManager.<Get>d__0.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
  at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
  at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
  at System.Runtime.CompilerServices.TaskAwaiter.ValidateEnd(Task task)
  at MediaBrowser.Providers.People.MovieDbPersonProvider.<GetSearchResults>d__6.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
  at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
  at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
  at MediaBrowser.Providers.People.MovieDbPersonProvider.<GetTmdbId>d__20.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
  at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
  at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
  at MediaBrowser.Providers.People.MovieDbPersonProvider.<GetMetadata>d__14.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
  at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
  at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
  at MediaBrowser.Providers.Manager.MetadataService`2.<ExecuteRemoteProviders>d__55.MoveNext()

5665dd3b70c0c_2015120719_21_17kurgan_338

Edited by Ian-Highlander
Link to post
Share on other sites
Happy2Play

thanks for all the responses. I have 16GB of ram, with almost nothing else running...  here are updated server logs, with "people" turned off.

 

https://www.dropbox.com/sh/tlzw7ph1dssrye8/AAD4HqOHcw-HJ0NuGdKVWWJpa?dl=0 

How did you turn "People" off?

 

 Line 19990: 2015-12-08 03:00:00.0071 Info App: DailyTrigger fired for task: Refresh people

 Line 19992: 2015-12-08 03:00:00.0071 Info App: Executing Refresh people

 Line 631509: 2015-12-08 03:30:25.5325 Info App: Refresh people Completed after 30 minute(s) and 25 seconds

Link to post
Share on other sites
imagineverything

How did you turn "People" off?

 

 Line 19990: 2015-12-08 03:00:00.0071 Info App: DailyTrigger fired for task: Refresh people

 Line 19992: 2015-12-08 03:00:00.0071 Info App: Executing Refresh people

 Line 631509: 2015-12-08 03:30:25.5325 Info App: Refresh people Completed after 30 minute(s) and 25 seconds

I turned it off on friday. Metadata seems to have an issue downloading and the library scan seems to get stuck still, at 18.7%

Link to post
Share on other sites
Happy2Play

Only thing that stands out is 'System.OutOfMemoryException' thrown during your Refresh Guide task.

Link to post
Share on other sites
JeremyFr79

I was having all kinds of issues similar to those posted.  Turns out my metadata had become corrupted.  I wiped EMBY completely off my server, did a fresh install and rebuilt my Libraries, after less than 24 hours everything was back and haven't had an issue since.

Link to post
Share on other sites
JeremyFr79

 id rather not go down the wipe all metadata route.....

 

Honestly it's not THAT bad, I have roughly 1500 Movies, and a Ton of T.V. took less than 4 hours to rebuild the entire library.

Link to post
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now
×
×
  • Create New...