Emby server crashes when using search.

[Karl Blixt 8]

I just upgraded to 3.5.2.0 on my ubuntu 16.04 machine and wanted to check out if the new search feature that finds original title as well as the title was included. and to my delight when searching for "shaw" it very correctly found the movie shawshank redemption even though I've titled it "nyckeln till friheten" which is the swedish title for the movie.

 

However, I quickly found out that the server had crashed for some reason. By the looks of it it crashed after the search because but while fetching the posters because some of the posters for the results shows up while others are just unloaded black.

 

I've done some testing and I've found that the server dosen't crash as long as you search for movies that have the same title as original title. I'm not sure if it has to be exactly the same or just simmilar enough to not cause a crash for some reason. anyway, whenever the search result contains a movie with an altarantive title the server crashes.

 

I'll attach the logs of a session that experienced a crash but when looking into the server loggs there are a few entries called "unhandled_[hex based hash i presume].txt with the following exception:

 

System.IO.IOException: Unable to read data from the transport connection: Connection reset by peer. ---> System.Net.Sockets.SocketException: Connection reset by peer

--- End of inner exception stack trace ---
at System.Net.Sockets.Socket.AwaitableSocketAsyncEventArgs.ThrowException(SocketError error)
at System.Net.Sockets.Socket.AwaitableSocketAsyncEventArgs.GetResult(Int16 token)
at System.Net.FixedSizeReader.ReadPacketAsync(Stream transport, AsyncProtocolRequest request)
at System.Net.Security.SslState.ThrowIfExceptional()
at System.Net.Security.SslState.CheckThrow(Boolean authSuccessCheck, Boolean shutdownCheck)
at System.Net.Security.SslState.CheckOldKeyDecryptedData(Memory`1 buffer)
at System.Net.Security.SslState.HandleQueuedCallback(Object& queuedStateRequest)
at System.Net.Security.SslState.FinishHandshakeRead(Int32 newState)
at System.Net.Security.SslState.FinishHandshake(Exception e, AsyncProtocolRequest asyncRequest)
at System.Net.Security.SslState.RehandshakeCompleteCallback(IAsyncResult result)
at System.Net.LazyAsyncResult.Complete(IntPtr userToken)
at System.Net.LazyAsyncResult.ProtectedInvokeCallback(Object result, IntPtr userToken)
at System.Net.AsyncProtocolRequest.CompleteUserWithError(Exception e)
at System.Net.FixedSizeReader.ReadPacketAsync(Stream transport, AsyncProtocolRequest request)
at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state)
--- End of stack trace from previous location where exception was thrown ---
at System.Threading.ThreadPoolWorkQueue.Dispatch()
System.IO.IOException
at System.Net.Sockets.Socket.AwaitableSocketAsyncEventArgs.ThrowException(SocketError error)
at System.Net.Sockets.Socket.AwaitableSocketAsyncEventArgs.GetResult(Int16 token)
at System.Net.FixedSizeReader.ReadPacketAsync(Stream transport, AsyncProtocolRequest request)
at System.Net.Security.SslState.ThrowIfExceptional()
at System.Net.Security.SslState.CheckThrow(Boolean authSuccessCheck, Boolean shutdownCheck)
at System.Net.Security.SslState.CheckOldKeyDecryptedData(Memory`1 buffer)
at System.Net.Security.SslState.HandleQueuedCallback(Object& queuedStateRequest)
at System.Net.Security.SslState.FinishHandshakeRead(Int32 newState)
at System.Net.Security.SslState.FinishHandshake(Exception e, AsyncProtocolRequest asyncRequest)
at System.Net.Security.SslState.RehandshakeCompleteCallback(IAsyncResult result)
at System.Net.LazyAsyncResult.Complete(IntPtr userToken)
at System.Net.LazyAsyncResult.ProtectedInvokeCallback(Object result, IntPtr userToken)
at System.Net.AsyncProtocolRequest.CompleteUserWithError(Exception e)
at System.Net.FixedSizeReader.ReadPacketAsync(Stream transport, AsyncProtocolRequest request)
at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state)
--- End of stack trace from previous location where exception was thrown ---
at System.Threading.ThreadPoolWorkQueue.Dispatch()
InnerException: System.Net.Sockets.SocketException

System.Net.Sockets.SocketException (104): Connection reset by peer 

embyserver-63668706532.txt

[Luke 36992]

It's because you still have audiodb urls saved in your database unfortunately. audiodb has been removed in 3.5.2 but since you already have urls saved, what you'll need to do is remove your music library, run a scan, and then add it back.

[Karl Blixt 8]

Thank you, this resolved my issue.

[Luke 36992]

Thanks for the feedback.

[letterman 33]

I have the same problem.

 

Is removing and re-adding the ONLY solution???

Everything worked very well till 3.4.2

 

It is incredible that you change core engines in that deep way, without having a working library update tool.

 

Deleting and re-adding all libraries takes about 2 weeks and what about my collections??? The content is linked to the library and it will be lost after deleting and re-adding.

 

sorry, your suggestion is not satisfying.

Edited August 8, 2018 by letterman

[Luke 36992]

I have the same problem.

 

Is removing and re-adding the ONLY solution???

I have 39(!) libraries and everything worked very well till 3.4.2

 

It is incredible that you change core engines in that deep way, without having a working library update tool.

 

Deleting and re-adding all libraries takes about 2 weeks and what about my collections??? The content is linked to the library and it will be lost after deleting and re-adding.

 

sorry, your suggestion is not satisfying.

 

Can you please attach the emby server log? you can learn how to do that here:

https://emby.media/community/index.php?/topic/739-how-to-report-a-problem/

Thanks.

[letterman 33]

Attached my logs.

 

Crashing seems to happen after search clicking on search results, e.g. a song

Edited August 8, 2018 by letterman

[letterman 33]

Crash is not only after search, but also, when browsing.

Attached a 2nd crash log.

Edited August 8, 2018 by letterman

[evil_beast666 4]

Hi!

 

Sometimes, Emby service crash when trying to scan my music library.

 

This is because sometimes "The Audio DB" provider response with "null" value and I think you can control this behavior by "twice" lines of code

 

Real example:

2018-08-07 16:11:53.001 Info HttpClient: GET http://www.theaudiodb.com/api/v1/json/49jhsf8248yfahka89724011/album-mb.php?i=3a2714a8-12f7-4f00-ad0a-8cec1315ced2

Result:

{"album":null}

Workaround:

 

I've disabled temporarily this provider in my music library and work good by the moment

 

 

Attaching my complete log file anonymized.

 

Thx!

embyserver-63669255277.txt

[Luke 36992]

That's not going to cause a crash though. The server can handle that and continue onward.

[evil_beast666 4]

It should but sadly crash :-\

 

I've done more test:

@@evil_beast666

 

While I browse for web interface I've clicked in some music artist link and also crash with next docker log messages:

UnhandledException
UnhandledException
System.IO.IOException: Unable to read data from the transport connection: Connection reset by peer. ---> System.Net.Sockets.SocketException: Connection reset by peer
   --- End of inner exception stack trace ---
   at System.Net.Sockets.Socket.AwaitableSocketAsyncEventArgs.ThrowException(SocketError error)
   at System.Net.Sockets.Socket.AwaitableSocketAsyncEventArgs.GetResult(Int16 token)
   at System.Net.FixedSizeReader.ReadPacketAsync(Stream transport, AsyncProtocolRequest request)
   at System.Net.Security.SslState.ThrowIfExceptional()
   at System.Net.Security.SslState.CheckThrow(Boolean authSuccessCheck, Boolean shutdownCheck)
   at System.Net.Security.SslState.CheckOldKeyDecryptedData(Memory`1 buffer)
   at System.Net.Security.SslState.HandleQueuedCallback(Object& queuedStateRequest)
   at System.Net.Security.SslState.FinishHandshakeRead(Int32 newState)
   at System.Net.Security.SslState.FinishHandshake(Exception e, AsyncProtocolRequest asyncRequest)
   at System.Net.Security.SslState.RehandshakeCompleteCallback(IAsyncResult result)
   at System.Net.LazyAsyncResult.Complete(IntPtr userToken)
   at System.Net.LazyAsyncResult.ProtectedInvokeCallback(Object result, IntPtr userToken)
   at System.Net.AsyncProtocolRequest.CompleteUserWithError(Exception e)
   at System.Net.FixedSizeReader.ReadPacketAsync(Stream transport, AsyncProtocolRequest request)
   at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state)
--- End of stack trace from previous location where exception was thrown ---
   at System.Threading.ThreadPoolWorkQueue.Dispatch()
System.IO.IOException
   at System.Net.Sockets.Socket.AwaitableSocketAsyncEventArgs.ThrowException(SocketError error)
   at System.Net.Sockets.Socket.AwaitableSocketAsyncEventArgs.GetResult(Int16 token)
   at System.Net.FixedSizeReader.ReadPacketAsync(Stream transport, AsyncProtocolRequest request)
   at System.Net.Security.SslState.ThrowIfExceptional()
   at System.Net.Security.SslState.CheckThrow(Boolean authSuccessCheck, Boolean shutdownCheck)
   at System.Net.Security.SslState.CheckOldKeyDecryptedData(Memory`1 buffer)
   at System.Net.Security.SslState.HandleQueuedCallback(Object& queuedStateRequest)
   at System.Net.Security.SslState.FinishHandshakeRead(Int32 newState)
   at System.Net.Security.SslState.FinishHandshake(Exception e, AsyncProtocolRequest asyncRequest)
   at System.Net.Security.SslState.RehandshakeCompleteCallback(IAsyncResult result)
   at System.Net.LazyAsyncResult.Complete(IntPtr userToken)
   at System.Net.LazyAsyncResult.ProtectedInvokeCallback(Object result, IntPtr userToken)
   at System.Net.AsyncProtocolRequest.CompleteUserWithError(Exception e)
   at System.Net.FixedSizeReader.ReadPacketAsync(Stream transport, AsyncProtocolRequest request)
   at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state)
--- End of stack trace from previous location where exception was thrown ---
   at System.Threading.ThreadPoolWorkQueue.Dispatch()
InnerException: System.Net.Sockets.SocketException
System.Net.Sockets.SocketException (104): Connection reset by peer


System.IO.IOException: Unable to read data from the transport connection: Connection reset by peer. ---> System.Net.Sockets.SocketException: Connection reset by peer
   --- End of inner exception stack trace ---
   at System.Net.Sockets.Socket.AwaitableSocketAsyncEventArgs.ThrowException(SocketError error)
   at System.Net.Sockets.Socket.AwaitableSocketAsyncEventArgs.GetResult(Int16 token)
   at System.Net.FixedSizeReader.ReadPacketAsync(Stream transport, AsyncProtocolRequest request)
   at System.Net.Security.SslState.ThrowIfExceptional()
   at System.Net.Security.SslState.CheckThrow(Boolean authSuccessCheck, Boolean shutdownCheck)
   at System.Net.Security.SslState.CheckOldKeyDecryptedData(Memory`1 buffer)
   at System.Net.Security.SslState.HandleQueuedCallback(Object& queuedStateRequest)
   at System.Net.Security.SslState.FinishHandshakeRead(Int32 newState)
   at System.Net.Security.SslState.FinishHandshake(Exception e, AsyncProtocolRequest asyncRequest)
   at System.Net.Security.SslState.RehandshakeCompleteCallback(IAsyncResult result)
   at System.Net.LazyAsyncResult.Complete(IntPtr userToken)
   at System.Net.LazyAsyncResult.ProtectedInvokeCallback(Object result, IntPtr userToken)
   at System.Net.AsyncProtocolRequest.CompleteUserWithError(Exception e)
   at System.Net.FixedSizeReader.ReadPacketAsync(Stream transport, AsyncProtocolRequest request)
   at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state)
--- End of stack trace from previous location where exception was thrown ---
   at System.Threading.ThreadPoolWorkQueue.Dispatch()
System.IO.IOException
   at System.Net.Sockets.Socket.AwaitableSocketAsyncEventArgs.ThrowException(SocketError error)
   at System.Net.Sockets.Socket.AwaitableSocketAsyncEventArgs.GetResult(Int16 token)
   at System.Net.FixedSizeReader.ReadPacketAsync(Stream transport, AsyncProtocolRequest request)
   at System.Net.Security.SslState.ThrowIfExceptional()
   at System.Net.Security.SslState.CheckThrow(Boolean authSuccessCheck, Boolean shutdownCheck)
   at System.Net.Security.SslState.CheckOldKeyDecryptedData(Memory`1 buffer)
   at System.Net.Security.SslState.HandleQueuedCallback(Object& queuedStateRequest)
   at System.Net.Security.SslState.FinishHandshakeRead(Int32 newState)
   at System.Net.Security.SslState.FinishHandshake(Exception e, AsyncProtocolRequest asyncRequest)
   at System.Net.Security.SslState.RehandshakeCompleteCallback(IAsyncResult result)
   at System.Net.LazyAsyncResult.Complete(IntPtr userToken)
   at System.Net.LazyAsyncResult.ProtectedInvokeCallback(Object result, IntPtr userToken)
   at System.Net.AsyncProtocolRequest.CompleteUserWithError(Exception e)
   at System.Net.FixedSizeReader.ReadPacketAsync(Stream transport, AsyncProtocolRequest request)
   at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state)
--- End of stack trace from previous location where exception was thrown ---
   at System.Threading.ThreadPoolWorkQueue.Dispatch()
InnerException: System.Net.Sockets.SocketException
System.Net.Sockets.SocketException (104): Connection reset by peer


[cont-finish.d] executing container finish scripts...
[cont-finish.d] done.
[s6-finish] syncing disks.
[s6-finish] sending all processes the TERM signal.
[s6-finish] sending all processes the KILL signal and exiting.

[Happy2Play 8224]

Merging topics.

[Luke 36992]

Crash is not only after search, but also, when browsing.

Attached a 2nd crash log.

 

@@letterman Unfortunately this defect occurred in 3.5.0, and even with Audiodb disabled in 3.5.2, you still have Audiodb urls saved in your database. Your choices to recover are:

• Remove your music library, run a library scan, then add the music library back
• Use a database editing tool to remove or update the saved audiodb urls

[Happy2Play 8224]

Is this a non-Windows issue as I am not seeing this on my servers.

[letterman 33]

@Luke:

• Remove your music library, run a library scan, then add the music library back
  This is impossible - it would take weeks and it was running very stable till update...
  In case of remove & re-adding. What will happen to my collections ([boxsets])? Will they be lost?

• Use a database editing tool to remove or update the saved audiodb urls
  Where is the database stored (QNAP)? I found the "library.db"-file in .../programdata/data? Is this the one you mean? Gosh, The size is 2.3 GB...
  Which string do I have to search & replace for?
  Which database format is it and which editor do you suggest? (QNAP)
  
  
  This is a real bug - why don't you publish a tool or plugin for this (if you know what to do) easy procedure?

Edited August 7, 2018 by letterman

[evil_beast666 4]

 

@@letterman Unfortunately this defect occurred in 3.5.0, and even with Audiodb disabled in 3.5.2, you still have Audiodb urls saved in your database. Your choices to recover are:

• Remove your music library, run a library scan, then add the music library back
• Use a database editing tool to remove or update the saved audiodb urls

 

 

I choose second option

 

1. Download sqlite browser tool: https://github.com/sqlitebrowser/sqlitebrowser/releases

2. Stop Emby server.

3. Search "library.db" file in your "data" dir. (/opt/emby/config_beta/data in my case).

4. (OPTIONAL) Perform a backup.

5. Open "library.db" with sqlitebrowser and go to "Execute SQL".

6. Paste the next, run and save changes:

UPDATE TypedBaseItems SET Images = null WHERE Images LIKE '%theaudiodb%';

7. Substitute your new modified "library.db".

8. Start Emby server again.

9. It works for me!

 

Enjoy!

[evil_beast666 4]

 

@Luke:

• Remove your music library, run a library scan, then add the music library back

  This is impossible - too many collections & libraries

• Use a database editing tool to remove or update the saved audiodb urls

  Which string do I have to search & replace for?

  Where is the database stored (QNAP)

  Which editor do you suggest? (QNAP)

   

   

  This is a real bug - why don't you publish a tool or plugin for this (if you know what to do) easy procedure? It's really a big mess.... Please help.

 

 

Dude...keep calm and search in Google for 5 minutes...or you can enjoy my instructions

[evil_beast666 4]

I choose second option

 

1. Download sqlite browser tool: https://github.com/sqlitebrowser/sqlitebrowser/releases

2. Stop Emby server.

3. Search "library.db" file in your "data" dir. (/opt/emby/config_beta/data in my case).

4. (OPTIONAL) Perform a backup.

5. Open "library.db" with sqlitebrowser and go to "Execute SQL".

6. Paste the next, run and save changes:

UPDATE TypedBaseItems SET Images = null WHERE Images LIKE '%theaudiodb%';

7. Substitute your new modified "library.db".

8. Start Emby server again.

9. It works for me!

 

Enjoy!

 

After run that sql query, the server works again. But it curious, because when I trying to identify artist metadata still using "theaudiodb" provider but it's manually disabled in my library :-\

 

Also I tried to set the "musicbrainz" id manually but the next it did is get in "theaudiodb" as you can see:

2018-08-07 21:45:34.278 Info App: Setting provider id's to item 946e1eca-9e4b-9f0f-bbfd-2f89aed298d5-GRiZ: {"MusicBrainzArtist":"c62327d3-df81-46ce-bd94-5bd3040f8699"}
2018-08-07 21:45:34.392 Info HttpClient: GET http://www.theaudiodb.com/api/v1/json/49jhsf8248yfahka89724011/artist-mb.php?i=c62327d3-df81-46ce-bd94-5bd3040f8699
2018-08-07 21:45:34.478 Error Main: UnhandledException
        *** Error Report ***
        Version: 3.6.0.2

bla bla bla the same above errors

Maybe is better idea set null Images and ProviderIds...

UPDATE TypedBaseItems SET ProviderIds = null,Images = null WHERE Images LIKE '%theaudiodb%';

What you think @@Luke? Some definitive solution to avoid "theaudiodb"?

 

 

Thx in advance

Edited August 7, 2018 by evil_beast666

[Luke 36992]

Yes that's a good way of wiping out existing audiodb data.

[letterman 33]

It worked for me. Emby is stable again and it seems to be much faster than 3.4.x

Thanks for the help @@evil_beast666

[maegibbons 1267]

Well done @evil_beast666 for a well documented solution.

 

Krs

 

Mark

[Luke 36992]

Well done.

[Luke 36992]

@@letterman please let us know if this helps. Thanks !

[letterman 33]

@@letterman please let us know if this helps. Thanks !

it works.

 

I would mark the solution from evil_beast666 as BEST ANSWER

[Luke 36992]

Thanks for the feedback.