Jump to content

Server crawls to a halt

craigdabbs

By craigdabbs
in Linux

 Share

Recommended Posts

craigdabbs

craigdabbs   0

Posted
Posted

Hi there,

 

I have emby installed on a arch system, ive been running it for a few months now but i have had to periodically restart it to speed things back up.  All clients browse the content really slowly but the streaming is unaffected.

 

Has anyone else had this issue?

 

I get these random write issues in the logs, hope someone can decipher this.

2015-04-06 08:19:15.8695 Error - HttpAsyncTaskHandler: Error occured while Processing Request: Write failure
	*** Error Report ***
	Version: 3.0.5572.0
	Command line: /usr/lib/emby-server/MediaBrowser.Server.Mono.exe -programdata /var/lib/emby-server -ffmpeg /usr/bin/ffmpeg -ffprobe /usr/bin/ffprobe
	Operating system: Unix 3.18.6.1
	Processor count: 4
	64-Bit OS: True
	64-Bit Process: True
	Program data path: /var/lib/emby-server
	Mono: 3.12.1 (tarball Thu Mar 12 06:31:20 UTC 2015)
	Application Path: /usr/lib/emby-server/MediaBrowser.Server.Mono.exe
	Write failure
	System.IO.IOException
	  at System.Net.Sockets.NetworkStream.Write (System.Byte[] buffer, Int32 offset, Int32 size) [0x00000] in <filename unknown>:0 
	  at SocketHttpListener.Net.ResponseStream.InternalWrite (System.Byte[] buffer, Int32 offset, Int32 count) [0x00000] in <filename unknown>:0 
	  at SocketHttpListener.Net.ResponseStream.Write (System.Byte[] buffer, Int32 offset, Int32 count) [0x00000] in <filename unknown>:0 
	  at ServiceStack.Formats.HtmlFormat.SerializeToStream (IRequest request, System.Object response, IResponse httpRes) [0x00000] in <filename unknown>:0 
	  at ServiceStack.HttpResponseExtensionsInternal.WriteErrorToResponse (IResponse httpRes, IRequest httpReq, System.String contentType, System.String operationName, System.String errorMessage, System.Exception ex, Int32 statusCode) [0x00000] in <filename unknown>:0 
	  at ServiceStack.ServiceStackHost.OnUncaughtException (IRequest httpReq, IResponse httpRes, System.String operationName, System.Exception ex) [0x00000] in <filename unknown>:0 
	  at ServiceStack.HostContext.RaiseUncaughtException (IRequest httpReq, IResponse httpRes, System.String operationName, System.Exception ex) [0x00000] in <filename unknown>:0 
	  at ServiceStack.HttpResponseExtensionsInternal.WriteToResponse (IResponse response, System.Object result, ServiceStack.Web.ResponseSerializerDelegate defaultAction, IRequest request, System.Byte[] bodyPrefix, System.Byte[] bodySuffix) [0x00000] in <filename unknown>:0 
	  at ServiceStack.HttpResponseExtensionsInternal.WriteToResponse (IResponse httpRes, IRequest httpReq, System.Object result, System.Byte[] bodyPrefix, System.Byte[] bodySuffix) [0x00000] in <filename unknown>:0 
	  at ServiceStack.HttpResponseExtensionsInternal.WriteToResponse (IResponse httpRes, IRequest httpReq, System.Object result) [0x00000] in <filename unknown>:0 
	  at ServiceStack.Host.RestHandler+<>c__DisplayClass5.<ProcessRequestAsync>b__0 (System.Object response) [0x00000] in <filename unknown>:0 
	  at ServiceStack.Host.Handlers.ServiceStackHandlerBase.HandleResponse (System.Object response, System.Func`2 callback, System.Func`2 errorCallback) [0x00000] in <filename unknown>:0 
	InnerException: System.Net.Sockets.SocketException
	The socket has been shut down
	  at System.Net.Sockets.Socket.Send (System.Byte[] buf, Int32 offset, Int32 size, SocketFlags flags) [0x00000] in <filename unknown>:0 
	  at System.Net.Sockets.NetworkStream.Write (System.Byte[] buffer, Int32 offset, Int32 size) [0x00000] in <filename unknown>:0

Thanks 

 

Craig

Link to comment
Share on other sites
ctung

ctung   0

Posted
Posted (edited)

I've got this same error.  This causes lots of stutters.  

I'm running on some unusual hardware: a gizmo-2 running lubuntu from a msata ssd, and my moves on a USB external drive.

 

2015-04-11 23:25:37.2520 Error - HttpAsyncTaskHandler: Error occured while Processing Request: Write failure
    *** Error Report ***
    Version: 3.0.5572.0
    Command line: /opt/mediabrowser/MediaBrowser.Server.Mono.exe -programdata /var/lib/mediabrowser
    Operating system: Unix 3.16.0.34
    Processor count: 2
    64-Bit OS: True
    64-Bit Process: True
    Program data path: /var/lib/mediabrowser
    Mono: 3.10.0 (tarball Wed Nov 5 12:50:04 UTC 2014)
    Application Path: /opt/mediabrowser/MediaBrowser.Server.Mono.exe
    Write failure
    System.IO.IOException
     at System.Net.Sockets.NetworkStream.Write (System.Byte[] buffer, Int32 offset, Int32 size) [0x00000] in <filename unknown>:0
     at SocketHttpListener.Net.ResponseStream.InternalWrite (System.Byte[] buffer, Int32 offset, Int32 count) [0x00000] in <filename unknown>:0
     at SocketHttpListener.Net.ResponseStream.Write (System.Byte[] buffer, Int32 offset, Int32 count) [0x00000] in <filename unknown>:0
     at ServiceStack.Formats.HtmlFormat.SerializeToStream (IRequest request, System.Object response, IResponse httpRes) [0x00000] in <filename unknown>:0
     at ServiceStack.HttpResponseExtensionsInternal.WriteErrorToResponse (IResponse httpRes, IRequest httpReq, System.String contentType, System.String operationName, System.String errorMessage, System.Exception ex, Int32 statusCode) [0x00000] in <filename unknown>:0
     at ServiceStack.ServiceStackHost.OnUncaughtException (IRequest httpReq, IResponse httpRes, System.String operationName, System.Exception ex) [0x00000] in <filename unknown>:0
     at ServiceStack.HostContext.RaiseUncaughtException (IRequest httpReq, IResponse httpRes, System.String operationName, System.Exception ex) [0x00000] in <filename unknown>:0
     at ServiceStack.HttpResponseExtensionsInternal.WriteToResponse (IResponse response, System.Object result, ServiceStack.Web.ResponseSerializerDelegate defaultAction, IRequest request, System.Byte[] bodyPrefix, System.Byte[] bodySuffix) [0x00000] in <filename unknown>:0
     at ServiceStack.HttpResponseExtensionsInternal.WriteToResponse (IResponse httpRes, IRequest httpReq, System.Object result, System.Byte[] bodyPrefix, System.Byte[] bodySuffix) [0x00000] in <filename unknown>:0
     at ServiceStack.HttpResponseExtensionsInternal.WriteToResponse (IResponse httpRes, IRequest httpReq, System.Object result) [0x00000] in <filename unknown>:0
     at ServiceStack.Host.RestHandler+<>c__DisplayClass5.<ProcessRequestAsync>b__0 (System.Object response) [0x00000] in <filename unknown>:0
     at ServiceStack.Host.Handlers.ServiceStackHandlerBase.HandleResponse (System.Object response, System.Func`2 callback, System.Func`2 errorCallback) [0x00000] in <filename unknown>:0
    InnerException: System.Net.Sockets.SocketException
    The socket has been shut down
     at System.Net.Sockets.Socket.Send (System.Byte[] buf, Int32 offset, Int32 size, SocketFlags flags) [0x00000] in <filename unknown>:0
     at System.Net.Sockets.NetworkStream.Write (System.Byte[] buffer, Int32 offset, Int32 size) [0x00000] in <filename unknown>:0 

 

Full log: http://paste2.org/dkaKgLM2

Edited by ctung
Link to comment
Share on other sites
gsnerf

gsnerf   26

Posted
Posted

@craigdabbs: can you provide a full log please? The snipped alone isn't enough to see whats going on.

 

@ctung: As far as I can see you are trying to stream to the emby web client, is this correct? Are you starting and stopping the movie from the interface or just start it and the stream killing happens from server side?

Link to comment
Share on other sites
ctung

ctung   0

Posted
Posted

Correct, I am streaming to the emby web client.  I tried switching the resolution several times and eventually stopped it.  I tried moving one movie off the USB and onto the harddrive and had the same result.  Right now I'm trying to install emby + ArchLinux.  If that doesn't work I'll reinstall Lubuntu+Emby and get a fresh log without the resolution switches.

Link to comment
Share on other sites
gsnerf

gsnerf   26

Posted
Posted

@ctung: the errors you are seeing are propably because of the switching. Whenever you switch some setting the server has to stop the current transcoding process and restart (at the old place) with the new settings. When this happens some times the old transcoding doesn't stop but tries to continue but has no where to send the new data to. This leads to the write failure (can't write on the now closed stream to the client). This is perfectly normal behaviour and shouldn't affect your actual viewing experience.

 

My guess would be, that the machine isn't strong enough for on the fly recoding of your media to support the browsers viewing format. Maybe the solution to your problem would be to use a client that doesn't require transcoding, like Emby Theatre oder the Kodi plugin?

Link to comment
Share on other sites
craigdabbs

craigdabbs   0

Posted
  • Author
Posted (edited)

sorry for my late reply gsnerf,

 

I have a complete log of emby completely crashing last night, the log didnt show any reason for the crash from what i could see.

 

http://pastebin.com/puYyrNjZ

 

I have enabled debug logging to see if it shows any more details.

 

I also wonder what kind of issue ffmpeg keeps having, it crops up in the log all the time  

 /usr/lib/emby-server/MediaBrowser.Server.Mono.exe -programdata /var/lib/emby-server -ffmpeg /usr/bin/ffmpeg -ffprobe /usr/bin/ffprobe

Ill keep an eye on the amount of open files emby creates as at one stage it was much higher than i think it should of been.  THe last few updates seems to have been better.

 

My system spec:

Core i3 4th gen.

16gb ram

64gb SSD

8TB Raid 5

Arch Linux

 

Thanks again 

Edited by craigdabbs
Link to comment
Share on other sites
gsnerf

gsnerf   26

Posted
Posted

You are right, the log seems clean, none of the 'errors' there would actually cause the server to crash. You are however using mono 3.12 which reportedly has some instabilities. That could be a possible cause for this. If you don't have anything else depending on that mono version I'd suggest you try downgrading to 3.10 and see if that works better.

Link to comment
Share on other sites
gsnerf

gsnerf   26

Posted
Posted

I also wonder what kind of issue ffmpeg keeps having, it crops up in the log all the time  

 /usr/lib/emby-server/MediaBrowser.Server.Mono.exe -programdata /var/lib/emby-server -ffmpeg /usr/bin/ffmpeg -ffprobe /usr/bin/ffprobe

 

That is no problem, it's just an output of the command the server has been started with in the error report. The line is useful to determine what some parameters in the running server are.

Link to comment
Share on other sites
thefirstofthe300

thefirstofthe300   291

Posted
Posted

For those of you running Arch, downgrading to Mono 3.10 using the downgrade command (you can find it in the AUR I do believe) should help with stability. Also, please try the beta server as I have found it to be extremely stable and maybe the issues were resolved in this release. I have updated the Emby server beta package on the AUR.

Link to comment
Share on other sites
craigdabbs

craigdabbs   0

Posted
  • Author
Posted (edited)

Ive not restarted the server since being recommended to downgrade mono,  but it managed 4 days of use.  It did get slower to access the content as time passed.  I run the log on debug mode  to see if it shed any more light on the crash, whether it was mono  related or not.

 

systemd status

● emby-server.service - Media Browser Server is a home media server built using other popular open source technologies.
   Loaded: loaded (/usr/lib/systemd/system/emby-server.service; enabled; vendor preset: disabled)
   Active: active (running) since Tue 2015-04-14 07:58:07 BST; 4 days ago
 Main PID: 19608 (emby-server)
   CGroup: /system.slice/emby-server.service
           ├─19608 /bin/sh /usr/bin/emby-server
           └─19610 /usr/bin/mono /usr/lib/emby-server/MediaBrowser.Server.Mono.exe -programdata /var/lib/emby-server -ffmpeg /usr/bin/ffmpeg -ffprobe /usr/...

Full log

 

I noticed in the log that it takes 2000ms to respond to web requests.   Seems a long time.

 

Would the amount of media files make a difference in speed over time?   I have around 6800 media files.

 

I have now downgraded mono, see how that goes.  

 

Appreciate the advice so far :)

Edited by craigdabbs
Link to comment
Share on other sites
craigdabbs

craigdabbs   0

Posted
  • Author
Posted

Crashed again, 

2015-04-19 21:07:08.8981 Debug - HttpServer: HTTP Response 204 to 86.130.180.198. Response time: 5516.877 ms.
        Url: http://home.craigdabbs.co.uk:5030/mediabrowser/Sessions/Playing/Progress?itemId=bc99a7fa4b7eeb21fa0de1e62b9541fc&canSeek=true&PlayMethod=DirectStream&QueueableMediaTypes=Video&MediaSourceId=bc99a7fa4b7eeb21fa0de1e62b9541fc&PositionTicks=23541730957&AudioStreamIndex=1&IsPaused=false
2015-04-19 21:07:08.9317 Debug - HttpServer: Exiting HttpConnection.OnReadInternal because nread=0. ConnectionId: 166af167d32a47c19b2fc35a16b6c8e0
2015-04-19 21:07:20.8938 Debug - HttpServer: HTTP POST http://home.craigdabbs.co.uk:5030/mediabrowser/Sessions/Playing/Progress?itemId=bc99a7fa4b7eeb21fa0de1e62b9541fc&canSeek=true&PlayMethod=DirectStream&QueueableMediaTypes=Video&MediaSourceId=bc99a7fa4b7eeb21fa0de1e62b9541fc&PositionTicks=23717321777&AudioStreamIndex=1&IsPaused=false
        Ip: 86.130.180.198:57791. Headers: Host=home.craigdabbs.co.uk:5030,Content-Length=0,Accept-encoding=gzip,X-MediaBrowser-Token=8af71f6ee67245f495476af5e7ccd756,Accept-Charset=UTF-8,*,Content-Type=application/x-www-form-urlencoded,Authorization=MediaBrowser UserId="4a3178688d08fa725f4f09104ee4d6d1",Client="Kodi",Device="50p",DeviceId="D737EC88E6F24DE88934DD7D87D3B0ED",Version="0.9.826"
2015-04-19 21:07:23.4064 Debug - HttpServer: HTTP Response 204 to 86.130.180.198. Response time: 2512.433 ms.
        Url: http://home.craigdabbs.co.uk:5030/mediabrowser/Sessions/Playing/Progress?itemId=bc99a7fa4b7eeb21fa0de1e62b9541fc&canSeek=true&PlayMethod=DirectStream&QueueableMediaTypes=Video&MediaSourceId=bc99a7fa4b7eeb21fa0de1e62b9541fc&PositionTicks=23717321777&AudioStreamIndex=1&IsPaused=false
2015-04-19 21:07:23.4528 Debug - HttpServer: Exiting HttpConnection.OnReadInternal because nread=0. ConnectionId: 43c8e15cd6b44316ae1f15888ac6a501
2015-04-19 21:09:09.4562 Debug - HttpServer: HTTP Response 206 to 86.130.180.198. Response time: 2484089.891 ms.
        Url: http://home.craigdabbs.co.uk:5030/mediabrowser/Videos/bc99a7fa4b7eeb21fa0de1e62b9541fc/stream?static=true&AudioStreamIndex=1
 

Taking a long time to respond :).  I have checked the lsof command and its showing 85711 files open, seems excessive considering my main user running lots of other services shows 29000.

Command used  '  lsof | grep ' emby ' | awk '{print $NF}' | sort | wc -l  '

Link to comment
Share on other sites
  • 3 months later...
leme

leme   4

Posted
Posted

I'm running Arch and having these IOException errors as well. Video isn't playing smooth (pausing and then playing). Without transcoding it's running fine.

 

If you find a solution, please let me know.

Link to comment
Share on other sites
Luke Emby Team

Luke   37131

Posted
Posted

I'm running Arch and having these IOException errors as well. Video isn't playing smooth (pausing and then playing). Without transcoding it's running fine.

 

If you find a solution, please let me know.

 

That was a while ago. Please start over and see http://emby.media/community/index.php?/topic/790-how-to-report-a-problem/

Link to comment
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now
 Share
Go to topic listing
×
×
  • Create New...