anderbytes 139 Posted May 3, 2016 Author Share Posted May 3, 2016 Correction: my version is 4.2.3.4 , not 4.3.2.4 Link to comment Share on other sites More sharing options...
anderbytes 139 Posted May 17, 2016 Author Share Posted May 17, 2016 @@Luke and @@ebr, I'd like to go deeper in this problem, and I need your help. I believe the 5 min timeout hasn't solved the issue. 1. How is the workflow of new ports opening when a client connects? Is there an special for Emby Mobile connection? 2. Does Emby use a specific HttpServer implementation, such as Apache? or is it a generic one? 3. Is the connection (port) opening started by the Server or the Clients? 4. Can you point me the file in Emby Mobile (or better, the line of code) that actually requires a connection to Emby Server? 5. Can you point me the file in Emby Server (or better, the line of code) that actually creates the connection from the Emby Mobile requisition? ---------------------------------------------------- I've been reading about other softwares that deal with the CLOSE_WAIT problem in the following way: they discovered that HTTP/1.1 implementation in RFC 2616 (https://www.ietf.org/rfc/rfc2616.txt) tells that non-persistent connections (such as Emby, I believe) have to create extra header controls to ensure correct port closing after use, as seen below: 14.10 Connection "HTTP/1.1 defines the "close" connection option for the sender to signal that the connection will be closed after completion of the response. For example, Connection: close in either the request or the response header fields indicates that the connection SHOULD NOT be considered `persistent' (section 8.1) after the current request/response is complete. HTTP/1.1 applications that do not support persistent connections MUST include the "close" connection option in every message." so for those other softwares every HTTP Post method had to be adjusted. some kind of: postMethod.addHeader("Connection", "close"); ANOTHER SUGGESTION: Instead... it could be studied a way so EMBY uses Persistent connections... so some kind of connections Pool is used for each route the TCP Stack has. most pool implementations (such as PoolingHttpClientConnectionManager or MultiThreadedHttpConnectionManager) let you define how many connections per user and how many connections at max total. That solution above would force the reutilization of TCP connections, instead of ALWAYS creating new ones. And consequently using better the server resources. Please think seriously about those above...specially the reusable connections possibility. -------------------------------------------------------------------- References: - http://www.nuxeo.com/blog/using-httpclient-properly-avoid-closewait-tcp-connections/ - http://stackoverflow.com/a/33046331 Thanks! Link to comment Share on other sites More sharing options...
dcook 265 Posted May 17, 2016 Share Posted May 17, 2016 Just putting this out there, but if its an EMBY code issue, wouldn't the issue be showing on all version? Since this issue is specific to the Linux versions, could it not be argued that there must be something specific to Linux that is causing the issue? Link to comment Share on other sites More sharing options...
anderbytes 139 Posted May 17, 2016 Author Share Posted May 17, 2016 (edited) Just putting this out there, but if its an EMBY code issue, wouldn't the issue be showing on all version? Since this issue is specific to the Linux versions, could it not be argued that there must be something specific to Linux that is causing the issue? It happens that .NET implementation is the same. Windows runs it with .NET Framework... and Linux with Mono. .NET Framework probably is instructed to "don't expect the coder to do this, just do it" , while Mono respects the code more than .NET Framework does. So... if some code related to HTTP/1.1 is missing (possibly), Mono won't auto-close the ports thinking that the coder wants to use it later (not the case of Emby's code). Edited May 17, 2016 by anderbytes Link to comment Share on other sites More sharing options...
Luke 37045 Posted May 18, 2016 Share Posted May 18, 2016 the server has many places that create connections. incoming from the http server, outgoing to metadata sites, sockets for upnp messages, etc. it is probably best that you identify which of these you think is the area where ports aren't getting closed. Link to comment Share on other sites More sharing options...
anderbytes 139 Posted May 18, 2016 Author Share Posted May 18, 2016 the server has many places that create connections. incoming from the http server, outgoing to metadata sites, sockets for upnp messages, etc. it is probably best that you identify which of these you think is the area where ports aren't getting closed. As tested before... only connections incoming from Mobile (Android, at least) suffer from this issue. Any tips on where to look at? Link to comment Share on other sites More sharing options...
pdoroba 0 Posted June 7, 2016 Share Posted June 7, 2016 I'm having the same problem you are. I only noticed last night that there were several states only associated with Emby stuck in FIN_WAIT_2 . It's very frustrating! I've had to restart the emby service manually several times a day to get the Web UI to respond. Here's my post http://emby.media/community/index.php?/topic/35582-emby-web-ui-crashesstops-responding/ I'm running Emby on CentOS7 btw. Link to comment Share on other sites More sharing options...
ebr 14910 Posted June 7, 2016 Share Posted June 7, 2016 Are we confident now this is only an issue on Linux variants? Link to comment Share on other sites More sharing options...
anderbytes 139 Posted June 7, 2016 Author Share Posted June 7, 2016 Are we confident now this is only an issue on Linux variants? I was already convinced by at least a month... Are you asking this to ditch the topic to another forum category or to bring some SuperLinuxHero to help us? Link to comment Share on other sites More sharing options...
anderbytes 139 Posted June 7, 2016 Author Share Posted June 7, 2016 Anyone also experiencing the issue can try tweaking a few tcp tunings? http://kaivanov.blogspot.com.br/2010/09/linux-tcp-tuning.html I was thinking about: - Reducing TCP_FIN_TIMEOUT to 30 (default: 60) - Activating TCP_TW_RECYCLE to 1 (default: 0) - Activating TCP_TW_REUSE to 1 (default: 0) What do you think? Link to comment Share on other sites More sharing options...
anderbytes 139 Posted June 8, 2016 Author Share Posted June 8, 2016 Can anyone at this Linux Category actually help our cause? This is frustrating. Link to comment Share on other sites More sharing options...
runtimesandbox 152 Posted June 9, 2016 Share Posted June 9, 2016 I am quite confident this is only really a linux issue. I have run emby on both debian and ubuntu 32/64 bit variants and had the issue on both. I only seem to have the issue after connecting via the android app. However this is hard to prove as it doesn't happen straight away. But If i do not use the android app I have not noticed the issue Link to comment Share on other sites More sharing options...
anderbytes 139 Posted June 9, 2016 Author Share Posted June 9, 2016 A good test would be: when it gets the moment where Android can't connect anymore, then immediately try from web (same address:port), and see if it also hangs. Link to comment Share on other sites More sharing options...
anderbytes 139 Posted June 9, 2016 Author Share Posted June 9, 2016 @, have you tried with the 3 tcp tweaks I posted above? I'm beginning some tests with them. Others doing would be good, too Link to comment Share on other sites More sharing options...
runtimesandbox 152 Posted June 9, 2016 Share Posted June 9, 2016 @@anderbytes I will setup a test server again over the weekend once I've had a chance to rebuild my testing server. Will give those tweaks a try. Link to comment Share on other sites More sharing options...
Banjo 2 Posted June 9, 2016 Share Posted June 9, 2016 If it's any use, I'm running the latest Emby Docker container on a Ubuntu 14.04 host and I am not seeing this issue. Link to comment Share on other sites More sharing options...
runtimesandbox 152 Posted June 9, 2016 Share Posted June 9, 2016 @@Banjo do you use the android app at all? Link to comment Share on other sites More sharing options...
anderbytes 139 Posted June 9, 2016 Author Share Posted June 9, 2016 @@anderbytes I will setup a test server again over the weekend once I've had a chance to rebuild my testing server. Will give those tweaks a try. I've found this interesting site related to Linux and TIME-WAIT connection states. https://vincent.bernat.im/en/blog/2014-tcp-time-wait-state-linux.html Maybe it help us understand better... Link to comment Share on other sites More sharing options...
anderbytes 139 Posted June 9, 2016 Author Share Posted June 9, 2016 No good. The problem persists here... Can't the server sweep the open ports and manually close those whose endpoint doesn't answer??? Link to comment Share on other sites More sharing options...
Luke 37045 Posted June 9, 2016 Share Posted June 9, 2016 Mono 4.4 went stable today. you can try playing with that but just be warned we haven't tested it yet. You'll need to revert back to 4.2.3 if you have any problems. 1 Link to comment Share on other sites More sharing options...
anderbytes 139 Posted June 9, 2016 Author Share Posted June 9, 2016 Mono 4.4 went stable today. you can try playing with that but just be warned we haven't tested it yet. You'll need to revert back to 4.2.3 if you have any problems. Thanks, will give a try. But I have to wait until 4.4.0 hits Debian 7 repositories, as I use OMV 2 today. In http://www.mono-project.com/docs/about-mono/releases/4.4.0/ it is possible to know what changed. I see that some system functions are now supported. Can you tell me if now that support those below, we may have a chance? - LingerOption - SocketErrors - SocketException - SocketFlags - SocketShutdown - SocketType - etc... 1 Link to comment Share on other sites More sharing options...
Banjo 2 Posted June 10, 2016 Share Posted June 10, 2016 @@Banjo do you use the android app at all? Yes, I do. Link to comment Share on other sites More sharing options...
pdoroba 0 Posted June 11, 2016 Share Posted June 11, 2016 (edited) I removed the android app from my phone about 5 days ago and went 3 days without the Emby UI requiring me to restart the Emby service to make it accessible again. Two days ago I decided to try to login to Emby from my phone using Firefox browser again causing the Emby UI to become unresponsive. I then checked my firewall (pfSense) to find the FIN_WAIT_2 states present. I again had to restart the Emby service to clear the states. In my case it would seem it's not just the android app but perhaps android itself causing the issue? I haven't connected to Emby with android again since and Emby has been stable. I run Emby on a CentOS7 VM on ESXi with the recommended mono version 4.2.3.4 Edited June 11, 2016 by pdoroba Link to comment Share on other sites More sharing options...
xenu 10 Posted June 11, 2016 Share Posted June 11, 2016 I run Emby 3.0.5972.0 (mono 4.2.3.4) on a Centos 7.2.1511 KVM (proxmox). I tried opening media using web-player (firefox, win10), Emby Theater (win10), Emby for Android TV (nvidia shield) and lastly Emby Android App (Android MM). All of those closed connections fine after waiting about 5 (?) minutes. Windows 10 desktop is on a different VLAN than emby server (though it gets routed directly by the switch) while Android and Android TV apps have to pass a pfSense firewall first (NAT). Apps connect over http port 8096 since there is a problem with mono and https currently (which is mentioned in a different thread). If I can provide any info to help please let me know. I didn't apply any of the tcp tweaks mentioned above. netstat during playback through emby for android: [20:48:40] me@[member="emby"] ~ $ sudo netstat -anoe | grep ":8920\|8096" tcp 0 0 0.0.0.0:8920 0.0.0.0:* LISTEN 994 23529 off (0.00/0/0) tcp 0 0 0.0.0.0:8096 0.0.0.0:* LISTEN 994 23528 off (0.00/0/0) tcp 0 0 10.0.20.6:8096 192.168.178.22:60973 ESTABLISHED 994 60615 off (0.00/0/0) tcp 0 0 10.0.20.6:8096 192.168.178.22:51394 ESTABLISHED 994 57334 off (0.00/0/0) tcp 0 0 10.0.20.6:8096 192.168.178.22:51393 ESTABLISHED 994 58758 off (0.00/0/0) tcp 0 0 10.0.20.6:8096 192.168.178.22:51390 ESTABLISHED 994 57932 off (0.00/0/0) tcp 0 0 10.0.20.6:8096 192.168.178.22:51388 ESTABLISHED 994 61493 off (0.00/0/0) tcp 0 0 10.0.20.6:8096 192.168.178.22:51391 ESTABLISHED 994 54728 off (0.00/0/0) tcp 0 0 10.0.20.6:58113 10.0.20.6:8096 ESTABLISHED 994 57928 off (0.00/0/0) tcp 0 0 10.0.20.6:8096 192.168.178.22:51385 ESTABLISHED 994 55883 off (0.00/0/0) tcp 0 1045256 10.0.20.6:8096 192.168.178.22:51400 ESTABLISHED 994 48761 on (0.23/0/0) tcp 0 0 10.0.20.6:8096 192.168.178.22:51387 ESTABLISHED 994 57930 off (0.00/0/0) tcp 0 0 10.0.20.6:8096 192.168.178.22:51386 ESTABLISHED 994 55886 off (0.00/0/0) tcp 0 0 10.0.20.6:8096 192.168.178.22:51395 ESTABLISHED 994 57335 off (0.00/0/0) tcp 0 0 10.0.20.6:8096 192.168.178.22:51396 ESTABLISHED 994 58759 off (0.00/0/0) tcp 0 0 10.0.20.6:8096 10.0.20.6:58113 ESTABLISHED 994 58748 off (0.00/0/0) right after playback and closing emby for android: [20:49:05] me@[member="emby"] ~ $ sudo netstat -anoe | grep ":8920\|8096" tcp 0 0 0.0.0.0:8920 0.0.0.0:* LISTEN 994 23529 off (0.00/0/0) tcp 0 0 0.0.0.0:8096 0.0.0.0:* LISTEN 994 23528 off (0.00/0/0) tcp 0 0 10.0.20.6:58113 10.0.20.6:8096 ESTABLISHED 994 57928 off (0.00/0/0) tcp 0 0 10.0.20.6:8096 192.168.178.22:51387 TIME_WAIT 0 0 timewait (53.42/0/0) tcp 0 0 10.0.20.6:8096 10.0.20.6:58113 ESTABLISHED 994 58748 off (0.00/0/0) 5 minutes later: [20:54:46] me@[member="emby"] ~ $ sudo netstat -anoe | grep ":8920\|8096" tcp 0 0 0.0.0.0:8920 0.0.0.0:* LISTEN 994 23529 off (0.00/0/0) tcp 0 0 0.0.0.0:8096 0.0.0.0:* LISTEN 994 23528 off (0.00/0/0) Link to comment Share on other sites More sharing options...
Luke 37045 Posted June 11, 2016 Share Posted June 11, 2016 Good info, thanks! that corresponds exactly with the 5 minute keep alive duration that the server is using. Link to comment Share on other sites More sharing options...
Recommended Posts
Create an account or sign in to comment
You need to be a member in order to leave a comment
Create an account
Sign up for a new account in our community. It's easy!
Register a new accountSign in
Already have an account? Sign in here.
Sign In Now