Jump to content

Emby isn't closing used network ports.


anderbytes
 Share

Recommended Posts

anderbytes

Correction: my version is 4.2.3.4 , not 4.3.2.4

Link to comment
Share on other sites

  • 2 weeks later...
anderbytes

@@Luke and @@ebr, I'd like to go deeper in this problem, and I need your help.

 

I believe the 5 min timeout hasn't solved the issue.

 

1. How is the workflow of new ports opening when a client connects? Is there an special for Emby Mobile connection?

2. Does Emby use a specific HttpServer implementation, such as Apache? or is it a generic one?

3. Is the connection (port) opening started by the Server or the Clients?

4. Can you point me the file in Emby Mobile (or better, the line of code) that actually requires a connection to Emby Server?

5. Can you point me the file in Emby Server (or better, the line of code) that actually creates the connection from the Emby Mobile requisition?

 

----------------------------------------------------

I've been reading about other softwares that deal with the CLOSE_WAIT problem in the following way: they discovered that HTTP/1.1 implementation in RFC 2616 (https://www.ietf.org/rfc/rfc2616.txt) tells that non-persistent connections (such as Emby, I believe) have to create extra header controls to ensure correct port closing after use, as seen below:

14.10 Connection

"HTTP/1.1 defines the "close" connection option for the sender to signal that the connection will be closed after completion
of the response. For example,

       Connection: close

in either the request or the response header fields indicates that the connection SHOULD NOT be considered `persistent'
(section 8.1) after the current request/response is complete.

HTTP/1.1 applications that do not support persistent connections MUST include the "close" connection option in every
message."

so for those other softwares every HTTP Post method had to be adjusted. some kind of: postMethod.addHeader("Connection", "close");

 

 

ANOTHER SUGGESTION:

Instead... it could be studied a way so EMBY uses Persistent connections... so some kind of connections Pool is used for each route the TCP Stack has.

most pool implementations (such as PoolingHttpClientConnectionManager or MultiThreadedHttpConnectionManager) let you define how many connections per user and how many connections at max total.

 

That solution above would force the reutilization of TCP connections, instead of ALWAYS creating new ones. And consequently using better the server resources.

 

Please think seriously about those above...specially the reusable connections possibility.

 

--------------------------------------------------------------------

References:

- http://www.nuxeo.com/blog/using-httpclient-properly-avoid-closewait-tcp-connections/

- http://stackoverflow.com/a/33046331

 

Thanks!

Link to comment
Share on other sites

dcook

Just putting this out there, but if its an EMBY code issue, wouldn't the issue be showing on all version?  

 

Since this issue is specific to the Linux versions, could it not be argued that there must be something specific to Linux that is causing the issue?

Link to comment
Share on other sites

anderbytes

Just putting this out there, but if its an EMBY code issue, wouldn't the issue be showing on all version?  

 

Since this issue is specific to the Linux versions, could it not be argued that there must be something specific to Linux that is causing the issue?

 

It happens that .NET implementation is the same. Windows runs it with .NET Framework... and Linux with Mono.

 

.NET Framework probably is instructed to "don't expect the coder to do this, just do it" , while Mono respects the code more than .NET Framework does.

So... if some code related to HTTP/1.1 is missing (possibly), Mono won't auto-close the ports thinking that the coder wants to use it later (not the case of Emby's code).

Edited by anderbytes
Link to comment
Share on other sites

the server has many places that create connections. incoming from the http server, outgoing to metadata sites, sockets for upnp messages, etc. it is probably best that you identify which of these you think is the area where ports aren't getting closed.

Link to comment
Share on other sites

anderbytes

the server has many places that create connections. incoming from the http server, outgoing to metadata sites, sockets for upnp messages, etc. it is probably best that you identify which of these you think is the area where ports aren't getting closed.

 

As tested before... only connections incoming from Mobile (Android, at least) suffer from this issue.

Any tips on where to look at?

Link to comment
Share on other sites

  • 3 weeks later...
pdoroba

I'm having the same problem you are. I only noticed last night that there were several states only associated with Emby stuck in FIN_WAIT_2 .  It's very frustrating! I've had to restart the emby service manually several times a day to get the Web UI to respond. Here's my post http://emby.media/community/index.php?/topic/35582-emby-web-ui-crashesstops-responding/ I'm running Emby on CentOS7 btw.

Link to comment
Share on other sites

anderbytes

Are we confident now this is only an issue on Linux variants?

 

I was already convinced by at least a month...

 

Are you asking this to ditch the topic to another forum category or to bring some SuperLinuxHero to help us?

Link to comment
Share on other sites

anderbytes

Anyone also experiencing the issue can try tweaking a few tcp tunings?

 

http://kaivanov.blogspot.com.br/2010/09/linux-tcp-tuning.html

 

I was thinking about:

- Reducing TCP_FIN_TIMEOUT to 30 (default: 60)

- Activating TCP_TW_RECYCLE to 1 (default: 0)

- Activating TCP_TW_REUSE to 1 (default: 0)

 

What do you think?

Link to comment
Share on other sites

anderbytes

Can anyone at this Linux Category actually help our cause? This is frustrating.

Link to comment
Share on other sites

runtimesandbox

I am quite confident this is only really a linux issue. 

 

I have run emby on both debian and ubuntu 32/64 bit variants and had the issue on both. 

 

I only seem to have the issue after connecting via the android app. However this is hard to prove as it doesn't happen straight away. But If i do not use the android app I have not noticed the issue

Link to comment
Share on other sites

anderbytes

A good test would be: when it gets the moment where Android can't connect anymore, then immediately try from web (same address:port), and see if it also hangs.

Link to comment
Share on other sites

anderbytes

@, have you tried with the 3 tcp tweaks I posted above?

 

I'm beginning some tests with them. Others doing would be good, too

Link to comment
Share on other sites

runtimesandbox

@@anderbytes I will setup a test server again over the weekend once I've had a chance to rebuild my testing server.

Will give those tweaks a try.

Link to comment
Share on other sites

Banjo

If it's any use, I'm running the latest Emby Docker container on a Ubuntu 14.04 host and I am not seeing this issue.

Link to comment
Share on other sites

anderbytes

@@anderbytes I will setup a test server again over the weekend once I've had a chance to rebuild my testing server.

Will give those tweaks a try.

 

I've found this interesting site related to Linux and TIME-WAIT connection states.

https://vincent.bernat.im/en/blog/2014-tcp-time-wait-state-linux.html

 

Maybe it help us understand better...

Link to comment
Share on other sites

anderbytes

No good.

 

The problem persists here...

 

 

Can't the server sweep the open ports and manually close those whose endpoint doesn't answer???

Link to comment
Share on other sites

Mono 4.4 went stable today. you can try playing with that but just be warned we haven't tested it yet. You'll need to revert back to 4.2.3 if you have any problems.

  • Like 1
Link to comment
Share on other sites

anderbytes

Mono 4.4 went stable today. you can try playing with that but just be warned we haven't tested it yet. You'll need to revert back to 4.2.3 if you have any problems.

 

Thanks, will give a try.

But I have to wait until 4.4.0 hits Debian 7 repositories, as I use OMV 2 today.

 

In http://www.mono-project.com/docs/about-mono/releases/4.4.0/ it is possible to know what changed.

I see that some system functions are now supported.

 

Can you tell me if now that support those below, we may have a chance?

- LingerOption

- SocketErrors

- SocketException

- SocketFlags

- SocketShutdown

- SocketType

- etc...

  • Like 1
Link to comment
Share on other sites

pdoroba

I removed the android app from my phone about 5 days ago and went 3 days without the Emby UI requiring me to restart the Emby service to make it accessible again.  Two days ago I decided to try to login to Emby from my phone using Firefox browser again causing the Emby UI to become unresponsive.  I then checked my firewall (pfSense) to find the FIN_WAIT_2 states present.  I again had to restart the Emby service to clear the states.  In my case it would seem it's not just the android app but perhaps android itself causing the issue?  I haven't connected to Emby with android again since and Emby has been stable.

 

I run Emby on a CentOS7 VM on ESXi with the recommended mono version 4.2.3.4

Edited by pdoroba
Link to comment
Share on other sites

I run Emby 3.0.5972.0 (mono 4.2.3.4) on a Centos 7.2.1511 KVM (proxmox).

 

I tried opening media using web-player (firefox, win10), Emby Theater (win10), Emby for Android TV (nvidia shield) and lastly Emby Android App (Android MM). All of those closed connections fine after waiting about 5 (?) minutes.

Windows 10 desktop is on a different VLAN than emby server (though it gets routed directly by the switch) while Android and Android TV apps have to pass a pfSense firewall first (NAT). Apps connect over http port 8096 since there is a problem with mono and https currently (which is mentioned in a different thread).

If I can provide any info to help please let me know.  I didn't apply any of the tcp tweaks mentioned above.

 

netstat during playback through emby for android:

[20:48:40] me@[member="emby"] ~
$ sudo netstat -anoe | grep ":8920\|8096"
tcp        0      0 0.0.0.0:8920            0.0.0.0:*               LISTEN      994        23529      off (0.00/0/0)
tcp        0      0 0.0.0.0:8096            0.0.0.0:*               LISTEN      994        23528      off (0.00/0/0)
tcp        0      0 10.0.20.6:8096          192.168.178.22:60973    ESTABLISHED 994        60615      off (0.00/0/0)
tcp        0      0 10.0.20.6:8096          192.168.178.22:51394    ESTABLISHED 994        57334      off (0.00/0/0)
tcp        0      0 10.0.20.6:8096          192.168.178.22:51393    ESTABLISHED 994        58758      off (0.00/0/0)
tcp        0      0 10.0.20.6:8096          192.168.178.22:51390    ESTABLISHED 994        57932      off (0.00/0/0)
tcp        0      0 10.0.20.6:8096          192.168.178.22:51388    ESTABLISHED 994        61493      off (0.00/0/0)
tcp        0      0 10.0.20.6:8096          192.168.178.22:51391    ESTABLISHED 994        54728      off (0.00/0/0)
tcp        0      0 10.0.20.6:58113         10.0.20.6:8096          ESTABLISHED 994        57928      off (0.00/0/0)
tcp        0      0 10.0.20.6:8096          192.168.178.22:51385    ESTABLISHED 994        55883      off (0.00/0/0)
tcp        0 1045256 10.0.20.6:8096          192.168.178.22:51400    ESTABLISHED 994        48761      on (0.23/0/0)
tcp        0      0 10.0.20.6:8096          192.168.178.22:51387    ESTABLISHED 994        57930      off (0.00/0/0)
tcp        0      0 10.0.20.6:8096          192.168.178.22:51386    ESTABLISHED 994        55886      off (0.00/0/0)
tcp        0      0 10.0.20.6:8096          192.168.178.22:51395    ESTABLISHED 994        57335      off (0.00/0/0)
tcp        0      0 10.0.20.6:8096          192.168.178.22:51396    ESTABLISHED 994        58759      off (0.00/0/0)
tcp        0      0 10.0.20.6:8096          10.0.20.6:58113         ESTABLISHED 994        58748      off (0.00/0/0)

 right after playback and closing emby for android:

[20:49:05] me@[member="emby"] ~
$ sudo netstat -anoe | grep ":8920\|8096"
tcp        0      0 0.0.0.0:8920            0.0.0.0:*               LISTEN      994        23529      off (0.00/0/0)
tcp        0      0 0.0.0.0:8096            0.0.0.0:*               LISTEN      994        23528      off (0.00/0/0)
tcp        0      0 10.0.20.6:58113         10.0.20.6:8096          ESTABLISHED 994        57928      off (0.00/0/0)
tcp        0      0 10.0.20.6:8096          192.168.178.22:51387    TIME_WAIT   0          0          timewait (53.42/0/0)
tcp        0      0 10.0.20.6:8096          10.0.20.6:58113         ESTABLISHED 994        58748      off (0.00/0/0)

5 minutes later:

[20:54:46] me@[member="emby"] ~
$ sudo netstat -anoe | grep ":8920\|8096"
tcp        0      0 0.0.0.0:8920            0.0.0.0:*               LISTEN      994        23529      off (0.00/0/0)
tcp        0      0 0.0.0.0:8096            0.0.0.0:*               LISTEN      994        23528      off (0.00/0/0)
Link to comment
Share on other sites

Good info, thanks! that corresponds exactly with the 5 minute keep alive duration that the server is using.

Link to comment
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now
 Share

×
×
  • Create New...