Jump to content

Emby isn't closing used network ports.


anderbytes

Recommended Posts

anderbytes

Hello guys,
 
I guess I've found a nasty bug that can degrade overall Emby performance.
 
Server: v3.0.5821 (latest stable)
 
I accidentaly found that in some cases the ports that Emby use to allow clients connections aren`t closed as it should, after being used.

Today by morning I was trying to access the WebClient interface... and it wouldn`t connect at all...at didn't returned any error, either.
I confirmed that 8096 and 8920 were been listened by Emby, and both server and client were correctly connected to internal network.
 
So... after some diagnose, I discovered that several connections were estabilished. I used the command netstat -anoe | grep ":8" in my server.
While one TV was using the direct streaming, I couldn't confirm it was normal or not (strange, at most).

 

When restarted the server, the open ports cleared, and I could connect again.

 
Now, several hours later, with no TV or AndroidApp or Theater connected. I ran the same command and confirmed my worries:

tcp 0 0 0.0.0.0:8096 0.0.0.0:* LISTEN 116 312514 off (0.00/0/0)
tcp 0 0 0.0.0.0:8920 0.0.0.0:* LISTEN 116 312515 off (0.00/0/0)
tcp 0 0 192.168.0.4:8096 192.168.0.101:60651 ESTABLISHED 116 316922 off (0.00/0/0)
tcp 0 0 192.168.0.4:8096 192.168.0.4:53790 ESTABLISHED 116 379329 off (0.00/0/0)
tcp 0 0 192.168.0.4:8096 192.168.0.107:36751 ESTABLISHED 116 307935 off (0.00/0/0)
tcp 0 0 192.168.0.4:8096 192.168.0.101:43291 ESTABLISHED 116 350878 off (0.00/0/0)
tcp 0 0 192.168.0.4:8096 192.168.0.107:36766 ESTABLISHED 116 312806 off (0.00/0/0)
tcp 0 0 192.168.0.4:53790 192.168.0.4:8096 ESTABLISHED 116 378719 off (0.00/0/0)

Not even IP .101 or .107 are online anymore since 12pm (4 hours agora). But their used ports are still opened and using O.S. resources.

 

Can anyone reproduce this?

 

Are server logs necessary? I'm not fond of distributing logs for every reason, as I'm paranoic about security.

 

UPDATE: I noticed that every time I refresh the webgui (F5 in browser), it opens new ports and doesn`t closed the others, keeping the ESTABILISHED status working 4ever   :-((((

 

UPDATE 2: After closing the browser, all connections get closed. Well... okay for now.

But I forgot to tell one thing. IPs .101 and .107 above were respectively an Android Emby App and a Samsung Emby App.

Edited by anderbytes
Link to comment
Share on other sites

anderbytes

I just had this problem again. I'm in emby connect (Android client), and opened ports now piled up in close_wait state and never vanish. This is taking lots of resources and doesn't let me connect again

 

Please read this:

http://blogs.technet.com/b/janelewis/archive/2010/03/09/explaining-close-wait.aspx

 

See the close_wait part. That is just like what I'm seeing here.

 

Can you confirm the server manages to close its ports correctly? How is this enforced?

 

Thanks in advance

Link to comment
Share on other sites

the server closes all of it's sockets, although it's always possible there's an issue in a library we're using.

Link to comment
Share on other sites

anderbytes

When using netstat command with -anoe parameter you can see the CLOSE_WAIT countdown to auto-close, on each connection.

 

In the cases I described, they already reached zero and still don't go away :-(

Edited by anderbytes
Link to comment
Share on other sites

anderbytes

If you help me find a better way to analyze it, I can try to diagnose better.

 

Will debug mode help me somehow? Does it log when a port is closed by the server?

Link to comment
Share on other sites

anderbytes

Another output for netstat. The same connections are there for hours, since my last messages...

root@server{/home/MyUser}: netstat -anoe | grep '8096\|8920'
tcp        0      0 0.0.0.0:8096            0.0.0.0:*               LISTEN      116        1320658     off (0.00/0/0)
tcp        0      0 0.0.0.0:8920            0.0.0.0:*               LISTEN      116        1320659     off (0.00/0/0)
tcp        0      0 192.168.0.4:8920        191.40.61.190:43533     ESTABLISHED 116        1429136     off (0.00/0/0)
tcp        0      0 192.168.0.4:8096        192.168.0.101:39181     ESTABLISHED 116        1461463     off (0.00/0/0)
tcp        1      0 192.168.0.4:8920        179.238.113.193:55820   CLOSE_WAIT  0          0           off (0.00/0/0)
tcp        0      0 192.168.0.4:8920        179.238.115.221:58572   ESTABLISHED 116        1429673     off (0.00/0/0)
tcp      206      0 192.168.0.4:8920        179.237.25.246:57697    CLOSE_WAIT  0          0           off (0.00/0/0)
tcp        0      0 192.168.0.4:8920        179.238.115.221:58571   ESTABLISHED 116        1429068     off (0.00/0/0)
tcp        0      0 192.168.0.4:8096        192.168.0.101:39191     ESTABLISHED 116        1462425     off (0.00/0/0)
tcp        0      0 192.168.0.4:8096        192.168.0.101:39078     ESTABLISHED 116        1452792     off (0.00/0/0)
tcp      182      0 192.168.0.4:8920        179.237.25.246:52182    CLOSE_WAIT  0          0           off (0.00/0/0)
tcp        0      0 192.168.0.4:8920        179.238.115.221:58567   ESTABLISHED 116        1429641     off (0.00/0/0)
tcp        0      0 192.168.0.4:8920        192.168.0.100:52473     ESTABLISHED 116        1320673     off (0.00/0/0)
tcp        0      0 192.168.0.4:8096        192.168.0.101:36872     ESTABLISHED 116        1477012     off (0.00/0/0)
tcp        0      0 192.168.0.4:8920        179.238.115.221:58577   ESTABLISHED 116        1429071     off (0.00/0/0)
tcp        0      0 192.168.0.4:8096        192.168.0.106:52576     ESTABLISHED 116        1466172     off (0.00/0/0)
tcp        0      0 192.168.0.4:8920        177.176.190.222:39878   ESTABLISHED 116        1430107     off (0.00/0/0)
tcp      206      0 192.168.0.4:8920        179.237.25.246:49447    CLOSE_WAIT  0          0           off (0.00/0/0)
tcp        0      0 192.168.0.4:8096        192.168.0.101:39190     ESTABLISHED 116        1461470     off (0.00/0/0)
tcp        0      0 192.168.0.4:8096        192.168.0.101:39194     ESTABLISHED 116        1461474     off (0.00/0/0)
tcp        0      0 192.168.0.4:8920        179.238.115.221:60005   ESTABLISHED 116        1429075     off (0.00/0/0)
tcp        0      0 192.168.0.4:8096        192.168.0.101:39192     ESTABLISHED 116        1462426     off (0.00/0/0)
tcp        0      0 192.168.0.4:8920        179.238.115.221:58568   ESTABLISHED 116        1430564     off (0.00/0/0)
tcp        0      0 192.168.0.4:8920        192.168.0.100:52482     ESTABLISHED 116        1320676     off (0.00/0/0)
tcp      206      0 192.168.0.4:8920        179.237.25.246:50649    CLOSE_WAIT  0          0           off (0.00/0/0)
tcp        0      0 192.168.0.4:8096        192.168.0.101:39100     ESTABLISHED 116        1455228     off (0.00/0/0)
tcp      206      0 192.168.0.4:8920        179.238.113.193:55818   CLOSE_WAIT  0          0           off (0.00/0/0)
tcp        0      0 192.168.0.4:8920        191.40.61.190:43538     ESTABLISHED 116        1429792     off (0.00/0/0)
tcp        0      0 192.168.0.4:8096        192.168.0.101:39099     ESTABLISHED 116        1455227     off (0.00/0/0)
tcp        0      0 192.168.0.4:8096        192.168.0.101:39079     ESTABLISHED 116        1453867     off (0.00/0/0)
tcp        0      0 192.168.0.4:8920        179.238.115.221:58929   ESTABLISHED 116        1429698     off (0.00/0/0)
tcp        0      0 192.168.0.4:8096        192.168.0.101:39189     ESTABLISHED 116        1461469     off (0.00/0/0)
tcp      206      0 192.168.0.4:8920        179.237.25.246:51235    CLOSE_WAIT  0          0           off (0.00/0/0)
tcp        0      0 192.168.0.4:8096        192.168.0.101:39098     ESTABLISHED 116        1455226     off (0.00/0/0)
tcp        0      0 192.168.0.4:8920        192.168.0.100:52464     ESTABLISHED 116        1323072     off (0.00/0/0)
tcp        0      0 192.168.0.4:8096        192.168.0.101:39077     ESTABLISHED 116        1451975     off (0.00/0/0)
tcp      183      0 192.168.0.4:8920        192.168.0.100:52696     CLOSE_WAIT  0          0           off (0.00/0/0)
tcp      206      0 192.168.0.4:8920        179.238.113.193:55817   CLOSE_WAIT  0          0           off (0.00/0/0)
tcp      206      0 192.168.0.4:8920        179.238.113.193:55804   CLOSE_WAIT  0          0           off (0.00/0/0)
tcp      206      0 192.168.0.4:8920        179.237.25.246:61568    CLOSE_WAIT  0          0           off (0.00/0/0)
tcp        0      0 192.168.0.4:58750       192.168.0.4:8096        ESTABLISHED 116        1482388     off (0.00/0/0)
tcp        0      0 192.168.0.4:8920        179.238.115.221:58565   ESTABLISHED 116        1427677     off (0.00/0/0)
tcp        0      0 192.168.0.4:8096        192.168.0.101:39080     ESTABLISHED 116        1451977     off (0.00/0/0)
tcp        0      0 192.168.0.4:8096        192.168.0.4:58750       ESTABLISHED 116        1483981     off (0.00/0/0)
tcp      206      0 192.168.0.4:8920        179.237.25.246:59730    CLOSE_WAIT  0          0           off (0.00/0/0)
tcp        0      0 192.168.0.4:8096        192.168.0.101:50352     ESTABLISHED 116        1479029     off (0.00/0/0)
tcp      206      0 192.168.0.4:8920        179.237.25.246:52397    CLOSE_WAIT  0          0           off (0.00/0/0)
tcp      182      0 192.168.0.4:8920        179.237.25.246:54390    CLOSE_WAIT  0          0           off (0.00/0/0)
tcp        0      0 192.168.0.4:8096        192.168.0.101:39096     ESTABLISHED 116        1452856     off (0.00/0/0)
tcp        0      0 192.168.0.4:8096        192.168.0.101:53523     ESTABLISHED 116        1452820     off (0.00/0/0)
tcp      206      0 192.168.0.4:8920        179.237.25.246:53115    CLOSE_WAIT  0          0           off (0.00/0/0)
tcp      206      0 192.168.0.4:8920        179.237.25.246:57447    CLOSE_WAIT  0          0           off (0.00/0/0)
tcp      206      0 192.168.0.4:8920        179.237.25.246:49274    CLOSE_WAIT  0          0           off (0.00/0/0)
tcp        0      0 192.168.0.4:8096        192.168.0.101:38707     ESTABLISHED 116        1456191     off (0.00/0/0)
tcp      206      0 192.168.0.4:8920        179.238.113.193:55821   CLOSE_WAIT  0          0           off (0.00/0/0)
tcp      206      0 192.168.0.4:8920        179.237.25.246:54499    CLOSE_WAIT  0          0           off (0.00/0/0)
tcp        0      0 192.168.0.4:8920        191.40.61.190:43536     ESTABLISHED 116        1429139     off (0.00/0/0)
tcp        1      0 192.168.0.4:8920        179.237.25.246:62790    CLOSE_WAIT  0          0           off (0.00/0/0)
tcp        0      0 192.168.0.4:8096        192.168.0.101:41741     ESTABLISHED 116        1462456     off (0.00/0/0)
tcp        0      0 192.168.0.4:8096        192.168.0.106:52587     ESTABLISHED 116        1466206     off (0.00/0/0)
tcp      206      0 192.168.0.4:8920        179.238.113.193:55815   CLOSE_WAIT  0          0           off (0.00/0/0)
tcp      195      0 192.168.0.4:8920        192.168.0.100:52695     CLOSE_WAIT  0          0           off (0.00/0/0)
tcp      206      0 192.168.0.4:8920        179.237.25.246:64745    CLOSE_WAIT  0          0           off (0.00/0/0)
tcp        1      0 192.168.0.4:8920        179.237.25.246:53549    CLOSE_WAIT  0          0           off (0.00/0/0)
tcp      206      0 192.168.0.4:8920        179.238.113.193:55822   CLOSE_WAIT  0          0           off (0.00/0/0)
tcp      194      0 192.168.0.4:8920        192.168.0.100:52690     ESTABLISHED 0          0           off (0.00/0/0)
tcp        0      0 192.168.0.4:8096        192.168.0.101:39097     ESTABLISHED 116        1452860     off (0.00/0/0)
tcp      206      0 192.168.0.4:8920        179.237.25.246:54166    CLOSE_WAIT  0          0           off (0.00/0/0)
tcp        1      0 192.168.0.4:8920        179.237.25.246:63614    CLOSE_WAIT  0          0           off (0.00/0/0)
tcp      206      0 192.168.0.4:8920        179.238.113.193:55827   CLOSE_WAIT  0          0           off (0.00/0/0)
tcp        0      0 192.168.0.4:8096        192.168.0.101:39081     ESTABLISHED 116        1452793     off (0.00/0/0)
tcp        0      0 192.168.0.4:8920        192.168.0.100:52487     ESTABLISHED 116        1323323     off (0.00/0/0)
tcp        1      0 192.168.0.4:8920        179.238.113.193:55816   CLOSE_WAIT  0          0           off (0.00/0/0)
tcp      206      0 192.168.0.4:8920        179.237.25.246:54972    CLOSE_WAIT  0          0           off (0.00/0/0)
tcp        0      0 192.168.0.4:8920        179.238.115.221:58576   ESTABLISHED 116        1429674     off (0.00/0/0)
tcp        0      0 192.168.0.4:8920        192.168.0.100:52481     ESTABLISHED 116        1323154     off (0.00/0/0)
tcp        0      0 192.168.0.4:8920        179.237.25.246:50889    ESTABLISHED 0          0           off (0.00/0/0)
tcp        0      0 192.168.0.4:8920        191.46.125.223:50096    ESTABLISHED 116        1429609     off (0.00/0/0)
tcp      206      0 192.168.0.4:8920        179.237.25.246:50219    CLOSE_WAIT  0          0           off (0.00/0/0)
tcp        0      0 192.168.0.4:8096        192.168.0.101:39188     ESTABLISHED 116        1462424     off (0.00/0/0)
tcp        1      0 192.168.0.4:8920        179.237.25.246:54820    CLOSE_WAIT  0          0           off (0.00/0/0)
tcp        0      0 192.168.0.4:8920        191.40.61.190:43537     ESTABLISHED 116        1429140     off (0.00/0/0)
tcp        0      0 192.168.0.4:8096        192.168.0.101:41106     ESTABLISHED 116        1462458     off (0.00/0/0)
tcp        0      0 192.168.0.4:8096        192.168.0.101:39350     ESTABLISHED 116        1461281     off (0.00/0/0)
tcp        1      0 192.168.0.4:8920        179.237.25.246:60427    CLOSE_WAIT  0          0           off (0.00/0/0)
tcp        0      0 192.168.0.4:8096        192.168.0.101:39193     ESTABLISHED 116        1461471     off (0.00/0/0)
tcp      206      0 192.168.0.4:8920        179.238.113.193:55828   CLOSE_WAIT  0          0           off (0.00/0/0)
tcp      195      0 192.168.0.4:8920        192.168.0.100:52694     CLOSE_WAIT  0          0           off (0.00/0/0)
tcp        0      0 192.168.0.4:8920        192.168.0.100:52460     ESTABLISHED 116        1321321     off (0.00/0/0)
tcp        0      0 192.168.0.4:8096        192.168.0.101:60347     ESTABLISHED 116        1478051     off (0.00/0/0)
tcp      206      0 192.168.0.4:8920        179.238.113.193:55819   CLOSE_WAIT  0          0           off (0.00/0/0)
tcp      194      0 192.168.0.4:8920        192.168.0.100:52697     ESTABLISHED 0          0           off (0.00/0/0)
tcp        1      0 192.168.0.4:8920        179.238.113.193:55805   CLOSE_WAIT  0          0           off (0.00/0/0)
tcp      195      0 192.168.0.4:8920        192.168.0.100:52693     CLOSE_WAIT  0          0           off (0.00/0/0)
tcp      206      0 192.168.0.4:8920        179.238.113.193:55823   CLOSE_WAIT  0          0           off (0.00/0/0)
tcp      205      0 192.168.0.4:8920        179.237.25.246:63756    ESTABLISHED 0          0           off (0.00/0/0)
tcp      206      0 192.168.0.4:8920        179.237.25.246:58542    CLOSE_WAIT  0          0           off (0.00/0/0)
tcp      206      0 192.168.0.4:8920        179.237.25.246:60170    CLOSE_WAIT  0          0           off (0.00/0/0)
tcp      206      0 192.168.0.4:8920        179.238.113.193:55806   CLOSE_WAIT  0          0           off (0.00/0/0)
tcp        0      0 192.168.0.4:8920        191.40.61.190:43539     ESTABLISHED 116        1429141     off (0.00/0/0)

That is bringing my server down!

I have to restart Emby several times each day.

 

PS: Even for all those ESTABILISHED, there's something wrong. I'm not connected with several clients or connections right now. Only user a browser.

Edited by anderbytes
Link to comment
Share on other sites

anderbytes

After restarting the server... new updates of the connections,

Look at that!

root@server{/home/MyUser}: service emby restart
Stopping emby
Starting emby
root@server{/home/MyUser}: netstat -anoe | grep '8096\|8920'
tcp        0      1 192.168.0.4:8920        191.40.61.190:43533     FIN_WAIT1   0          0           on (1.69/3/0)
tcp        0      1 192.168.0.4:8920        179.238.115.221:58572   FIN_WAIT1   0          0           on (0.66/3/0)
tcp        0      1 192.168.0.4:8920        179.238.115.221:58571   FIN_WAIT1   0          0           on (0.92/3/0)
tcp        0      1 192.168.0.4:8920        179.238.115.221:58567   FIN_WAIT1   0          0           on (1.11/3/0)
tcp        0      1 192.168.0.4:8920        192.168.0.100:52473     FIN_WAIT1   0          0           on (0.66/3/0)
tcp        0      0 192.168.0.4:8096        192.168.0.101:36872     FIN_WAIT2   0          0           timewait (56.99/0/0)
tcp        0      1 192.168.0.4:8920        179.238.115.221:58577   FIN_WAIT1   0          0           on (1.18/3/0)
tcp        0      1 192.168.0.4:8920        177.176.190.222:39878   FIN_WAIT1   0          0           on (1.11/3/0)
tcp        0      1 192.168.0.4:8920        179.238.115.221:60005   FIN_WAIT1   0          0           on (0.44/2/0)
tcp        0      1 192.168.0.4:8920        179.238.115.221:58568   FIN_WAIT1   0          0           on (1.75/3/0)
tcp        0      1 192.168.0.4:8920        192.168.0.100:52482     FIN_WAIT1   0          0           on (2.81/3/0)
tcp        0      1 192.168.0.4:8920        191.40.61.190:43538     FIN_WAIT1   0          0           on (0.70/2/0)
tcp        0      1 192.168.0.4:8920        179.238.115.221:58929   FIN_WAIT1   0          0           on (2.99/2/0)
tcp        0      1 192.168.0.4:8920        192.168.0.100:52464     FIN_WAIT1   0          0           on (0.40/3/0)
tcp        0      1 192.168.0.4:8920        179.238.115.221:58565   FIN_WAIT1   0          0           on (2.52/3/0)
tcp        0      0 192.168.0.4:8096        192.168.0.101:50352     FIN_WAIT2   0          0           timewait (56.99/0/0)
tcp        0      1 192.168.0.4:8920        191.40.61.190:43536     FIN_WAIT1   0          0           on (3.25/2/0)
tcp        0      0 192.168.0.4:8096        192.168.0.4:58925       TIME_WAIT   0          0           timewait (56.82/0/0)
tcp        0      1 192.168.0.4:8920        192.168.0.100:52487     FIN_WAIT1   0          0           on (2.92/3/0)
tcp        0      1 192.168.0.4:8920        179.238.115.221:58576   FIN_WAIT1   0          0           on (1.50/3/0)
tcp        0      1 192.168.0.4:8920        192.168.0.100:52481     FIN_WAIT1   0          0           on (1.72/3/0)
tcp        0      1 192.168.0.4:8920        191.46.125.223:50096    FIN_WAIT1   0          0           on (0.40/2/0)
tcp        0      1 192.168.0.4:8920        191.40.61.190:43537     FIN_WAIT1   0          0           on (2.08/1/0)
tcp        0      1 192.168.0.4:8920        192.168.0.100:52460     FIN_WAIT1   0          0           on (2.31/2/0)
tcp        0      0 192.168.0.4:8096        192.168.0.101:60347     FIN_WAIT2   0          0           timewait (56.99/0/0)
tcp        0      1 192.168.0.4:8920        191.40.61.190:43539     FIN_WAIT1   0          0           on (3.59/1/0)
Link to comment
Share on other sites

  • 2 weeks later...
anderbytes

the server closes all of it's sockets, although it's always possible there's an issue in a library we're using.

 

@@Luke , please take a deeper look into this.

 

1. Connecting from external network (haven't tested internally yet)

2. I´ve opened Emby Web Client from Firefox (yeah... I know... it's not Chrome)

3. One new connection appeared in NETSTAT... as Estabilished... OK

4. Closed the browser tab where Web Client was

5. Connection remains opened forever as estabilished  :-(

 

I've noticed that there's no usage of "SetTcpKeepAlive" in your code when calling Mono, as implemented in Mono a long time ago: http://marc.info/?l=mono-patches&m=129496132601658

 

https://github.com/mono/mono/blob/082a47eed5dd823191e10ffefa57274a660e60e4/mcs/class/System/System.Net/ServicePoint.cs#L191-L215

 

So... with this enabled, I guess it will be possible for Mono or Emby to check if a connection is really active or not.

 

 

With this implemented... hopefully we won't have more problems with unused connections... be it in ESTABILISHED or CLOSE_WAIT state

Edited by anderbytes
Link to comment
Share on other sites

anderbytes

@@Luke and @@Happy2Play , please consider this issue as serious. I now am forced to restart my server frequently as 2x a day just because of the piled up unused connections.

 

Probably people that run Emby in Linux may be having the same problem.... but just restart the service without knowing what's wrong.

 

 

Looking at the code in GitHub and doing some research about zombie CLOSE_WAIT and zombie ESTABILISHED, I guess that the problem is around HttpListener , HttpClient and ConnectionManager.

 

Can you please read this below, and maybe consider using   MultiThreadedHttpConnectionManager  to allow the reuse of sockets (instead of alwars creating new ones)???

 

To be read: http://www.nuxeo.com/blog/using-httpclient-properly-avoid-closewait-tcp-connections/

 

I can't explain in more details... but in short, the opened sockets by Emby converse asynchronously with the clients, and when one of those cliente close and for some reason can't contact the server (broken network connection, client crash, etc...) to tell him that he should also close the connection, creates a condition where the server has ports that will never again receive data, and will never auto-close, too. Because they hope that one day they will be contacted by it's faithful client

 

 

- Is the MultiThreadedHttpConnectionManager solution above useful for Emby?

- Is it possible to "ping" the connection once a minute to know if there's someone at the other side? And then force close the socket if noone responds.

 

THANKS!

 

ps: take a look again at post #6 above. There was only me using the server. That's monstruous!

Edited by anderbytes
Link to comment
Share on other sites

Redshirt

I had no idea this was happening, wow. I've got 30+ connections from only 2 android devices.

 

I'm not sure it's possible to entirely close connections with Android currently. The sync account is always running in the background polling at default intervals and then there's the websocket updates from the server.

Edited by Redshirt
Link to comment
Share on other sites

anderbytes

I had no idea this was happening, wow. I've got 30+ connections from only 2 android devices.

 

I'm not sure it's possible to entirely close connections with Android currently. The sync account is always running in the background polling at default intervals and then there's the websocket updates from the server.

Even with those, there would be only a few connections active.

Link to comment
Share on other sites

anderbytes

this is probably just due to keep-alive, otherwise known as persistent connections.

Okay, but they are TOO persistent. :-/

 

Data conversations are broken but server stills thinks that socket will be useful, when it's not, just using resources.

 

Can't you sanitize periodically (each X minutes) network sockets opened by Emby?

Link to comment
Share on other sites

MSattler

this is probably just due to keep-alive, otherwise known as persistent connections. 

 

Would it not make sense to look into it further instead of just dismissing it completely?

Link to comment
Share on other sites

MSattler

Okay, but they are TOO persistent. :-/

 

Data conversations are broken but server stills thinks that socket will be useful, when it's not, just using resources.

 

Can't you sanitize periodically (each X minutes) network sockets opened by Emby?

 

Agreed, and along with the fact that they are waiting in Fin_wait after the application cycles.  Shouldn't all those connections be cleared the minute the service stops?

Link to comment
Share on other sites

anderbytes

As far as I've seen, FIN_WAIT always get auto-closed very quickly, so maybe that could be ignored... I guess.

  • Like 1
Link to comment
Share on other sites

dethknite

Yea, TCP connections that are not properly closed (TIMEWAIT) status will hang around until the OS kills them.  Windows has a registry key that determines that (default 240 seconds).

 

.\HKLM\SYSTEM\CurrentControlSet\Services\Tcpip\Parameters  --> TcpTimedWaitDelay

 

I am not personally experiencing this issue though.  Perhaps there is a firewall issue or something else causing this...

Edited by dethknite
Link to comment
Share on other sites

anderbytes

Yea, TCP connections that are not properly closed (TIMEWAIT) status will hang around until the OS kills them. Windows has a registry key that determines that (default 240 seconds).

 

.\HKLM\SYSTEM\CurrentControlSet\Services\Tcpip\Parameters --> TcpTimedWaitDelay

 

I am not personally experiencing this issue though. Perhaps there is a firewall issue or something else causing this...

You are talking about Windows.

My Debian Linux is working differently, he doesn't close it by itself. Maybe it's supposed to work that way.

 

Maybe some Linux expert could try to investigate the reason...

Link to comment
Share on other sites

anderbytes

If you are a Linux user and read this... please try to reproduce the issue with the ports, and tell us if happened or not... and whats your distribution.

 

Thanks for all.

Link to comment
Share on other sites

anderbytes

i'm adjusting the keep alive behavior. you can play with it in upcoming dev and beta builds.

OK Luke. Because of this, for the first time I'll run the beta version.

Link to comment
Share on other sites

kjp4756

I'm not seeing this issue on freebsd.  I wonder if emby is relying on the OS to close the inactive ports.  Windows and freebsd appear to handle this fine.  

 

I did find that after restarting emby the fin_wait sticks around for a while.

Link to comment
Share on other sites

runtimesandbox

As discussed in this thread http://emby.media/community/index.php?/topic/31089-emby-server-https-stops-responding/ i can no longer access the server via the https port 8920

 

Running the netstat command I get the following output

root@media-server:/home/ms# netstat -anoe | grep ":8"
tcp        0      0 0.0.0.0:8920            0.0.0.0:*               LISTEN      109        1961283     off (0.00/0/0)
tcp        0      0 0.0.0.0:8096            0.0.0.0:*               LISTEN      109        1961282     off (0.00/0/0)
tcp      195      0 192.168.1.5:8920        10.0.0.1:59111          ESTABLISHED 0          0           off (0.00/0/0)
tcp      195      0 192.168.1.5:8920        10.0.0.1:45621          ESTABLISHED 0          0           off (0.00/0/0)
tcp      195      0 192.168.1.5:8920        10.0.0.1:11983          ESTABLISHED 0          0           off (0.00/0/0)
tcp        0      0 192.168.1.5:8096        10.0.0.174:46062        ESTABLISHED 109        2165503     off (0.00/0/0)
tcp        0      0 192.168.1.5:8096        10.0.0.174:46067        ESTABLISHED 109        2175570     off (0.00/0/0)
tcp      202      0 192.168.1.5:8920        10.0.0.1:34930          ESTABLISHED 0          0           off (0.00/0/0)
tcp      195      0 192.168.1.5:8920        94.197.121.227:10496    ESTABLISHED 0          0           off (0.00/0/0)
tcp      195      0 192.168.1.5:8920        94.197.121.227:10493    ESTABLISHED 0          0           off (0.00/0/0)
tcp      195      0 192.168.1.5:8920        10.0.0.1:48591          ESTABLISHED 0          0           off (0.00/0/0)
tcp        0      0 192.168.1.5:8096        10.0.0.174:46065        ESTABLISHED 109        2168277     off (0.00/0/0)
tcp        0   1343 192.168.1.5:8920        94.197.121.227:9659     CLOSE_WAIT  109        2200931     on (33.93/10/0)
tcp      195      0 192.168.1.5:8920        10.0.0.1:45620          ESTABLISHED 0          0           off (0.00/0/0)
tcp        0      0 192.168.1.5:8096        10.0.0.174:46058        ESTABLISHED 109        2169134     off (0.00/0/0)
tcp      195      0 192.168.1.5:8920        10.0.0.1:18302          ESTABLISHED 0          0           off (0.00/0/0)
tcp      195      0 192.168.1.5:8920        10.0.0.1:60620          ESTABLISHED 0          0           off (0.00/0/0)
tcp      195      0 192.168.1.5:8920        10.0.0.1:45619          ESTABLISHED 0          0           off (0.00/0/0)
tcp      195      0 192.168.1.5:8920        10.0.0.1:59880          ESTABLISHED 0          0           off (0.00/0/0)
tcp        0      0 192.168.1.5:8096        10.0.0.34:47670         ESTABLISHED 109        1960296     off (0.00/0/0)
tcp      195      0 192.168.1.5:8920        10.0.0.1:60613          ESTABLISHED 0          0           off (0.00/0/0)
tcp      195      0 192.168.1.5:8920        94.197.121.227:10494    ESTABLISHED 0          0           off (0.00/0/0)
tcp      195      0 192.168.1.5:8920        10.0.0.1:45622          ESTABLISHED 0          0           off (0.00/0/0)
tcp        0      0 192.168.1.5:8096        10.0.0.160:37829        ESTABLISHED 109        2199591     off (0.00/0/0)
tcp        0      0 192.168.1.5:8096        10.0.0.174:46064        ESTABLISHED 109        2165506     off (0.00/0/0)
tcp      195      0 192.168.1.5:8920        10.0.0.1:37133          ESTABLISHED 0          0           off (0.00/0/0)
tcp        0      0 192.168.1.5:8096        10.0.0.174:46476        ESTABLISHED 109        2195978     off (0.00/0/0)
tcp        0      0 192.168.1.5:8096        10.0.0.227:21153        ESTABLISHED 109        2201500     off (0.00/0/0)
tcp        1      0 192.168.1.5:53510       104.28.22.116:80        CLOSE_WAIT  109        2201499     off (0.00/0/0)
tcp      195      0 192.168.1.5:8920        10.0.0.1:60623          ESTABLISHED 0          0           off (0.00/0/0)
tcp      195      0 192.168.1.5:8920        10.0.0.1:46350          ESTABLISHED 0          0           off (0.00/0/0)
tcp        0      0 192.168.1.5:8096        10.0.0.174:45568        ESTABLISHED 109        2188130     off (0.00/0/0)
tcp        0      0 192.168.1.5:8096        10.0.0.174:46060        ESTABLISHED 109        2168275     off (0.00/0/0)
tcp        0   1336 192.168.1.5:8920        94.197.121.227:10492    ESTABLISHED 109        2200971     on (15.50/10/0)
tcp      195      0 192.168.1.5:8920        10.0.0.1:45616          ESTABLISHED 0          0           off (0.00/0/0)
tcp      195      0 192.168.1.5:8920        10.0.0.1:9834           ESTABLISHED 0          0           off (0.00/0/0)
tcp      195      0 192.168.1.5:8920        10.0.0.1:45617          ESTABLISHED 0          0           off (0.00/0/0)
tcp        0      0 192.168.1.5:8096        10.0.0.174:46063        ESTABLISHED 109        2165504     off (0.00/0/0)
tcp      195      0 192.168.1.5:8920        10.0.0.1:26604          ESTABLISHED 0          0           off (0.00/0/0)
tcp        0      0 192.168.1.5:54464       192.168.1.5:8096        ESTABLISHED 109        2199420     off (0.00/0/0)
tcp      195      0 192.168.1.5:8920        10.0.0.1:60622          ESTABLISHED 0          0           off (0.00/0/0)
tcp      195      0 192.168.1.5:8920        10.0.0.1:27367          ESTABLISHED 0          0           off (0.00/0/0)
tcp      202      0 192.168.1.5:8920        10.0.0.1:34925          ESTABLISHED 0          0           off (0.00/0/0)
tcp        0      0 192.168.1.5:8096        10.0.0.22:34772         ESTABLISHED 109        1957632     off (0.00/0/0)
tcp      195      0 192.168.1.5:8920        10.0.0.1:60618          ESTABLISHED 0          0           off (0.00/0/0)
tcp      195      0 192.168.1.5:8920        10.0.0.1:60625          ESTABLISHED 0          0           off (0.00/0/0)
tcp      195      0 192.168.1.5:8920        10.0.0.1:60615          ESTABLISHED 0          0           off (0.00/0/0)
tcp      195      0 192.168.1.5:8920        10.0.0.1:45618          ESTABLISHED 0          0           off (0.00/0/0)
tcp      195      0 192.168.1.5:8920        10.0.0.1:16838          ESTABLISHED 0          0           off (0.00/0/0)
tcp      202      0 192.168.1.5:8920        10.0.0.1:34927          ESTABLISHED 0          0           off (0.00/0/0)
tcp      195      0 192.168.1.5:8920        94.197.121.227:10497    ESTABLISHED 0          0           off (0.00/0/0)
tcp      195      0 192.168.1.5:8920        10.0.0.1:45623          ESTABLISHED 0          0           off (0.00/0/0)
tcp      195      0 192.168.1.5:8920        94.197.121.227:10495    ESTABLISHED 0          0           off (0.00/0/0)
tcp        0      0 192.168.1.5:8096        10.0.0.174:46066        ESTABLISHED 109        2165533     off (0.00/0/0)
tcp      195      0 192.168.1.5:8920        10.0.0.1:9109           ESTABLISHED 0          0           off (0.00/0/0)
tcp      195      0 192.168.1.5:8920        10.0.0.1:17598          ESTABLISHED 0          0           off (0.00/0/0)
tcp      195      0 192.168.1.5:8920        10.0.0.1:60614          ESTABLISHED 0          0           off (0.00/0/0)
tcp        0      0 192.168.1.5:8096        10.0.0.34:47668         ESTABLISHED 109        1960295     off (0.00/0/0)
tcp      195      0 192.168.1.5:8920        10.0.0.1:60624          ESTABLISHED 0          0           off (0.00/0/0)
tcp      195      0 192.168.1.5:8920        10.0.0.1:2792           ESTABLISHED 0          0           off (0.00/0/0)
tcp        0      0 192.168.1.5:8096        10.0.0.22:34770         ESTABLISHED 109        1961301     off (0.00/0/0)
tcp      202      0 192.168.1.5:8920        10.0.0.1:34926          ESTABLISHED 0          0           off (0.00/0/0)
tcp        0      0 192.168.1.5:8096        10.0.0.227:21158        ESTABLISHED 109        2201497     off (0.00/0/0)
tcp      195      0 192.168.1.5:8920        10.0.0.1:60619          ESTABLISHED 0          0           off (0.00/0/0)
tcp        0      0 192.168.1.5:8096        192.168.1.5:54464       ESTABLISHED 109        2201432     off (0.00/0/0)
tcp      195      0 192.168.1.5:8920        10.0.0.1:35262          ESTABLISHED 0          0           off (0.00/0/0)
tcp      195      0 192.168.1.5:8920        10.0.0.1:60621          ESTABLISHED 0          0           off (0.00/0/0)
tcp      195      0 192.168.1.5:8920        10.0.0.1:53546          ESTABLISHED 0          0           off (0.00/0/0)
tcp      202      0 192.168.1.5:8920        10.0.0.1:34929          ESTABLISHED 0          0           off (0.00/0/0)
tcp        0      0 192.168.1.5:8096        10.0.0.174:46061        ESTABLISHED 109        2168276     off (0.00/0/0)
tcp      202      0 192.168.1.5:8920        10.0.0.1:34928          ESTABLISHED 0          0           off (0.00/0/0)
tcp      195      0 192.168.1.5:8920        10.0.0.1:60616          ESTABLISHED 0          0           off (0.00/0/0)
tcp      195      0 192.168.1.5:8920        10.0.0.1:60617          ESTABLISHED 0          0           off (0.00/0/0)

This issue has appeared on both ubuntu 14 and debian 8 (64 bit)

 

EDIT: The external ip was a mobile connection i attempted from over 3 days ago!

 

After rebooting the server I get these FIN_WAIT connections

root@media-server:/home/ms# netstat -anoe | grep ":8"
tcp        0   1337 192.168.1.5:8920        94.197.121.227:10493    FIN_WAIT1   0          0           on (17.45/10/0)
tcp        0      0 192.168.1.5:8096        10.0.0.227:21793        TIME_WAIT   0          0           timewait (58.41/0/0)
tcp        0      0 192.168.1.5:8096        10.0.0.34:47670         FIN_WAIT2   0          0           timewait (58.83/0/0)
tcp        0      0 192.168.1.5:8096        10.0.0.160:37829        FIN_WAIT2   0          0           timewait (58.94/0/0)
tcp        0      0 192.168.1.5:8096        10.0.0.174:46476        FIN_WAIT2   0          0           timewait (59.02/0/0)
tcp        0      0 192.168.1.5:8096        10.0.0.227:21790        TIME_WAIT   0          0           timewait (57.40/0/0)
tcp        0      0 192.168.1.5:8096        10.0.0.227:21789        TIME_WAIT   0          0           timewait (57.90/0/0)
tcp        0      0 192.168.1.5:8096        10.0.0.227:21795        TIME_WAIT   0          0           timewait (57.28/0/0)
tcp        0      0 192.168.1.5:8096        10.0.0.174:58549        FIN_WAIT2   0          0           timewait (59.02/0/0)
tcp        0      0 192.168.1.5:8096        10.0.0.22:34772         FIN_WAIT2   0          0           timewait (58.83/0/0)
tcp        0      0 192.168.1.5:8096        10.0.0.22:34770         TIME_WAIT   0          0           timewait (57.27/0/0)
tcp        0      0 192.168.1.5:54472       192.168.1.5:8096        TIME_WAIT   0          0           timewait (7.63/0/0)

Edited by spudy12
Link to comment
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now
×
×
  • Create New...