anderbytes 139 Posted February 8, 2016 Share Posted February 8, 2016 (edited) Hello guys, I guess I've found a nasty bug that can degrade overall Emby performance. Server: v3.0.5821 (latest stable) I accidentaly found that in some cases the ports that Emby use to allow clients connections aren`t closed as it should, after being used. Today by morning I was trying to access the WebClient interface... and it wouldn`t connect at all...at didn't returned any error, either.I confirmed that 8096 and 8920 were been listened by Emby, and both server and client were correctly connected to internal network. So... after some diagnose, I discovered that several connections were estabilished. I used the command netstat -anoe | grep ":8" in my server.While one TV was using the direct streaming, I couldn't confirm it was normal or not (strange, at most). When restarted the server, the open ports cleared, and I could connect again. Now, several hours later, with no TV or AndroidApp or Theater connected. I ran the same command and confirmed my worries: tcp 0 0 0.0.0.0:8096 0.0.0.0:* LISTEN 116 312514 off (0.00/0/0) tcp 0 0 0.0.0.0:8920 0.0.0.0:* LISTEN 116 312515 off (0.00/0/0) tcp 0 0 192.168.0.4:8096 192.168.0.101:60651 ESTABLISHED 116 316922 off (0.00/0/0) tcp 0 0 192.168.0.4:8096 192.168.0.4:53790 ESTABLISHED 116 379329 off (0.00/0/0) tcp 0 0 192.168.0.4:8096 192.168.0.107:36751 ESTABLISHED 116 307935 off (0.00/0/0) tcp 0 0 192.168.0.4:8096 192.168.0.101:43291 ESTABLISHED 116 350878 off (0.00/0/0) tcp 0 0 192.168.0.4:8096 192.168.0.107:36766 ESTABLISHED 116 312806 off (0.00/0/0) tcp 0 0 192.168.0.4:53790 192.168.0.4:8096 ESTABLISHED 116 378719 off (0.00/0/0) Not even IP .101 or .107 are online anymore since 12pm (4 hours agora). But their used ports are still opened and using O.S. resources. Can anyone reproduce this? Are server logs necessary? I'm not fond of distributing logs for every reason, as I'm paranoic about security. UPDATE: I noticed that every time I refresh the webgui (F5 in browser), it opens new ports and doesn`t closed the others, keeping the ESTABILISHED status working 4ever :-(((( UPDATE 2: After closing the browser, all connections get closed. Well... okay for now. But I forgot to tell one thing. IPs .101 and .107 above were respectively an Android Emby App and a Samsung Emby App. Edited February 8, 2016 by anderbytes Link to comment Share on other sites More sharing options...
anderbytes 139 Posted February 12, 2016 Author Share Posted February 12, 2016 I just had this problem again. I'm in emby connect (Android client), and opened ports now piled up in close_wait state and never vanish. This is taking lots of resources and doesn't let me connect again Please read this: http://blogs.technet.com/b/janelewis/archive/2010/03/09/explaining-close-wait.aspx See the close_wait part. That is just like what I'm seeing here. Can you confirm the server manages to close its ports correctly? How is this enforced? Thanks in advance Link to comment Share on other sites More sharing options...
Luke 37065 Posted February 12, 2016 Share Posted February 12, 2016 the server closes all of it's sockets, although it's always possible there's an issue in a library we're using. Link to comment Share on other sites More sharing options...
anderbytes 139 Posted February 12, 2016 Author Share Posted February 12, 2016 (edited) When using netstat command with -anoe parameter you can see the CLOSE_WAIT countdown to auto-close, on each connection. In the cases I described, they already reached zero and still don't go away :-( Edited February 12, 2016 by anderbytes Link to comment Share on other sites More sharing options...
anderbytes 139 Posted February 12, 2016 Author Share Posted February 12, 2016 If you help me find a better way to analyze it, I can try to diagnose better. Will debug mode help me somehow? Does it log when a port is closed by the server? Link to comment Share on other sites More sharing options...
anderbytes 139 Posted February 13, 2016 Author Share Posted February 13, 2016 (edited) Another output for netstat. The same connections are there for hours, since my last messages... root@server{/home/MyUser}: netstat -anoe | grep '8096\|8920' tcp 0 0 0.0.0.0:8096 0.0.0.0:* LISTEN 116 1320658 off (0.00/0/0) tcp 0 0 0.0.0.0:8920 0.0.0.0:* LISTEN 116 1320659 off (0.00/0/0) tcp 0 0 192.168.0.4:8920 191.40.61.190:43533 ESTABLISHED 116 1429136 off (0.00/0/0) tcp 0 0 192.168.0.4:8096 192.168.0.101:39181 ESTABLISHED 116 1461463 off (0.00/0/0) tcp 1 0 192.168.0.4:8920 179.238.113.193:55820 CLOSE_WAIT 0 0 off (0.00/0/0) tcp 0 0 192.168.0.4:8920 179.238.115.221:58572 ESTABLISHED 116 1429673 off (0.00/0/0) tcp 206 0 192.168.0.4:8920 179.237.25.246:57697 CLOSE_WAIT 0 0 off (0.00/0/0) tcp 0 0 192.168.0.4:8920 179.238.115.221:58571 ESTABLISHED 116 1429068 off (0.00/0/0) tcp 0 0 192.168.0.4:8096 192.168.0.101:39191 ESTABLISHED 116 1462425 off (0.00/0/0) tcp 0 0 192.168.0.4:8096 192.168.0.101:39078 ESTABLISHED 116 1452792 off (0.00/0/0) tcp 182 0 192.168.0.4:8920 179.237.25.246:52182 CLOSE_WAIT 0 0 off (0.00/0/0) tcp 0 0 192.168.0.4:8920 179.238.115.221:58567 ESTABLISHED 116 1429641 off (0.00/0/0) tcp 0 0 192.168.0.4:8920 192.168.0.100:52473 ESTABLISHED 116 1320673 off (0.00/0/0) tcp 0 0 192.168.0.4:8096 192.168.0.101:36872 ESTABLISHED 116 1477012 off (0.00/0/0) tcp 0 0 192.168.0.4:8920 179.238.115.221:58577 ESTABLISHED 116 1429071 off (0.00/0/0) tcp 0 0 192.168.0.4:8096 192.168.0.106:52576 ESTABLISHED 116 1466172 off (0.00/0/0) tcp 0 0 192.168.0.4:8920 177.176.190.222:39878 ESTABLISHED 116 1430107 off (0.00/0/0) tcp 206 0 192.168.0.4:8920 179.237.25.246:49447 CLOSE_WAIT 0 0 off (0.00/0/0) tcp 0 0 192.168.0.4:8096 192.168.0.101:39190 ESTABLISHED 116 1461470 off (0.00/0/0) tcp 0 0 192.168.0.4:8096 192.168.0.101:39194 ESTABLISHED 116 1461474 off (0.00/0/0) tcp 0 0 192.168.0.4:8920 179.238.115.221:60005 ESTABLISHED 116 1429075 off (0.00/0/0) tcp 0 0 192.168.0.4:8096 192.168.0.101:39192 ESTABLISHED 116 1462426 off (0.00/0/0) tcp 0 0 192.168.0.4:8920 179.238.115.221:58568 ESTABLISHED 116 1430564 off (0.00/0/0) tcp 0 0 192.168.0.4:8920 192.168.0.100:52482 ESTABLISHED 116 1320676 off (0.00/0/0) tcp 206 0 192.168.0.4:8920 179.237.25.246:50649 CLOSE_WAIT 0 0 off (0.00/0/0) tcp 0 0 192.168.0.4:8096 192.168.0.101:39100 ESTABLISHED 116 1455228 off (0.00/0/0) tcp 206 0 192.168.0.4:8920 179.238.113.193:55818 CLOSE_WAIT 0 0 off (0.00/0/0) tcp 0 0 192.168.0.4:8920 191.40.61.190:43538 ESTABLISHED 116 1429792 off (0.00/0/0) tcp 0 0 192.168.0.4:8096 192.168.0.101:39099 ESTABLISHED 116 1455227 off (0.00/0/0) tcp 0 0 192.168.0.4:8096 192.168.0.101:39079 ESTABLISHED 116 1453867 off (0.00/0/0) tcp 0 0 192.168.0.4:8920 179.238.115.221:58929 ESTABLISHED 116 1429698 off (0.00/0/0) tcp 0 0 192.168.0.4:8096 192.168.0.101:39189 ESTABLISHED 116 1461469 off (0.00/0/0) tcp 206 0 192.168.0.4:8920 179.237.25.246:51235 CLOSE_WAIT 0 0 off (0.00/0/0) tcp 0 0 192.168.0.4:8096 192.168.0.101:39098 ESTABLISHED 116 1455226 off (0.00/0/0) tcp 0 0 192.168.0.4:8920 192.168.0.100:52464 ESTABLISHED 116 1323072 off (0.00/0/0) tcp 0 0 192.168.0.4:8096 192.168.0.101:39077 ESTABLISHED 116 1451975 off (0.00/0/0) tcp 183 0 192.168.0.4:8920 192.168.0.100:52696 CLOSE_WAIT 0 0 off (0.00/0/0) tcp 206 0 192.168.0.4:8920 179.238.113.193:55817 CLOSE_WAIT 0 0 off (0.00/0/0) tcp 206 0 192.168.0.4:8920 179.238.113.193:55804 CLOSE_WAIT 0 0 off (0.00/0/0) tcp 206 0 192.168.0.4:8920 179.237.25.246:61568 CLOSE_WAIT 0 0 off (0.00/0/0) tcp 0 0 192.168.0.4:58750 192.168.0.4:8096 ESTABLISHED 116 1482388 off (0.00/0/0) tcp 0 0 192.168.0.4:8920 179.238.115.221:58565 ESTABLISHED 116 1427677 off (0.00/0/0) tcp 0 0 192.168.0.4:8096 192.168.0.101:39080 ESTABLISHED 116 1451977 off (0.00/0/0) tcp 0 0 192.168.0.4:8096 192.168.0.4:58750 ESTABLISHED 116 1483981 off (0.00/0/0) tcp 206 0 192.168.0.4:8920 179.237.25.246:59730 CLOSE_WAIT 0 0 off (0.00/0/0) tcp 0 0 192.168.0.4:8096 192.168.0.101:50352 ESTABLISHED 116 1479029 off (0.00/0/0) tcp 206 0 192.168.0.4:8920 179.237.25.246:52397 CLOSE_WAIT 0 0 off (0.00/0/0) tcp 182 0 192.168.0.4:8920 179.237.25.246:54390 CLOSE_WAIT 0 0 off (0.00/0/0) tcp 0 0 192.168.0.4:8096 192.168.0.101:39096 ESTABLISHED 116 1452856 off (0.00/0/0) tcp 0 0 192.168.0.4:8096 192.168.0.101:53523 ESTABLISHED 116 1452820 off (0.00/0/0) tcp 206 0 192.168.0.4:8920 179.237.25.246:53115 CLOSE_WAIT 0 0 off (0.00/0/0) tcp 206 0 192.168.0.4:8920 179.237.25.246:57447 CLOSE_WAIT 0 0 off (0.00/0/0) tcp 206 0 192.168.0.4:8920 179.237.25.246:49274 CLOSE_WAIT 0 0 off (0.00/0/0) tcp 0 0 192.168.0.4:8096 192.168.0.101:38707 ESTABLISHED 116 1456191 off (0.00/0/0) tcp 206 0 192.168.0.4:8920 179.238.113.193:55821 CLOSE_WAIT 0 0 off (0.00/0/0) tcp 206 0 192.168.0.4:8920 179.237.25.246:54499 CLOSE_WAIT 0 0 off (0.00/0/0) tcp 0 0 192.168.0.4:8920 191.40.61.190:43536 ESTABLISHED 116 1429139 off (0.00/0/0) tcp 1 0 192.168.0.4:8920 179.237.25.246:62790 CLOSE_WAIT 0 0 off (0.00/0/0) tcp 0 0 192.168.0.4:8096 192.168.0.101:41741 ESTABLISHED 116 1462456 off (0.00/0/0) tcp 0 0 192.168.0.4:8096 192.168.0.106:52587 ESTABLISHED 116 1466206 off (0.00/0/0) tcp 206 0 192.168.0.4:8920 179.238.113.193:55815 CLOSE_WAIT 0 0 off (0.00/0/0) tcp 195 0 192.168.0.4:8920 192.168.0.100:52695 CLOSE_WAIT 0 0 off (0.00/0/0) tcp 206 0 192.168.0.4:8920 179.237.25.246:64745 CLOSE_WAIT 0 0 off (0.00/0/0) tcp 1 0 192.168.0.4:8920 179.237.25.246:53549 CLOSE_WAIT 0 0 off (0.00/0/0) tcp 206 0 192.168.0.4:8920 179.238.113.193:55822 CLOSE_WAIT 0 0 off (0.00/0/0) tcp 194 0 192.168.0.4:8920 192.168.0.100:52690 ESTABLISHED 0 0 off (0.00/0/0) tcp 0 0 192.168.0.4:8096 192.168.0.101:39097 ESTABLISHED 116 1452860 off (0.00/0/0) tcp 206 0 192.168.0.4:8920 179.237.25.246:54166 CLOSE_WAIT 0 0 off (0.00/0/0) tcp 1 0 192.168.0.4:8920 179.237.25.246:63614 CLOSE_WAIT 0 0 off (0.00/0/0) tcp 206 0 192.168.0.4:8920 179.238.113.193:55827 CLOSE_WAIT 0 0 off (0.00/0/0) tcp 0 0 192.168.0.4:8096 192.168.0.101:39081 ESTABLISHED 116 1452793 off (0.00/0/0) tcp 0 0 192.168.0.4:8920 192.168.0.100:52487 ESTABLISHED 116 1323323 off (0.00/0/0) tcp 1 0 192.168.0.4:8920 179.238.113.193:55816 CLOSE_WAIT 0 0 off (0.00/0/0) tcp 206 0 192.168.0.4:8920 179.237.25.246:54972 CLOSE_WAIT 0 0 off (0.00/0/0) tcp 0 0 192.168.0.4:8920 179.238.115.221:58576 ESTABLISHED 116 1429674 off (0.00/0/0) tcp 0 0 192.168.0.4:8920 192.168.0.100:52481 ESTABLISHED 116 1323154 off (0.00/0/0) tcp 0 0 192.168.0.4:8920 179.237.25.246:50889 ESTABLISHED 0 0 off (0.00/0/0) tcp 0 0 192.168.0.4:8920 191.46.125.223:50096 ESTABLISHED 116 1429609 off (0.00/0/0) tcp 206 0 192.168.0.4:8920 179.237.25.246:50219 CLOSE_WAIT 0 0 off (0.00/0/0) tcp 0 0 192.168.0.4:8096 192.168.0.101:39188 ESTABLISHED 116 1462424 off (0.00/0/0) tcp 1 0 192.168.0.4:8920 179.237.25.246:54820 CLOSE_WAIT 0 0 off (0.00/0/0) tcp 0 0 192.168.0.4:8920 191.40.61.190:43537 ESTABLISHED 116 1429140 off (0.00/0/0) tcp 0 0 192.168.0.4:8096 192.168.0.101:41106 ESTABLISHED 116 1462458 off (0.00/0/0) tcp 0 0 192.168.0.4:8096 192.168.0.101:39350 ESTABLISHED 116 1461281 off (0.00/0/0) tcp 1 0 192.168.0.4:8920 179.237.25.246:60427 CLOSE_WAIT 0 0 off (0.00/0/0) tcp 0 0 192.168.0.4:8096 192.168.0.101:39193 ESTABLISHED 116 1461471 off (0.00/0/0) tcp 206 0 192.168.0.4:8920 179.238.113.193:55828 CLOSE_WAIT 0 0 off (0.00/0/0) tcp 195 0 192.168.0.4:8920 192.168.0.100:52694 CLOSE_WAIT 0 0 off (0.00/0/0) tcp 0 0 192.168.0.4:8920 192.168.0.100:52460 ESTABLISHED 116 1321321 off (0.00/0/0) tcp 0 0 192.168.0.4:8096 192.168.0.101:60347 ESTABLISHED 116 1478051 off (0.00/0/0) tcp 206 0 192.168.0.4:8920 179.238.113.193:55819 CLOSE_WAIT 0 0 off (0.00/0/0) tcp 194 0 192.168.0.4:8920 192.168.0.100:52697 ESTABLISHED 0 0 off (0.00/0/0) tcp 1 0 192.168.0.4:8920 179.238.113.193:55805 CLOSE_WAIT 0 0 off (0.00/0/0) tcp 195 0 192.168.0.4:8920 192.168.0.100:52693 CLOSE_WAIT 0 0 off (0.00/0/0) tcp 206 0 192.168.0.4:8920 179.238.113.193:55823 CLOSE_WAIT 0 0 off (0.00/0/0) tcp 205 0 192.168.0.4:8920 179.237.25.246:63756 ESTABLISHED 0 0 off (0.00/0/0) tcp 206 0 192.168.0.4:8920 179.237.25.246:58542 CLOSE_WAIT 0 0 off (0.00/0/0) tcp 206 0 192.168.0.4:8920 179.237.25.246:60170 CLOSE_WAIT 0 0 off (0.00/0/0) tcp 206 0 192.168.0.4:8920 179.238.113.193:55806 CLOSE_WAIT 0 0 off (0.00/0/0) tcp 0 0 192.168.0.4:8920 191.40.61.190:43539 ESTABLISHED 116 1429141 off (0.00/0/0) That is bringing my server down! I have to restart Emby several times each day. PS: Even for all those ESTABILISHED, there's something wrong. I'm not connected with several clients or connections right now. Only user a browser. Edited February 13, 2016 by anderbytes Link to comment Share on other sites More sharing options...
anderbytes 139 Posted February 13, 2016 Author Share Posted February 13, 2016 After restarting the server... new updates of the connections, Look at that! root@server{/home/MyUser}: service emby restart Stopping emby Starting emby root@server{/home/MyUser}: netstat -anoe | grep '8096\|8920' tcp 0 1 192.168.0.4:8920 191.40.61.190:43533 FIN_WAIT1 0 0 on (1.69/3/0) tcp 0 1 192.168.0.4:8920 179.238.115.221:58572 FIN_WAIT1 0 0 on (0.66/3/0) tcp 0 1 192.168.0.4:8920 179.238.115.221:58571 FIN_WAIT1 0 0 on (0.92/3/0) tcp 0 1 192.168.0.4:8920 179.238.115.221:58567 FIN_WAIT1 0 0 on (1.11/3/0) tcp 0 1 192.168.0.4:8920 192.168.0.100:52473 FIN_WAIT1 0 0 on (0.66/3/0) tcp 0 0 192.168.0.4:8096 192.168.0.101:36872 FIN_WAIT2 0 0 timewait (56.99/0/0) tcp 0 1 192.168.0.4:8920 179.238.115.221:58577 FIN_WAIT1 0 0 on (1.18/3/0) tcp 0 1 192.168.0.4:8920 177.176.190.222:39878 FIN_WAIT1 0 0 on (1.11/3/0) tcp 0 1 192.168.0.4:8920 179.238.115.221:60005 FIN_WAIT1 0 0 on (0.44/2/0) tcp 0 1 192.168.0.4:8920 179.238.115.221:58568 FIN_WAIT1 0 0 on (1.75/3/0) tcp 0 1 192.168.0.4:8920 192.168.0.100:52482 FIN_WAIT1 0 0 on (2.81/3/0) tcp 0 1 192.168.0.4:8920 191.40.61.190:43538 FIN_WAIT1 0 0 on (0.70/2/0) tcp 0 1 192.168.0.4:8920 179.238.115.221:58929 FIN_WAIT1 0 0 on (2.99/2/0) tcp 0 1 192.168.0.4:8920 192.168.0.100:52464 FIN_WAIT1 0 0 on (0.40/3/0) tcp 0 1 192.168.0.4:8920 179.238.115.221:58565 FIN_WAIT1 0 0 on (2.52/3/0) tcp 0 0 192.168.0.4:8096 192.168.0.101:50352 FIN_WAIT2 0 0 timewait (56.99/0/0) tcp 0 1 192.168.0.4:8920 191.40.61.190:43536 FIN_WAIT1 0 0 on (3.25/2/0) tcp 0 0 192.168.0.4:8096 192.168.0.4:58925 TIME_WAIT 0 0 timewait (56.82/0/0) tcp 0 1 192.168.0.4:8920 192.168.0.100:52487 FIN_WAIT1 0 0 on (2.92/3/0) tcp 0 1 192.168.0.4:8920 179.238.115.221:58576 FIN_WAIT1 0 0 on (1.50/3/0) tcp 0 1 192.168.0.4:8920 192.168.0.100:52481 FIN_WAIT1 0 0 on (1.72/3/0) tcp 0 1 192.168.0.4:8920 191.46.125.223:50096 FIN_WAIT1 0 0 on (0.40/2/0) tcp 0 1 192.168.0.4:8920 191.40.61.190:43537 FIN_WAIT1 0 0 on (2.08/1/0) tcp 0 1 192.168.0.4:8920 192.168.0.100:52460 FIN_WAIT1 0 0 on (2.31/2/0) tcp 0 0 192.168.0.4:8096 192.168.0.101:60347 FIN_WAIT2 0 0 timewait (56.99/0/0) tcp 0 1 192.168.0.4:8920 191.40.61.190:43539 FIN_WAIT1 0 0 on (3.59/1/0) Link to comment Share on other sites More sharing options...
anderbytes 139 Posted February 22, 2016 Author Share Posted February 22, 2016 (edited) the server closes all of it's sockets, although it's always possible there's an issue in a library we're using. @@Luke , please take a deeper look into this. 1. Connecting from external network (haven't tested internally yet) 2. I´ve opened Emby Web Client from Firefox (yeah... I know... it's not Chrome) 3. One new connection appeared in NETSTAT... as Estabilished... OK 4. Closed the browser tab where Web Client was 5. Connection remains opened forever as estabilished :-( I've noticed that there's no usage of "SetTcpKeepAlive" in your code when calling Mono, as implemented in Mono a long time ago: http://marc.info/?l=mono-patches&m=129496132601658 https://github.com/mono/mono/blob/082a47eed5dd823191e10ffefa57274a660e60e4/mcs/class/System/System.Net/ServicePoint.cs#L191-L215 So... with this enabled, I guess it will be possible for Mono or Emby to check if a connection is really active or not. With this implemented... hopefully we won't have more problems with unused connections... be it in ESTABILISHED or CLOSE_WAIT state Edited February 22, 2016 by anderbytes Link to comment Share on other sites More sharing options...
anderbytes 139 Posted February 26, 2016 Author Share Posted February 26, 2016 (edited) @@Luke and @@Happy2Play , please consider this issue as serious. I now am forced to restart my server frequently as 2x a day just because of the piled up unused connections. Probably people that run Emby in Linux may be having the same problem.... but just restart the service without knowing what's wrong. Looking at the code in GitHub and doing some research about zombie CLOSE_WAIT and zombie ESTABILISHED, I guess that the problem is around HttpListener , HttpClient and ConnectionManager. Can you please read this below, and maybe consider using MultiThreadedHttpConnectionManager to allow the reuse of sockets (instead of alwars creating new ones)??? To be read: http://www.nuxeo.com/blog/using-httpclient-properly-avoid-closewait-tcp-connections/ I can't explain in more details... but in short, the opened sockets by Emby converse asynchronously with the clients, and when one of those cliente close and for some reason can't contact the server (broken network connection, client crash, etc...) to tell him that he should also close the connection, creates a condition where the server has ports that will never again receive data, and will never auto-close, too. Because they hope that one day they will be contacted by it's faithful client - Is the MultiThreadedHttpConnectionManager solution above useful for Emby? - Is it possible to "ping" the connection once a minute to know if there's someone at the other side? And then force close the socket if noone responds. THANKS! ps: take a look again at post #6 above. There was only me using the server. That's monstruous! Edited February 26, 2016 by anderbytes Link to comment Share on other sites More sharing options...
Redshirt 1487 Posted February 26, 2016 Share Posted February 26, 2016 (edited) I had no idea this was happening, wow. I've got 30+ connections from only 2 android devices. I'm not sure it's possible to entirely close connections with Android currently. The sync account is always running in the background polling at default intervals and then there's the websocket updates from the server. Edited February 26, 2016 by Redshirt Link to comment Share on other sites More sharing options...
anderbytes 139 Posted February 26, 2016 Author Share Posted February 26, 2016 I had no idea this was happening, wow. I've got 30+ connections from only 2 android devices. I'm not sure it's possible to entirely close connections with Android currently. The sync account is always running in the background polling at default intervals and then there's the websocket updates from the server. Even with those, there would be only a few connections active. Link to comment Share on other sites More sharing options...
Luke 37065 Posted February 26, 2016 Share Posted February 26, 2016 this is probably just due to keep-alive, otherwise known as persistent connections. Link to comment Share on other sites More sharing options...
anderbytes 139 Posted February 26, 2016 Author Share Posted February 26, 2016 this is probably just due to keep-alive, otherwise known as persistent connections. Okay, but they are TOO persistent. :-/ Data conversations are broken but server stills thinks that socket will be useful, when it's not, just using resources. Can't you sanitize periodically (each X minutes) network sockets opened by Emby? Link to comment Share on other sites More sharing options...
MSattler 387 Posted February 26, 2016 Share Posted February 26, 2016 this is probably just due to keep-alive, otherwise known as persistent connections. Would it not make sense to look into it further instead of just dismissing it completely? Link to comment Share on other sites More sharing options...
MSattler 387 Posted February 26, 2016 Share Posted February 26, 2016 Okay, but they are TOO persistent. :-/ Data conversations are broken but server stills thinks that socket will be useful, when it's not, just using resources. Can't you sanitize periodically (each X minutes) network sockets opened by Emby? Agreed, and along with the fact that they are waiting in Fin_wait after the application cycles. Shouldn't all those connections be cleared the minute the service stops? Link to comment Share on other sites More sharing options...
anderbytes 139 Posted February 26, 2016 Author Share Posted February 26, 2016 As far as I've seen, FIN_WAIT always get auto-closed very quickly, so maybe that could be ignored... I guess. 1 Link to comment Share on other sites More sharing options...
dethknite 33 Posted February 26, 2016 Share Posted February 26, 2016 (edited) Yea, TCP connections that are not properly closed (TIMEWAIT) status will hang around until the OS kills them. Windows has a registry key that determines that (default 240 seconds). .\HKLM\SYSTEM\CurrentControlSet\Services\Tcpip\Parameters --> TcpTimedWaitDelay I am not personally experiencing this issue though. Perhaps there is a firewall issue or something else causing this... Edited February 26, 2016 by dethknite Link to comment Share on other sites More sharing options...
anderbytes 139 Posted February 26, 2016 Author Share Posted February 26, 2016 Yea, TCP connections that are not properly closed (TIMEWAIT) status will hang around until the OS kills them. Windows has a registry key that determines that (default 240 seconds). .\HKLM\SYSTEM\CurrentControlSet\Services\Tcpip\Parameters --> TcpTimedWaitDelay I am not personally experiencing this issue though. Perhaps there is a firewall issue or something else causing this... You are talking about Windows. My Debian Linux is working differently, he doesn't close it by itself. Maybe it's supposed to work that way. Maybe some Linux expert could try to investigate the reason... Link to comment Share on other sites More sharing options...
anderbytes 139 Posted February 27, 2016 Author Share Posted February 27, 2016 If you are a Linux user and read this... please try to reproduce the issue with the ports, and tell us if happened or not... and whats your distribution. Thanks for all. Link to comment Share on other sites More sharing options...
Luke 37065 Posted February 27, 2016 Share Posted February 27, 2016 i'm adjusting the keep alive behavior. you can play with it in upcoming dev and beta builds. Link to comment Share on other sites More sharing options...
anderbytes 139 Posted February 27, 2016 Author Share Posted February 27, 2016 i'm adjusting the keep alive behavior. you can play with it in upcoming dev and beta builds. OK Luke. Because of this, for the first time I'll run the beta version. Link to comment Share on other sites More sharing options...
kjp4756 41 Posted February 27, 2016 Share Posted February 27, 2016 I'm not seeing this issue on freebsd. I wonder if emby is relying on the OS to close the inactive ports. Windows and freebsd appear to handle this fine. I did find that after restarting emby the fin_wait sticks around for a while. Link to comment Share on other sites More sharing options...
runtimesandbox 152 Posted February 28, 2016 Share Posted February 28, 2016 (edited) As discussed in this thread http://emby.media/community/index.php?/topic/31089-emby-server-https-stops-responding/ i can no longer access the server via the https port 8920 Running the netstat command I get the following output root@media-server:/home/ms# netstat -anoe | grep ":8" tcp 0 0 0.0.0.0:8920 0.0.0.0:* LISTEN 109 1961283 off (0.00/0/0) tcp 0 0 0.0.0.0:8096 0.0.0.0:* LISTEN 109 1961282 off (0.00/0/0) tcp 195 0 192.168.1.5:8920 10.0.0.1:59111 ESTABLISHED 0 0 off (0.00/0/0) tcp 195 0 192.168.1.5:8920 10.0.0.1:45621 ESTABLISHED 0 0 off (0.00/0/0) tcp 195 0 192.168.1.5:8920 10.0.0.1:11983 ESTABLISHED 0 0 off (0.00/0/0) tcp 0 0 192.168.1.5:8096 10.0.0.174:46062 ESTABLISHED 109 2165503 off (0.00/0/0) tcp 0 0 192.168.1.5:8096 10.0.0.174:46067 ESTABLISHED 109 2175570 off (0.00/0/0) tcp 202 0 192.168.1.5:8920 10.0.0.1:34930 ESTABLISHED 0 0 off (0.00/0/0) tcp 195 0 192.168.1.5:8920 94.197.121.227:10496 ESTABLISHED 0 0 off (0.00/0/0) tcp 195 0 192.168.1.5:8920 94.197.121.227:10493 ESTABLISHED 0 0 off (0.00/0/0) tcp 195 0 192.168.1.5:8920 10.0.0.1:48591 ESTABLISHED 0 0 off (0.00/0/0) tcp 0 0 192.168.1.5:8096 10.0.0.174:46065 ESTABLISHED 109 2168277 off (0.00/0/0) tcp 0 1343 192.168.1.5:8920 94.197.121.227:9659 CLOSE_WAIT 109 2200931 on (33.93/10/0) tcp 195 0 192.168.1.5:8920 10.0.0.1:45620 ESTABLISHED 0 0 off (0.00/0/0) tcp 0 0 192.168.1.5:8096 10.0.0.174:46058 ESTABLISHED 109 2169134 off (0.00/0/0) tcp 195 0 192.168.1.5:8920 10.0.0.1:18302 ESTABLISHED 0 0 off (0.00/0/0) tcp 195 0 192.168.1.5:8920 10.0.0.1:60620 ESTABLISHED 0 0 off (0.00/0/0) tcp 195 0 192.168.1.5:8920 10.0.0.1:45619 ESTABLISHED 0 0 off (0.00/0/0) tcp 195 0 192.168.1.5:8920 10.0.0.1:59880 ESTABLISHED 0 0 off (0.00/0/0) tcp 0 0 192.168.1.5:8096 10.0.0.34:47670 ESTABLISHED 109 1960296 off (0.00/0/0) tcp 195 0 192.168.1.5:8920 10.0.0.1:60613 ESTABLISHED 0 0 off (0.00/0/0) tcp 195 0 192.168.1.5:8920 94.197.121.227:10494 ESTABLISHED 0 0 off (0.00/0/0) tcp 195 0 192.168.1.5:8920 10.0.0.1:45622 ESTABLISHED 0 0 off (0.00/0/0) tcp 0 0 192.168.1.5:8096 10.0.0.160:37829 ESTABLISHED 109 2199591 off (0.00/0/0) tcp 0 0 192.168.1.5:8096 10.0.0.174:46064 ESTABLISHED 109 2165506 off (0.00/0/0) tcp 195 0 192.168.1.5:8920 10.0.0.1:37133 ESTABLISHED 0 0 off (0.00/0/0) tcp 0 0 192.168.1.5:8096 10.0.0.174:46476 ESTABLISHED 109 2195978 off (0.00/0/0) tcp 0 0 192.168.1.5:8096 10.0.0.227:21153 ESTABLISHED 109 2201500 off (0.00/0/0) tcp 1 0 192.168.1.5:53510 104.28.22.116:80 CLOSE_WAIT 109 2201499 off (0.00/0/0) tcp 195 0 192.168.1.5:8920 10.0.0.1:60623 ESTABLISHED 0 0 off (0.00/0/0) tcp 195 0 192.168.1.5:8920 10.0.0.1:46350 ESTABLISHED 0 0 off (0.00/0/0) tcp 0 0 192.168.1.5:8096 10.0.0.174:45568 ESTABLISHED 109 2188130 off (0.00/0/0) tcp 0 0 192.168.1.5:8096 10.0.0.174:46060 ESTABLISHED 109 2168275 off (0.00/0/0) tcp 0 1336 192.168.1.5:8920 94.197.121.227:10492 ESTABLISHED 109 2200971 on (15.50/10/0) tcp 195 0 192.168.1.5:8920 10.0.0.1:45616 ESTABLISHED 0 0 off (0.00/0/0) tcp 195 0 192.168.1.5:8920 10.0.0.1:9834 ESTABLISHED 0 0 off (0.00/0/0) tcp 195 0 192.168.1.5:8920 10.0.0.1:45617 ESTABLISHED 0 0 off (0.00/0/0) tcp 0 0 192.168.1.5:8096 10.0.0.174:46063 ESTABLISHED 109 2165504 off (0.00/0/0) tcp 195 0 192.168.1.5:8920 10.0.0.1:26604 ESTABLISHED 0 0 off (0.00/0/0) tcp 0 0 192.168.1.5:54464 192.168.1.5:8096 ESTABLISHED 109 2199420 off (0.00/0/0) tcp 195 0 192.168.1.5:8920 10.0.0.1:60622 ESTABLISHED 0 0 off (0.00/0/0) tcp 195 0 192.168.1.5:8920 10.0.0.1:27367 ESTABLISHED 0 0 off (0.00/0/0) tcp 202 0 192.168.1.5:8920 10.0.0.1:34925 ESTABLISHED 0 0 off (0.00/0/0) tcp 0 0 192.168.1.5:8096 10.0.0.22:34772 ESTABLISHED 109 1957632 off (0.00/0/0) tcp 195 0 192.168.1.5:8920 10.0.0.1:60618 ESTABLISHED 0 0 off (0.00/0/0) tcp 195 0 192.168.1.5:8920 10.0.0.1:60625 ESTABLISHED 0 0 off (0.00/0/0) tcp 195 0 192.168.1.5:8920 10.0.0.1:60615 ESTABLISHED 0 0 off (0.00/0/0) tcp 195 0 192.168.1.5:8920 10.0.0.1:45618 ESTABLISHED 0 0 off (0.00/0/0) tcp 195 0 192.168.1.5:8920 10.0.0.1:16838 ESTABLISHED 0 0 off (0.00/0/0) tcp 202 0 192.168.1.5:8920 10.0.0.1:34927 ESTABLISHED 0 0 off (0.00/0/0) tcp 195 0 192.168.1.5:8920 94.197.121.227:10497 ESTABLISHED 0 0 off (0.00/0/0) tcp 195 0 192.168.1.5:8920 10.0.0.1:45623 ESTABLISHED 0 0 off (0.00/0/0) tcp 195 0 192.168.1.5:8920 94.197.121.227:10495 ESTABLISHED 0 0 off (0.00/0/0) tcp 0 0 192.168.1.5:8096 10.0.0.174:46066 ESTABLISHED 109 2165533 off (0.00/0/0) tcp 195 0 192.168.1.5:8920 10.0.0.1:9109 ESTABLISHED 0 0 off (0.00/0/0) tcp 195 0 192.168.1.5:8920 10.0.0.1:17598 ESTABLISHED 0 0 off (0.00/0/0) tcp 195 0 192.168.1.5:8920 10.0.0.1:60614 ESTABLISHED 0 0 off (0.00/0/0) tcp 0 0 192.168.1.5:8096 10.0.0.34:47668 ESTABLISHED 109 1960295 off (0.00/0/0) tcp 195 0 192.168.1.5:8920 10.0.0.1:60624 ESTABLISHED 0 0 off (0.00/0/0) tcp 195 0 192.168.1.5:8920 10.0.0.1:2792 ESTABLISHED 0 0 off (0.00/0/0) tcp 0 0 192.168.1.5:8096 10.0.0.22:34770 ESTABLISHED 109 1961301 off (0.00/0/0) tcp 202 0 192.168.1.5:8920 10.0.0.1:34926 ESTABLISHED 0 0 off (0.00/0/0) tcp 0 0 192.168.1.5:8096 10.0.0.227:21158 ESTABLISHED 109 2201497 off (0.00/0/0) tcp 195 0 192.168.1.5:8920 10.0.0.1:60619 ESTABLISHED 0 0 off (0.00/0/0) tcp 0 0 192.168.1.5:8096 192.168.1.5:54464 ESTABLISHED 109 2201432 off (0.00/0/0) tcp 195 0 192.168.1.5:8920 10.0.0.1:35262 ESTABLISHED 0 0 off (0.00/0/0) tcp 195 0 192.168.1.5:8920 10.0.0.1:60621 ESTABLISHED 0 0 off (0.00/0/0) tcp 195 0 192.168.1.5:8920 10.0.0.1:53546 ESTABLISHED 0 0 off (0.00/0/0) tcp 202 0 192.168.1.5:8920 10.0.0.1:34929 ESTABLISHED 0 0 off (0.00/0/0) tcp 0 0 192.168.1.5:8096 10.0.0.174:46061 ESTABLISHED 109 2168276 off (0.00/0/0) tcp 202 0 192.168.1.5:8920 10.0.0.1:34928 ESTABLISHED 0 0 off (0.00/0/0) tcp 195 0 192.168.1.5:8920 10.0.0.1:60616 ESTABLISHED 0 0 off (0.00/0/0) tcp 195 0 192.168.1.5:8920 10.0.0.1:60617 ESTABLISHED 0 0 off (0.00/0/0) This issue has appeared on both ubuntu 14 and debian 8 (64 bit) EDIT: The external ip was a mobile connection i attempted from over 3 days ago! After rebooting the server I get these FIN_WAIT connections root@media-server:/home/ms# netstat -anoe | grep ":8" tcp 0 1337 192.168.1.5:8920 94.197.121.227:10493 FIN_WAIT1 0 0 on (17.45/10/0) tcp 0 0 192.168.1.5:8096 10.0.0.227:21793 TIME_WAIT 0 0 timewait (58.41/0/0) tcp 0 0 192.168.1.5:8096 10.0.0.34:47670 FIN_WAIT2 0 0 timewait (58.83/0/0) tcp 0 0 192.168.1.5:8096 10.0.0.160:37829 FIN_WAIT2 0 0 timewait (58.94/0/0) tcp 0 0 192.168.1.5:8096 10.0.0.174:46476 FIN_WAIT2 0 0 timewait (59.02/0/0) tcp 0 0 192.168.1.5:8096 10.0.0.227:21790 TIME_WAIT 0 0 timewait (57.40/0/0) tcp 0 0 192.168.1.5:8096 10.0.0.227:21789 TIME_WAIT 0 0 timewait (57.90/0/0) tcp 0 0 192.168.1.5:8096 10.0.0.227:21795 TIME_WAIT 0 0 timewait (57.28/0/0) tcp 0 0 192.168.1.5:8096 10.0.0.174:58549 FIN_WAIT2 0 0 timewait (59.02/0/0) tcp 0 0 192.168.1.5:8096 10.0.0.22:34772 FIN_WAIT2 0 0 timewait (58.83/0/0) tcp 0 0 192.168.1.5:8096 10.0.0.22:34770 TIME_WAIT 0 0 timewait (57.27/0/0) tcp 0 0 192.168.1.5:54472 192.168.1.5:8096 TIME_WAIT 0 0 timewait (7.63/0/0) Edited February 28, 2016 by spudy12 Link to comment Share on other sites More sharing options...
anderbytes 139 Posted February 28, 2016 Author Share Posted February 28, 2016 @, welcome to the boat :-( Link to comment Share on other sites More sharing options...
runtimesandbox 152 Posted February 29, 2016 Share Posted February 29, 2016 @@anderbytes I'm glad I'm not the only one experiencing this Link to comment Share on other sites More sharing options...
Recommended Posts
Create an account or sign in to comment
You need to be a member in order to leave a comment
Create an account
Sign up for a new account in our community. It's easy!
Register a new accountSign in
Already have an account? Sign in here.
Sign In Now