mengoshmink 16 Posted May 22, 2020 Posted May 22, 2020 Hi all,Sometimes while browsing my media, sometimes while starting video playback, or second video after the first has finished - the server becomes unresponsive. Most often access is from Firefox web browser in Fedora on the gigabit LAN.I can't ssh in and APCI shutdown doesn't work. I resort to pressing the reboot button or press and hold the power button. Not good, I know.It has become so frequent I can no longer use Emby, I am browsing my NFS shares and playing in VideoLAN. I have used different server distributions, deleted both the Docker image and container, deleted Emby configuration.Setup Overview My media collection is hosted on a local Synology NAS. Shared via NFS. Linux server distribution Ubuntu 20.4 server LTS, was running CEntOS 8 stream. Docker installed with apt-get Emby is installed via docker-ce 19.03.8 NAS SettingsServer Settings mengoshmink@phantom:~$ cat /etc/fstab | grep video guardian:/volume1/video /mnt/guardian/video nfs rw,intr,nfsvers=4,noexec 0 0 mengoshmink@phantom:~$ ls -lh /mnt/guardian/video/ total 0 drwxrwxrwx 1 root users 10 May 13 09:40 Archive drwxrwxrwx 1 root users 198 Jan 5 20:33 Video version: '3.7' services: emby: volumes: - /mnt/store/var/docker/emby:/config:rw - /mnt/guardian/video:/mnt/guardian/video:rw - /mnt/guardian/tmp:/mnt/guardian/tmp:rw network_mode: "host" restart: unless-stopped environment: - UID=1000 - GID=100 container_name: emby image: emby/embyserver:latest Log files attached Any more information needed, please ask. Any ideas?Thank you in advance,take care and be safe :-) mengoshmink hardware_detection-63725769007.txt embyserver-63725769460.txt hardware_detection-63725769467.txt hardware_detection-63725770749.txt embyserver-63725770741.txt embyserver.txt
Luke 38842 Posted May 22, 2020 Posted May 22, 2020 Hi, what time of day during these log files did it become unresponsive?
mengoshmink 16 Posted May 22, 2020 Author Posted May 22, 2020 (edited) Hi Luke, Thank you for your fast response. I would say about an hour ago. Around 19:53 (7.53pm) I did a deleted my Emby config, docker image, container and re-created this evening. Thank you, Edited May 22, 2020 by mengoshmink
mengoshmink 16 Posted May 22, 2020 Author Posted May 22, 2020 (edited) I don't know if it's related but ... Just gone back onto the web interface, the one folder seems to have been added but I can't playback any files. Sorry, may of fixed this bit by stopping/starting the container. Edited May 23, 2020 by mengoshmink
mastrmind11 718 Posted May 23, 2020 Posted May 23, 2020 sounds like a memory issue. figure out what's happening there.
Q-Droid 830 Posted May 23, 2020 Posted May 23, 2020 If the server is hanging you ought to check the syslogs.
mengoshmink 16 Posted May 23, 2020 Author Posted May 23, 2020 Hi mastrmind11, Really!? I hadn't considered my memory as it is just post setup, before any transcoding. Any ideas how I could test? I have logwatch installed but haven't configured yet. Maybe I could do something with Monit.
mengoshmink 16 Posted May 23, 2020 Author Posted May 23, 2020 If the server is hanging you ought to check the syslogs. Hi Q-Droid, I've never been good with log files. Any suggestions on what to grep? Thanks,
Q-Droid 830 Posted May 23, 2020 Posted May 23, 2020 Nothing specific since it's unknown. Look for anything suspicious around the time of the hang, give or take 30 min.
mengoshmink 16 Posted May 24, 2020 Author Posted May 24, 2020 mastrmind11, below is from top when it crashed earlier. Memory usage looks fine to me. top - 07:36:23 up 9:32, 3 users, load average: 0.50, 0.25, 0.14 Tasks: 140 total, 1 running, 139 sleeping, 0 stopped, 0 zombie %Cpu(s): 14.5 us, 1.7 sy, 0.0 ni, 83.1 id, 0.5 wa, 0.0 hi, 0.2 si, 0.0 st MiB Mem : 5692.4 total, 723.8 free, 528.9 used, 4439.7 buff/cache MiB Swap: 10240.0 total, 10240.0 free, 0.0 used. 4902.3 avail Mem PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 5173 mengosh+ 20 0 3703312 322672 72376 S 14.3 5.5 6:04.67 EmbyServer Nothing specific since it's unknown. Look for anything suspicious around the time of the hang, give or take 30 min. Didn't find anything peculiar in syslog either. May 23 23:59:33 phantom kernel: [ 6958.950514] [UFW BLOCK] IN=enp3s0 OUT= MAC=e0:cb:4e:cf:53:a5:38:35:fb:84:50:7c:08:00 SRC=192.168.1.254 DST=192.168.1.95 LEN=344 TOS=0x00 PREC=0x00 TTL=64 ID=0 DF PROTO=UDP SPT=1900 DPT=47736 LEN=324 May 23 23:59:33 phantom kernel: [ 6958.966608] [UFW BLOCK] IN=enp3s0 OUT= MAC=e0:cb:4e:cf:53:a5:38:35:fb:84:50:7c:08:00 SRC=192.168.1.254 DST=192.168.1.95 LEN=408 TOS=0x00 PREC=0x00 TTL=64 ID=0 DF PROTO=UDP SPT=1900 DPT=47736 LEN=388 May 23 23:59:33 phantom kernel: [ 6958.986593] [UFW BLOCK] IN=enp3s0 OUT= MAC=e0:cb:4e:cf:53:a5:38:35:fb:84:50:7c:08:00 SRC=192.168.1.254 DST=192.168.1.95 LEN=355 TOS=0x00 PREC=0x00 TTL=64 ID=0 DF PROTO=UDP SPT=1900 DPT=47736 LEN=335 May 24 00:00:24 phantom systemd[1]: Starting Rotate log files... May 24 00:00:24 phantom systemd[1]: Starting Daily man-db regeneration... client_loop: send disconnect: Broken pipe
mengoshmink 16 Posted May 24, 2020 Author Posted May 24, 2020 Tried again with debug logging turned on. Log files attached. $ tail -f /var/log/syslog May 24 10:00:11 phantom kernel: [ 1469.250329] [UFW BLOCK] IN=enp3s0 OUT= MAC=e0:cb:4e:cf:53:a5:38:35:fb:84:50:7c:08:00 SRC=192.168.1.254 DST=192.168.1.95 LEN=344 TOS=0x00 PREC=0x00 TTL=64 ID=0 DF PROTO=UDP SPT=1900 DPT=57077 LEN=324 May 24 10:00:11 phantom kernel: [ 1469.266429] [UFW BLOCK] IN=enp3s0 OUT= MAC=e0:cb:4e:cf:53:a5:38:35:fb:84:50:7c:08:00 SRC=192.168.1.254 DST=192.168.1.95 LEN=408 TOS=0x00 PREC=0x00 TTL=64 ID=0 DF PROTO=UDP SPT=1900 DPT=57077 LEN=388 May 24 10:00:11 phantom kernel: [ 1469.286589] [UFW BLOCK] IN=enp3s0 OUT= MAC=e0:cb:4e:cf:53:a5:38:35:fb:84:50:7c:08:00 SRC=192.168.1.254 DST=192.168.1.95 LEN=355 TOS=0x00 PREC=0x00 TTL=64 ID=0 DF PROTO=UDP SPT=1900 DPT=57077 LEN=335 May 24 10:00:37 phantom kernel: [ 1495.218518] [UFW BLOCK] IN=enp3s0 OUT= MAC=01:00:5e:00:00:01:38:35:fb:84:50:7c:08:00 SRC=192.168.1.254 DST=224.0.0.1 LEN=36 TOS=0x00 PREC=0xC0 TTL=1 ID=0 DF PROTO=2 May 24 10:01:10 phantom kernel: [ 1528.211361] [UFW BLOCK] IN=enp3s0 OUT= MAC=e0:cb:4e:cf:53:a5:38:35:fb:84:50:7c:08:00 SRC=192.168.1.254 DST=192.168.1.95 LEN=344 TOS=0x00 PREC=0x00 TTL=64 ID=0 DF PROTO=UDP SPT=1900 DPT=57077 LEN=324 May 24 10:01:10 phantom kernel: [ 1528.227469] [UFW BLOCK] IN=enp3s0 OUT= MAC=e0:cb:4e:cf:53:a5:38:35:fb:84:50:7c:08:00 SRC=192.168.1.254 DST=192.168.1.95 LEN=408 TOS=0x00 PREC=0x00 TTL=64 ID=0 DF PROTO=UDP SPT=1900 DPT=57077 LEN=388 May 24 10:02:12 phantom kernel: [ 1589.863645] [UFW BLOCK] IN=enp3s0 OUT= MAC=e0:cb:4e:cf:53:a5:38:35:fb:84:50:7c:08:00 SRC=192.168.1.254 DST=192.168.1.95 LEN=344 TOS=0x00 PREC=0x00 TTL=64 ID=0 DF PROTO=UDP SPT=1900 DPT=57077 LEN=324 May 24 10:02:12 phantom kernel: [ 1589.878671] [UFW BLOCK] IN=enp3s0 OUT= MAC=e0:cb:4e:cf:53:a5:38:35:fb:84:50:7c:08:00 SRC=192.168.1.254 DST=192.168.1.95 LEN=408 TOS=0x00 PREC=0x00 TTL=64 ID=0 DF PROTO=UDP SPT=1900 DPT=57077 LEN=388 May 24 10:02:12 phantom kernel: [ 1589.898678] [UFW BLOCK] IN=enp3s0 OUT= MAC=e0:cb:4e:cf:53:a5:38:35:fb:84:50:7c:08:00 SRC=192.168.1.254 DST=192.168.1.95 LEN=355 TOS=0x00 PREC=0x00 TTL=64 ID=0 DF PROTO=UDP SPT=1900 DPT=57077 LEN=335 May 24 10:02:42 phantom kernel: [ 1620.220771] [UFW BLOCK] IN=enp3s0 OUT= MAC=01:00:5e:00:00:01:38:35:fb:84:50:7c:08:00 SRC=192.168.1.254 DST=224.0.0.1 LEN=36 TOS=0x00 PREC=0xC0 TTL=1 ID=0 DF PROTO=2 May 24 10:03:10 phantom kernel: [ 1647.853536] [UFW BLOCK] IN=enp3s0 OUT= MAC=e0:cb:4e:cf:53:a5:38:35:fb:84:50:7c:08:00 SRC=192.168.1.254 DST=192.168.1.95 LEN=344 TOS=0x00 PREC=0x00 TTL=64 ID=0 DF PROTO=UDP SPT=1900 DPT=57077 LEN=324 May 24 10:03:10 phantom kernel: [ 1647.869708] [UFW BLOCK] IN=enp3s0 OUT= MAC=e0:cb:4e:cf:53:a5:38:35:fb:84:50:7c:08:00 SRC=192.168.1.254 DST=192.168.1.95 LEN=408 TOS=0x00 PREC=0x00 TTL=64 ID=0 DF PROTO=UDP SPT=1900 DPT=57077 LEN=388 May 24 10:04:10 phantom kernel: [ 1708.174552] [UFW BLOCK] IN=enp3s0 OUT= MAC=e0:cb:4e:cf:53:a5:38:35:fb:84:50:7c:08:00 SRC=192.168.1.254 DST=192.168.1.95 LEN=344 TOS=0x00 PREC=0x00 TTL=64 ID=0 DF PROTO=UDP SPT=1900 DPT=57077 LEN=324 May 24 10:04:10 phantom kernel: [ 1708.190727] [UFW BLOCK] IN=enp3s0 OUT= MAC=e0:cb:4e:cf:53:a5:38:35:fb:84:50:7c:08:00 SRC=192.168.1.254 DST=192.168.1.95 LEN=408 TOS=0x00 PREC=0x00 TTL=64 ID=0 DF PROTO=UDP SPT=1900 DPT=57077 LEN=388 May 24 10:04:10 phantom kernel: [ 1708.210662] [UFW BLOCK] IN=enp3s0 OUT= MAC=e0:cb:4e:cf:53:a5:38:35:fb:84:50:7c:08:00 SRC=192.168.1.254 DST=192.168.1.95 LEN=355 TOS=0x00 PREC=0x00 TTL=64 ID=0 DF PROTO=UDP SPT=1900 DPT=57077 LEN=335 May 24 10:04:47 phantom kernel: [ 1745.222977] [UFW BLOCK] IN=enp3s0 OUT= MAC=01:00:5e:00:00:01:38:35:fb:84:50:7c:08:00 SRC=192.168.1.254 DST=224.0.0.1 LEN=36 TOS=0x00 PREC=0xC0 TTL=1 ID=0 DF PROTO=2 May 24 10:05:11 phantom kernel: [ 1768.936405] [UFW BLOCK] IN=enp3s0 OUT= MAC=e0:cb:4e:cf:53:a5:38:35:fb:84:50:7c:08:00 SRC=192.168.1.254 DST=192.168.1.95 LEN=344 TOS=0x00 PREC=0x00 TTL=64 ID=0 DF PROTO=UDP SPT=1900 DPT=57077 LEN=324 May 24 10:05:11 phantom kernel: [ 1768.951864] [UFW BLOCK] IN=enp3s0 OUT= MAC=e0:cb:4e:cf:53:a5:38:35:fb:84:50:7c:08:00 SRC=192.168.1.254 DST=192.168.1.95 LEN=408 TOS=0x00 PREC=0x00 TTL=64 ID=0 DF PROTO=UDP SPT=1900 DPT=57077 LEN=388 May 24 10:06:09 phantom kernel: [ 1827.726780] [UFW BLOCK] IN=enp3s0 OUT= MAC=e0:cb:4e:cf:53:a5:38:35:fb:84:50:7c:08:00 SRC=192.168.1.254 DST=192.168.1.95 LEN=344 TOS=0x00 PREC=0x00 TTL=64 ID=0 DF PROTO=UDP SPT=1900 DPT=57077 LEN=324 May 24 10:06:09 phantom kernel: [ 1827.742902] [UFW BLOCK] IN=enp3s0 OUT= MAC=e0:cb:4e:cf:53:a5:38:35:fb:84:50:7c:08:00 SRC=192.168.1.254 DST=192.168.1.95 LEN=408 TOS=0x00 PREC=0x00 TTL=64 ID=0 DF PROTO=UDP SPT=1900 DPT=57077 LEN=388 May 24 10:06:09 phantom kernel: [ 1827.762891] [UFW BLOCK] IN=enp3s0 OUT= MAC=e0:cb:4e:cf:53:a5:38:35:fb:84:50:7c:08:00 SRC=192.168.1.254 DST=192.168.1.95 LEN=355 TOS=0x00 PREC=0x00 TTL=64 ID=0 DF PROTO=UDP SPT=1900 DPT=57077 LEN=335 $ top top - 10:06:41 up 30 min, 2 users, load average: 0.54, 0.58, 0.45 Tasks: 138 total, 1 running, 137 sleeping, 0 stopped, 0 zombie %Cpu(s): 15.9 us, 3.2 sy, 0.0 ni, 74.3 id, 5.7 wa, 0.0 hi, 1.0 si, 0.0 st MiB Mem : 5692.4 total, 122.7 free, 416.4 used, 5153.3 buff/cache MiB Swap: 10240.0 total, 10239.7 free, 0.2 used. 5039.5 avail Mem PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 3670 mengosh+ 20 0 3626976 206460 65356 S 22.6 3.5 4:02.84 EmbySer+ 7802 mengosh+ 20 0 70572 19320 15912 D 1.7 0.3 0:00.05 ffprobe embyserver.txt embyserver-63725906642.txt embyserver-63725907367.txt hardware_detection-63725906534.txt hardware_detection-63725906647.txt hardware_detection-63725907375.txt
mengoshmink 16 Posted May 24, 2020 Author Posted May 24, 2020 (edited) This is not an emby issue. Okay, any suggestions on what it might be? I have tried different distro's and with/without Docker. Thanks, Edited May 24, 2020 by mengoshmink
Cthalpa 10 Posted May 24, 2020 Posted May 24, 2020 You have it right there in your syslog:client_loop: send disconnect: Broken pipe You are experiencing broken pipes. It can be because of many things. I/O, permissions, poorly configured software, freeze on the NFS/NAS transport, insufficient hardware etc etc.. or something complete else.You have so many things in play here, it's impossible to guess.
mengoshmink 16 Posted May 24, 2020 Author Posted May 24, 2020 (edited) Ah, sorry, that's misleading. My ssh connection drops (times out - "broken pipe") when the Emby server crashes. When I have a keyboard and screen connected the server becomes unresponsive when Emby crashes. I am pretty sure it is Emby related because I have turned off all of my other containers, and this is almost a fresh install. Without Emby running everything appears to work fine. That said I am certainly *not* blaming Emby. I think it is some kind of miss configuration on my part. I was hoping someone would have some ideas. I have tried what I can think of eg various permissions changes. Edited May 24, 2020 by mengoshmink
Q-Droid 830 Posted May 24, 2020 Posted May 24, 2020 Do those syslog entries include the time window when the server was hanging?
mengoshmink 16 Posted May 24, 2020 Author Posted May 24, 2020 (edited) Do those syslog entries include the time window when the server was hanging? Yes Q-Droid. I rebooted. Logged in via ssh docker stop emby rm -rf docker/emby/ # configuration files and folders docker rm emby docker rmi emby/embyserver:latest cd docker docker-compose up -d ## Completed setup through web interface but didn't add any libraries docker stop emby # this seemed to help before docker start emby tail -f /var/log/syslog ## Logged into web interface and added a library with multiple folders https://pastebin.com/hg99QSFD I carried on doing stuff, not using the server until I saw the broken pipe. I rebooted and logged into web interface to get the log files and make my post on here. Thanks, Edited May 24, 2020 by mengoshmink
Q-Droid 830 Posted May 24, 2020 Posted May 24, 2020 Is EmbyServer running as mengoshmink and have you tried letting it run as emby?
mengoshmink 16 Posted May 24, 2020 Author Posted May 24, 2020 Is EmbyServer running as mengoshmink and have you tried letting it run as emby? I am running the Emby Docker image, so there is no local emby user.
mastrmind11 718 Posted May 24, 2020 Posted May 24, 2020 (edited) I am running the Emby Docker image, so there is no local emby user. so then who is environment: - UID=1000 - GID=100 ? These need to map to your emby user. Type id emby on the cli Edited May 24, 2020 by mastrmind11
Cthalpa 10 Posted May 24, 2020 Posted May 24, 2020 Emby create an emby user when its installed, so it should be running as that user. emby:x:998:998:Emby Server:/var/lib/emby:/usr/bin/nologin
Q-Droid 830 Posted May 24, 2020 Posted May 24, 2020 Docker doesn't prevent you from having a local user and it's a good general practice to create dedicated accounts for server type applications. That said nothing jumps out in the logs. If Emby is causing the hang I don't see anything to go on.
Cthalpa 10 Posted May 24, 2020 Posted May 24, 2020 I would ditch it, and start over if that case was mine. And I'd probably make it a less "complicated" setup, if my skills were not with me
mengoshmink 16 Posted May 24, 2020 Author Posted May 24, 2020 so then who is environment: - UID=1000 - GID=100 ? These need to map to your emby user. Type id emby on the cli The UID 1000 is for my user account, GID is the local users group. $ id mengoshmink uid=1000(mengoshmink) gid=1000(mengoshmink) groups=1000(mengoshmink) Emby create an emby user when its installed, so it should be running as that user. emby:x:998:998:Emby Server:/var/lib/emby:/usr/bin/nologin If I had installed the Emby .deb package, yes, I would of expected there to be a local Emby user account. I have tried stopping Docker and using the .deb package, but I had the same problem. Docker doesn't prevent you from having a local user and it's a good general practice to create dedicated accounts for server type applications. That said nothing jumps out in the logs. If Emby is causing the hang I don't see anything to go on. Ultimately I may run Docker containers as different local users. As you say, it is not good practice. It running as my unprivileged username I do not see as a major concern at present. I would ditch it, and start over if that case was mine. And I'd probably make it a less "complicated" setup, if my skills were not with me Ditch what? My configuration files? Using Emby? Using Docker? My metal server? Using a remote NAS? Thank you all for your help so far,
Cthalpa 10 Posted May 24, 2020 Posted May 24, 2020 (edited) Ditch what? My configuration files? Using Emby? Using Docker? My metal server? Using a remote NAS? Thank you all for your help so far, The emby installation entirely, with reinstall. Is there any particular reason you use docker? The emby installation does nothing to your system except installing its files in /var/lib/emby, and create a systemd file for start/stop/restart. You should easily be able to install without the need for docker, or any other kind of containment. If you're using a remote NAS, be sure that it's on a good tier, any interrupts on the network, or the network being too slow, will result in the kind of errors you are having now. Edit: I would not run NFS over WAN. I would recommend you to use a secure connectioon, and mount the remote NAS drive as a local remote with E.G sshfs. Edited May 24, 2020 by Cthalpa
Recommended Posts
Create an account or sign in to comment
You need to be a member in order to leave a comment
Create an account
Sign up for a new account in our community. It's easy!
Register a new accountSign in
Already have an account? Sign in here.
Sign In Now