Jump to content
mengoshmink

NAS + NFS + Ubuntu + Docker + Emby + Firefox

Recommended Posts

mengoshmink

Hi all,

Sometimes while browsing my media, sometimes while starting video playback, or second video after the first has finished - the server becomes unresponsive. Most often access is from Firefox web browser in Fedora on the gigabit LAN.

I can't ssh in and APCI shutdown doesn't work. I resort to pressing the reboot button or press and hold the power button. Not good, I know.

It has become so frequent I can no longer use Emby, I am browsing my NFS shares and playing in VideoLAN.

 

I have used different server distributions, deleted both the Docker image and container, deleted Emby configuration.

Setup Overview

  • My media collection is hosted on a local Synology NAS.
  • Shared via NFS.
  • Linux server distribution Ubuntu 20.4 server LTS, was running CEntOS 8 stream.
  • Docker installed with apt-get
  • Emby is installed via docker-ce 19.03.8

NAS Settings

5ec816805a32a_Screenshotfrom202005221904

5ec816981d2d9_Screenshotfrom202005221911

Server Settings


mengoshmink@phantom:~$ cat /etc/fstab | grep video
guardian:/volume1/video /mnt/guardian/video nfs rw,intr,nfsvers=4,noexec 0 0

mengoshmink@phantom:~$ ls -lh /mnt/guardian/video/
total 0
drwxrwxrwx 1 root users  10 May 13 09:40 Archive
drwxrwxrwx 1 root users 198 Jan  5 20:33 Video
version: '3.7'
services:

  emby:
    volumes:
      - /mnt/store/var/docker/emby:/config:rw
      - /mnt/guardian/video:/mnt/guardian/video:rw
      - /mnt/guardian/tmp:/mnt/guardian/tmp:rw
    network_mode: "host"
    restart: unless-stopped
    environment:
      - UID=1000
      - GID=100
    container_name: emby
    image: emby/embyserver:latest

Log files attached

5ec8319f924f8_Screenshotfrom202005222106

 

Any more information needed, please ask.

Any ideas?

Thank you in advance,
take care and be safe :-)

 

mengoshmink

hardware_detection-63725769007.txt

embyserver-63725769460.txt

hardware_detection-63725769467.txt

hardware_detection-63725770749.txt

embyserver-63725770741.txt

embyserver.txt

Share this post


Link to post
Share on other sites
Luke

Hi, what time of day during these log files did it become unresponsive?

Share this post


Link to post
Share on other sites
mengoshmink
Posted (edited)

Hi Luke,

 

Thank you for your fast response.

I would say about an hour ago. Around 19:53 (7.53pm)

I did a deleted my Emby config, docker image, container and re-created this evening.

 

Thank you,

Edited by mengoshmink

Share this post


Link to post
Share on other sites
mengoshmink
Posted (edited)

I don't know if it's related but ...

Just gone back onto the web interface, the one folder seems to have been added but I can't playback any files.

 

5ec8459ccc053_Screenshotfrom202005222235

 

Sorry, may of fixed this bit by stopping/starting the container.

Edited by mengoshmink

Share this post


Link to post
Share on other sites
mastrmind11

sounds like a memory issue.  figure out what's happening there.

Share this post


Link to post
Share on other sites
Q-Droid

If the server is hanging you ought to check the syslogs.

Share this post


Link to post
Share on other sites
mengoshmink

Hi mastrmind11,

 

Really!? I hadn't considered my memory as it is just post setup, before any transcoding.

 

Any ideas how I could test?

I have logwatch installed but haven't configured yet. Maybe I could do something with Monit.

Share this post


Link to post
Share on other sites
mengoshmink

If the server is hanging you ought to check the syslogs.

Hi Q-Droid,

 

I've never been good with log files. Any suggestions on what to grep?

Thanks,

Share this post


Link to post
Share on other sites
Q-Droid

Nothing specific since it's unknown. Look for anything suspicious around the time of the hang, give or take 30 min.

Share this post


Link to post
Share on other sites
mengoshmink

mastrmind11, below is from top when it crashed earlier. Memory usage looks fine to me.

top - 07:36:23 up  9:32,  3 users,  load average: 0.50, 0.25, 0.14
Tasks: 140 total,   1 running, 139 sleeping,   0 stopped,   0 zombie
%Cpu(s): 14.5 us,  1.7 sy,  0.0 ni, 83.1 id,  0.5 wa,  0.0 hi,  0.2 si,  0.0 st
MiB Mem :   5692.4 total,    723.8 free,    528.9 used,   4439.7 buff/cache
MiB Swap:  10240.0 total,  10240.0 free,      0.0 used.   4902.3 avail Mem

    PID USER      PR  NI    VIRT    RES    SHR S  %CPU  %MEM     TIME+ COMMAND                                                                                                                                                                
   5173 mengosh+  20   0 3703312 322672  72376 S  14.3   5.5   6:04.67 EmbyServer

Nothing specific since it's unknown. Look for anything suspicious around the time of the hang, give or take 30 min.

Didn't find anything peculiar in syslog either.

May 23 23:59:33 phantom kernel: [ 6958.950514] [UFW BLOCK] IN=enp3s0 OUT= MAC=e0:cb:4e:cf:53:a5:38:35:fb:84:50:7c:08:00 SRC=192.168.1.254 DST=192.168.1.95 LEN=344 TOS=0x00 PREC=0x00 TTL=64 ID=0 DF PROTO=UDP SPT=1900 DPT=47736 LEN=324
May 23 23:59:33 phantom kernel: [ 6958.966608] [UFW BLOCK] IN=enp3s0 OUT= MAC=e0:cb:4e:cf:53:a5:38:35:fb:84:50:7c:08:00 SRC=192.168.1.254 DST=192.168.1.95 LEN=408 TOS=0x00 PREC=0x00 TTL=64 ID=0 DF PROTO=UDP SPT=1900 DPT=47736 LEN=388
May 23 23:59:33 phantom kernel: [ 6958.986593] [UFW BLOCK] IN=enp3s0 OUT= MAC=e0:cb:4e:cf:53:a5:38:35:fb:84:50:7c:08:00 SRC=192.168.1.254 DST=192.168.1.95 LEN=355 TOS=0x00 PREC=0x00 TTL=64 ID=0 DF PROTO=UDP SPT=1900 DPT=47736 LEN=335
May 24 00:00:24 phantom systemd[1]: Starting Rotate log files...
May 24 00:00:24 phantom systemd[1]: Starting Daily man-db regeneration...

client_loop: send disconnect: Broken pipe

Share this post


Link to post
Share on other sites
mengoshmink

Tried again with debug logging turned on. Log files attached.

 

$ tail -f /var/log/syslog

May 24 10:00:11 phantom kernel: [ 1469.250329] [UFW BLOCK] IN=enp3s0 OUT= MAC=e0:cb:4e:cf:53:a5:38:35:fb:84:50:7c:08:00 SRC=192.168.1.254 DST=192.168.1.95 LEN=344 TOS=0x00 PREC=0x00 TTL=64 ID=0 DF PROTO=UDP SPT=1900 DPT=57077 LEN=324
May 24 10:00:11 phantom kernel: [ 1469.266429] [UFW BLOCK] IN=enp3s0 OUT= MAC=e0:cb:4e:cf:53:a5:38:35:fb:84:50:7c:08:00 SRC=192.168.1.254 DST=192.168.1.95 LEN=408 TOS=0x00 PREC=0x00 TTL=64 ID=0 DF PROTO=UDP SPT=1900 DPT=57077 LEN=388
May 24 10:00:11 phantom kernel: [ 1469.286589] [UFW BLOCK] IN=enp3s0 OUT= MAC=e0:cb:4e:cf:53:a5:38:35:fb:84:50:7c:08:00 SRC=192.168.1.254 DST=192.168.1.95 LEN=355 TOS=0x00 PREC=0x00 TTL=64 ID=0 DF PROTO=UDP SPT=1900 DPT=57077 LEN=335
May 24 10:00:37 phantom kernel: [ 1495.218518] [UFW BLOCK] IN=enp3s0 OUT= MAC=01:00:5e:00:00:01:38:35:fb:84:50:7c:08:00 SRC=192.168.1.254 DST=224.0.0.1 LEN=36 TOS=0x00 PREC=0xC0 TTL=1 ID=0 DF PROTO=2
May 24 10:01:10 phantom kernel: [ 1528.211361] [UFW BLOCK] IN=enp3s0 OUT= MAC=e0:cb:4e:cf:53:a5:38:35:fb:84:50:7c:08:00 SRC=192.168.1.254 DST=192.168.1.95 LEN=344 TOS=0x00 PREC=0x00 TTL=64 ID=0 DF PROTO=UDP SPT=1900 DPT=57077 LEN=324
May 24 10:01:10 phantom kernel: [ 1528.227469] [UFW BLOCK] IN=enp3s0 OUT= MAC=e0:cb:4e:cf:53:a5:38:35:fb:84:50:7c:08:00 SRC=192.168.1.254 DST=192.168.1.95 LEN=408 TOS=0x00 PREC=0x00 TTL=64 ID=0 DF PROTO=UDP SPT=1900 DPT=57077 LEN=388
May 24 10:02:12 phantom kernel: [ 1589.863645] [UFW BLOCK] IN=enp3s0 OUT= MAC=e0:cb:4e:cf:53:a5:38:35:fb:84:50:7c:08:00 SRC=192.168.1.254 DST=192.168.1.95 LEN=344 TOS=0x00 PREC=0x00 TTL=64 ID=0 DF PROTO=UDP SPT=1900 DPT=57077 LEN=324
May 24 10:02:12 phantom kernel: [ 1589.878671] [UFW BLOCK] IN=enp3s0 OUT= MAC=e0:cb:4e:cf:53:a5:38:35:fb:84:50:7c:08:00 SRC=192.168.1.254 DST=192.168.1.95 LEN=408 TOS=0x00 PREC=0x00 TTL=64 ID=0 DF PROTO=UDP SPT=1900 DPT=57077 LEN=388
May 24 10:02:12 phantom kernel: [ 1589.898678] [UFW BLOCK] IN=enp3s0 OUT= MAC=e0:cb:4e:cf:53:a5:38:35:fb:84:50:7c:08:00 SRC=192.168.1.254 DST=192.168.1.95 LEN=355 TOS=0x00 PREC=0x00 TTL=64 ID=0 DF PROTO=UDP SPT=1900 DPT=57077 LEN=335
May 24 10:02:42 phantom kernel: [ 1620.220771] [UFW BLOCK] IN=enp3s0 OUT= MAC=01:00:5e:00:00:01:38:35:fb:84:50:7c:08:00 SRC=192.168.1.254 DST=224.0.0.1 LEN=36 TOS=0x00 PREC=0xC0 TTL=1 ID=0 DF PROTO=2
May 24 10:03:10 phantom kernel: [ 1647.853536] [UFW BLOCK] IN=enp3s0 OUT= MAC=e0:cb:4e:cf:53:a5:38:35:fb:84:50:7c:08:00 SRC=192.168.1.254 DST=192.168.1.95 LEN=344 TOS=0x00 PREC=0x00 TTL=64 ID=0 DF PROTO=UDP SPT=1900 DPT=57077 LEN=324
May 24 10:03:10 phantom kernel: [ 1647.869708] [UFW BLOCK] IN=enp3s0 OUT= MAC=e0:cb:4e:cf:53:a5:38:35:fb:84:50:7c:08:00 SRC=192.168.1.254 DST=192.168.1.95 LEN=408 TOS=0x00 PREC=0x00 TTL=64 ID=0 DF PROTO=UDP SPT=1900 DPT=57077 LEN=388
May 24 10:04:10 phantom kernel: [ 1708.174552] [UFW BLOCK] IN=enp3s0 OUT= MAC=e0:cb:4e:cf:53:a5:38:35:fb:84:50:7c:08:00 SRC=192.168.1.254 DST=192.168.1.95 LEN=344 TOS=0x00 PREC=0x00 TTL=64 ID=0 DF PROTO=UDP SPT=1900 DPT=57077 LEN=324
May 24 10:04:10 phantom kernel: [ 1708.190727] [UFW BLOCK] IN=enp3s0 OUT= MAC=e0:cb:4e:cf:53:a5:38:35:fb:84:50:7c:08:00 SRC=192.168.1.254 DST=192.168.1.95 LEN=408 TOS=0x00 PREC=0x00 TTL=64 ID=0 DF PROTO=UDP SPT=1900 DPT=57077 LEN=388
May 24 10:04:10 phantom kernel: [ 1708.210662] [UFW BLOCK] IN=enp3s0 OUT= MAC=e0:cb:4e:cf:53:a5:38:35:fb:84:50:7c:08:00 SRC=192.168.1.254 DST=192.168.1.95 LEN=355 TOS=0x00 PREC=0x00 TTL=64 ID=0 DF PROTO=UDP SPT=1900 DPT=57077 LEN=335
May 24 10:04:47 phantom kernel: [ 1745.222977] [UFW BLOCK] IN=enp3s0 OUT= MAC=01:00:5e:00:00:01:38:35:fb:84:50:7c:08:00 SRC=192.168.1.254 DST=224.0.0.1 LEN=36 TOS=0x00 PREC=0xC0 TTL=1 ID=0 DF PROTO=2
May 24 10:05:11 phantom kernel: [ 1768.936405] [UFW BLOCK] IN=enp3s0 OUT= MAC=e0:cb:4e:cf:53:a5:38:35:fb:84:50:7c:08:00 SRC=192.168.1.254 DST=192.168.1.95 LEN=344 TOS=0x00 PREC=0x00 TTL=64 ID=0 DF PROTO=UDP SPT=1900 DPT=57077 LEN=324
May 24 10:05:11 phantom kernel: [ 1768.951864] [UFW BLOCK] IN=enp3s0 OUT= MAC=e0:cb:4e:cf:53:a5:38:35:fb:84:50:7c:08:00 SRC=192.168.1.254 DST=192.168.1.95 LEN=408 TOS=0x00 PREC=0x00 TTL=64 ID=0 DF PROTO=UDP SPT=1900 DPT=57077 LEN=388
May 24 10:06:09 phantom kernel: [ 1827.726780] [UFW BLOCK] IN=enp3s0 OUT= MAC=e0:cb:4e:cf:53:a5:38:35:fb:84:50:7c:08:00 SRC=192.168.1.254 DST=192.168.1.95 LEN=344 TOS=0x00 PREC=0x00 TTL=64 ID=0 DF PROTO=UDP SPT=1900 DPT=57077 LEN=324
May 24 10:06:09 phantom kernel: [ 1827.742902] [UFW BLOCK] IN=enp3s0 OUT= MAC=e0:cb:4e:cf:53:a5:38:35:fb:84:50:7c:08:00 SRC=192.168.1.254 DST=192.168.1.95 LEN=408 TOS=0x00 PREC=0x00 TTL=64 ID=0 DF PROTO=UDP SPT=1900 DPT=57077 LEN=388
May 24 10:06:09 phantom kernel: [ 1827.762891] [UFW BLOCK] IN=enp3s0 OUT= MAC=e0:cb:4e:cf:53:a5:38:35:fb:84:50:7c:08:00 SRC=192.168.1.254 DST=192.168.1.95 LEN=355 TOS=0x00 PREC=0x00 TTL=64 ID=0 DF PROTO=UDP SPT=1900 DPT=57077 LEN=335

 

 

$ top

top - 10:06:41 up 30 min,  2 users,  load average: 0.54, 0.58, 0.45
Tasks: 138 total,   1 running, 137 sleeping,   0 stopped,   0 zombie
%Cpu(s): 15.9 us,  3.2 sy,  0.0 ni, 74.3 id,  5.7 wa,  0.0 hi,  1.0 si,  0.0 st
MiB Mem :   5692.4 total,    122.7 free,    416.4 used,   5153.3 buff/cache
MiB Swap:  10240.0 total,  10239.7 free,      0.2 used.   5039.5 avail Mem

    PID USER      PR  NI    VIRT    RES    SHR S  %CPU  %MEM     TIME+ COMMAND  
   3670 mengosh+  20   0 3626976 206460  65356 S  22.6   3.5   4:02.84 EmbySer+
   7802 mengosh+  20   0   70572  19320  15912 D   1.7   0.3   0:00.05 ffprobe

 

embyserver.txt

embyserver-63725906642.txt

embyserver-63725907367.txt

hardware_detection-63725906534.txt

hardware_detection-63725906647.txt

hardware_detection-63725907375.txt

Share this post


Link to post
Share on other sites
Cthalpa

This is not an emby issue.

Share this post


Link to post
Share on other sites
mengoshmink
Posted (edited)

This is not an emby issue.

 

Okay, any suggestions on what it might be? I have tried different distro's and with/without Docker.

Thanks,

Edited by mengoshmink

Share this post


Link to post
Share on other sites
Cthalpa

You have it right there in your syslog:

client_loop: send disconnect: Broken pipe

 

You are experiencing broken pipes.

 

It can be because of many things. I/O, permissions, poorly configured software, freeze on the NFS/NAS transport, insufficient hardware etc etc.. or something complete else.
You have so many things in play here, it's impossible to guess.



 

Share this post


Link to post
Share on other sites
mengoshmink
Posted (edited)

Ah, sorry, that's misleading. My ssh connection drops (times out - "broken pipe") when the Emby server crashes. When I have a keyboard and screen connected the server becomes unresponsive when Emby crashes.

 

I am pretty sure it is Emby related because I have turned off all of my other containers, and this is almost a fresh install.

Without Emby running everything appears to work fine.

 

That said I am certainly *not* blaming Emby. I think it is some kind of miss configuration on my part. I was hoping someone would have some ideas. I have tried what I can think of eg various permissions changes.

Edited by mengoshmink

Share this post


Link to post
Share on other sites
Q-Droid

Do those syslog entries include the time window when the server was hanging?

Share this post


Link to post
Share on other sites
mengoshmink
Posted (edited)

Do those syslog entries include the time window when the server was hanging?

Yes Q-Droid.

I rebooted.

Logged in via ssh

docker stop emby
rm -rf docker/emby/    # configuration files and folders
docker rm emby
docker rmi emby/embyserver:latest
cd docker
docker-compose up -d
## Completed setup through web interface but didn't add any libraries
docker stop emby    # this seemed to help before
docker start emby
tail -f /var/log/syslog
## Logged into web interface and added a library with multiple folders

https://pastebin.com/hg99QSFD

 

I carried on doing stuff, not using the server until I saw the broken pipe.

I rebooted and logged into web interface to get the log files and make my post on here.

 

Thanks,

Edited by mengoshmink

Share this post


Link to post
Share on other sites
Q-Droid

Is EmbyServer running as mengoshmink and have you tried letting it run as emby?

Share this post


Link to post
Share on other sites
mengoshmink

Is EmbyServer running as mengoshmink and have you tried letting it run as emby?

I am running the Emby Docker image, so there is no local emby user.

Share this post


Link to post
Share on other sites
mastrmind11
Posted (edited)

I am running the Emby Docker image, so there is no local emby user.

so then who is 

environment:
      - UID=1000
      - GID=100

?

 

These need to map to your emby user.  Type id emby on the cli

Edited by mastrmind11

Share this post


Link to post
Share on other sites
Cthalpa

Emby create an emby user when its installed, so it should be running as that user.

emby:x:998:998:Emby Server:/var/lib/emby:/usr/bin/nologin

Share this post


Link to post
Share on other sites
Q-Droid

Docker doesn't prevent you from having a local user and it's a good general practice to create dedicated accounts for server type applications.

 

That said nothing jumps out in the logs. If Emby is causing the hang I don't see anything to go on.

Share this post


Link to post
Share on other sites
Cthalpa

I would ditch it, and start over if that case was mine. And I'd probably make it a less "complicated" setup, if my skills were not with me :D

Share this post


Link to post
Share on other sites
mengoshmink

so then who is 

environment:
      - UID=1000
      - GID=100

?

 

These need to map to your emby user.  Type id emby on the cli

The UID 1000 is for my user account, GID is the local users group.

$ id mengoshmink
uid=1000(mengoshmink) gid=1000(mengoshmink) groups=1000(mengoshmink)

 

Emby create an emby user when its installed, so it should be running as that user.

emby:x:998:998:Emby Server:/var/lib/emby:/usr/bin/nologin

 

 

If I had installed the Emby .deb package, yes, I would of expected there to be a local Emby user account.

I have tried stopping Docker and using the .deb package, but I had the same problem.

 

Docker doesn't prevent you from having a local user and it's a good general practice to create dedicated accounts for server type applications.

 

That said nothing jumps out in the logs. If Emby is causing the hang I don't see anything to go on.

 

Ultimately I may run Docker containers as different local users. As you say, it is not good practice. It running as my unprivileged username I do not see as a major concern at present.

 

I would ditch it, and start over if that case was mine. And I'd probably make it a less "complicated" setup, if my skills were not with me :D

Ditch what? My configuration files? Using Emby? Using Docker? My metal server? Using a remote NAS?

 

Thank you all for your help so far,

Share this post


Link to post
Share on other sites
Cthalpa
Posted (edited)

Ditch what? My configuration files? Using Emby? Using Docker? My metal server? Using a remote NAS?

 

Thank you all for your help so far,

 

The emby installation entirely, with reinstall.

Is there any particular reason you use docker?

 

The emby installation does nothing to your system except installing its files in /var/lib/emby, and create a systemd file for start/stop/restart.

You should easily be able to install without the need for docker, or any other kind of containment.

 

If you're using a remote NAS, be sure that it's on a good tier, any interrupts on the network, or the network being too slow, will result in the kind of errors you are having now.

 

Edit: I would not run NFS over WAN. I would recommend you to use a secure connectioon, and mount the remote NAS drive as a local remote with E.G sshfs.

Edited by Cthalpa

Share this post


Link to post
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now

×
×
  • Create New...