Emby Server - High Availability

[Luke 36997]

it looks pretty generalized so it might work as-is.

[ebr 14902]

On 10/11/2020 at 6:49 AM, ObsidianGr said:

I understand fully that with there is a device limit of 15 concurrent users with the premiere version

Hi. Our device limit is 25 and it is not concurrent users.

 Is there a limit to Emby Premiere

[JamieWasTaken 1]

Is there any plans in the near future to bring HA support to Emby?

[Luke 36997]

It's something that we're interested in for future updates. Thanks for the feedback.

[tznemet 7]

how can i make the cache bigger ....
have a extra ssd chip on pci-ex slot 
thanks

[CoderNemes 0]

I'd be interested in something like this also.  Built a four node docker swarm cluster out of four Raspberry Pi 4s.  One node has emby and depending on the compression, can take about 15% cpu per stream with hardware encoding, so would be nice to be able to support load balancing this across more nodes.  Had to set up the containers using the standalone docker on the node so it can mount the necessary video devices since swarm doesn't support that currently.  Then use a proxy from the swarm to the standalone container w/ ssl termination.  All the nodes share storage for docker volume data to an 500 gb ssd mounted over nfs from the manager node and a 5TB raid 10 share for multimedia also shared from the manager node over nfs.
 

Thought this way, if I needed more power, could just shell out another $50 for another 4 cores + 4 gb of ram lol.

[Thuzad 42]

On 10/12/2020 at 2:24 AM, pir8radio said:

maybe @Luke and team could create an emby specific version of this that can work well with emby without any hackery

 

On 10/12/2020 at 2:41 AM, Luke said:

it looks pretty generalized so it might work as-is.

 

I tried to set it up by installing emby on a remote server and using the file /opt/emby-server/bin/ffmpeg and /opt/emby-server/bin/ffprobe in the configuration.

Unfortunately and as I suspected, it doesn't work at all. It returns a stream error and it doesn't detect the hardware I'm using. I think there are several problems but I'm not sure.

If you have an idea how to set up this tool that would be great. But I understand that you wouldn't want to help me too much on a tool that hasn't been developed by yourself.

If you want I can send you the server logs.

 

Concerning the setup, I followed the documentation by replacing only the jellyfin path by emby.

my rffmpeg.cfg:

rffmpeg:
    # rffmpeg state configuration - YOU SHOULD NOT ALTER THESE
    state:
        # Temporary directory to store state
        tempdir: "/run/shm/rffmpeg"

        # Filename format for state instance files
        filename: "instance_{pid}.pid"

        # Contents of the state instance file
        contents: "{host}"

    # Logging configuration
    logging:
        # Enable or disable file logging
        file: true

        # Log messages to this file - ensure the user running rffmpeg can write to it
        logfile: "/var/log/emby/rffmpeg.log"

    # Remote (SSH) configuration
    remote:
        # A YAML list of remote hosts to connect to
        hosts:
            - localhost
#            - 192.168.1.182
#            - gpu2
#            - gpu3

        # The remote SSH user to connect as
        user: emby

        # A YAML list of additional SSH arguments (e.g. private keys),
        # one line per space-separated argument element.
        args:
            - "-i"
            - "/var/lib/emby/.ssh/id_rsa"


    # Remote command configuration
    commands:
        # A YAML list of prefixes to the ffmpeg command (e.g. sudo, nice, etc.),
        # one line per space-separated command element.
        pre:
            - ""

        # The (remote) ffmpeg and ffprobe command binary paths
        #ffmpeg: "/usr/lib/jellyfin-ffmpeg/ffmpeg"
        #ffprobe: "/usr/lib/jellyfin-ffmpeg/ffprobe"
        #emby-ffmpeg
        ffmpeg: "/opt/emby-server/bin/ffmpeg"
        ffprobe: "/opt/emby-server/bin/ffprobe"

        # An optional local fallback ffmpeg and ffprobe, if you wish this to be different from the above paths
        fallback_ffmpeg: "/opt/emby-server/bin/ffmpeg"
        fallback_ffprobe: "/opt/emby-server/bin/ffprobe"

ffmpeg-directstream-6f44c101-89ed-4a6b-b02f-3cb00a8ec86b_1.txt:

2021-02-27 02:40:09.191
Operating System: Linux version 5.4.0-65-generic (buildd@lcy01-amd64-018) (gcc version 9.3.0 (Ubuntu 9.3.0-17ubuntu1~20.04)) #73-Ubuntu SMP Mon Jan 18 17:25:17 UTC 2021
OS/Process: x64/x64
Emby Server version: 4.6.0.28
Operating system: Unix 5.4.0.65
Command Line: /opt/emby-server/system/EmbyServer.dll -programdata /var/lib/emby -ffdetect /opt/emby-server/bin/ffdetect -ffmpeg /opt/emby-server/bin/ffmpeg -ffprobe /opt/emby-server/bin/ffprobe -restartexitcode 3 -updatepackage emby-server-deb_{version}_amd64.deb

App: Emby Web 4.6.0.28
Chrome

http://streaming/emby/videos/842685/hls1/main/0.ts?DeviceId=8c587468-02b6-45df-acfa-dc65450fab59&MediaSourceId=b9285119b0eafe910557828f7bff5efa&PlaySessionId=a19737aeeb4d43b4bc5ff991ef494243&api_key=65c7b9a68408457fac1c5fbe0aae5b0d&VideoCodec=h264&AudioCodec=mp3,aac&VideoBitrate=139616000&AudioBitrate=384000&AudioStreamIndex=1&SubtitleStreamIndex=3&SubtitleMethod=Hls&TranscodingMaxAudioChannels=2&SegmentContainer=ts&MinSegments=1&BreakOnNonKeyFrames=True&ManifestSubtitles=vtt&h264-profile=high,main,baseline,constrainedbaseline,high10&h264-level=52&TranscodeReasons=SubtitleCodecNotSupported

{"Protocol":"File","Id":"b9285119b0eafe910557828f7bff5efa","Path":"/media/storage4/films/Paydirt (2020)/Paydirt.2020.MULTi.WEBrip.1080p.x264_JiHEFF.mkv","Type":"Default","Container":"mkv","Size":1992704871,"Name":"Paydirt.2020.MULTi.WEBrip.1080p.x264_JiHEFF","IsRemote":false,"RunTimeTicks":51225920000,"SupportsTranscoding":true,"SupportsDirectStream":true,"SupportsDirectPlay":true,"IsInfiniteStream":false,"RequiresOpening":false,"RequiresClosing":false,"RequiresLooping":false,"SupportsProbing":false,"MediaStreams":[{"Codec":"h264","ColorTransfer":"bt709","ColorPrimaries":"bt709","ColorSpace":"bt709","TimeBase":"1/1000","CodecTimeBase":"1001/48000","VideoRange":"SDR","DisplayTitle":"1080p H264","NalLengthSize":"4","IsInterlaced":false,"IsAVC":true,"BitRate":3112025,"BitDepth":8,"RefFrames":1,"IsDefault":true,"IsForced":false,"Height":796,"Width":1920,"AverageFrameRate":23.976025,"RealFrameRate":23.976025,"Profile":"High","Type":"Video","AspectRatio":"480:199","Index":0,"IsExternal":false,"IsTextSubtitleStream":false,"SupportsExternalStream":false,"Protocol":"File","PixelFormat":"yuv420p","Level":41,"IsAnamorphic":false},{"Codec":"ac3","Language":"fre","TimeBase":"1/1000","CodecTimeBase":"1/48000","DisplayTitle":"French AC3 5.1 (Default)","DisplayLanguage":"French","IsInterlaced":false,"ChannelLayout":"5.1","BitRate":384000,"Channels":6,"SampleRate":48000,"IsDefault":true,"IsForced":false,"Type":"Audio","Index":1,"IsExternal":false,"IsTextSubtitleStream":false,"SupportsExternalStream":false,"Protocol":"File"},{"Codec":"aac","Language":"eng","TimeBase":"1/1000","CodecTimeBase":"1/48000","DisplayTitle":"English AAC 5.1","DisplayLanguage":"English","IsInterlaced":false,"ChannelLayout":"5.1","BitRate":320000,"Channels":6,"SampleRate":48000,"IsDefault":false,"IsForced":false,"Profile":"LC","Type":"Audio","Index":2,"IsExternal":false,"IsTextSubtitleStream":false,"SupportsExternalStream":false,"Protocol":"File"},{"Codec":"subrip","Language":"fre","TimeBase":"1/1000","CodecTimeBase":"0/1","Title":"FR forcé","DisplayTitle":"French (Default Forced SUBRIP)","DisplayLanguage":"French","IsInterlaced":false,"IsDefault":true,"IsForced":true,"Type":"Subtitle","Index":3,"IsExternal":false,"IsTextSubtitleStream":true,"SupportsExternalStream":true,"Protocol":"File"},{"Codec":"subrip","Language":"fre","TimeBase":"1/1000","CodecTimeBase":"0/1","Title":"FR complet","DisplayTitle":"French (SUBRIP)","DisplayLanguage":"French","IsInterlaced":false,"IsDefault":false,"IsForced":false,"Type":"Subtitle","Index":4,"IsExternal":false,"IsTextSubtitleStream":true,"SupportsExternalStream":true,"Protocol":"File"},{"Codec":"subrip","Language":"eng","TimeBase":"1/1000","CodecTimeBase":"0/1","Title":"English full","DisplayTitle":"English (SUBRIP)","DisplayLanguage":"English","IsInterlaced":false,"IsDefault":false,"IsForced":false,"Type":"Subtitle","Index":5,"IsExternal":false,"IsTextSubtitleStream":true,"SupportsExternalStream":true,"Protocol":"File"}],"Formats":[],"Bitrate":3112025,"RequiredHttpHeaders":{},"ReadAtNativeFramerate":false}

>>>>>>  User policy for Floflobel
        EnablePlaybackRemuxing: True
        EnableVideoPlaybackTranscoding: True
        EnableAudioPlaybackTranscoding: True


>>>>>>  Processing Plan
Info    Name                                        CanDoInHardware      WillDoInHardware     Reason                                  
Info    Copy video stream                        >> False                False                Software Codec                           
Info    VideoInput                               >> False                False                Not a hardware decoder                   
Info    VideoOutput                              >> False                False                Not a hardware encoder                   
Info    Copy video stream                        >> False                False                Software Codec                           

>>>>>>  Projected Processing Formats
Info    Previous                HW-Context   Format       SW-Format           Size   Next
Info    h264                 >> -            yuv420p      yuv420p         1920x796 >> 

/opt/emby-server/bin/ffmpeg -y -copyts -start_at_zero -f matroska,webm -noaccurate_seek -c:v:0 h264 -i "/media/storage4/films/Paydirt (2020)/Paydirt.2020.MULTi.WEBrip.1080p.x264_JiHEFF.mkv" -map 0:0 -map 0:1 -sn -c:v:0 copy -bsf:v:0 h264_mp4toannexb -c:a:0 copy -metadata:s:a:0 language=fre -disposition:a:0 default -copypriorss:a:0 0 -max_delay 5000000 -avoid_negative_ts disabled -f segment -map_metadata -1 -map_chapters -1 -segment_format mpegts -segment_list "/var/lib/emby/transcoding-temp/transcoding-temp/8D0F0E.m3u8" -segment_list_type m3u8 -segment_time 6 -segment_start_number 0 -break_non_keyframes 1 -individual_header_trailer 0 -write_header_trailer 0 "/var/lib/emby/transcoding-temp/transcoding-temp/8D0F0E_%d.ts" -map 0:3 -vn -an -c:s:0 webvtt -max_delay 5000000 -avoid_negative_ts disabled -f segment -map_metadata -1 -map_chapters -1 -segment_format webvtt -segment_list "/var/lib/emby/transcoding-temp/transcoding-temp/8D0F0E_s3.m3u8" -segment_list_type m3u8 -segment_time 6 -segment_start_number 0 -break_non_keyframes 1 -individual_header_trailer 1 -write_header_trailer 0 -write_empty_segments 1 "/var/lib/emby/transcoding-temp/transcoding-temp/8D0F0E_s3_%d.vtt" -map 0:4 -vn -an -c:s:0 webvtt -max_delay 5000000 -avoid_negative_ts disabled -f segment -map_metadata -1 -map_chapters -1 -segment_format webvtt -segment_list "/var/lib/emby/transcoding-temp/transcoding-temp/8D0F0E_s4.m3u8" -segment_list_type m3u8 -segment_time 6 -segment_start_number 0 -break_non_keyframes 1 -individual_header_trailer 1 -write_header_trailer 0 -write_empty_segments 1 "/var/lib/emby/transcoding-temp/transcoding-temp/8D0F0E_s4_%d.vtt" -map 0:5 -vn -an -c:s:0 webvtt -max_delay 5000000 -avoid_negative_ts disabled -f segment -map_metadata -1 -map_chapters -1 -segment_format webvtt -segment_list "/var/lib/emby/transcoding-temp/transcoding-temp/8D0F0E_s5.m3u8" -segment_list_type m3u8 -segment_time 6 -segment_start_number 0 -break_non_keyframes 1 -individual_header_trailer 1 -write_header_trailer 0 -write_empty_segments 1 "/var/lib/emby/transcoding-temp/transcoding-temp/8D0F0E_s5_%d.vtt"

ssh: relocation error: ssh: symbol ENGINE_register_all_complete version OPENSSL_1_1_0 not defined in file libcrypto.so.1.1 with link time reference

 

[softworkz 3325]

Executing ffmpeg remotely won't get you very far as long as the remote transcoding results aren't streamed directly from the transcoding servers to clients.

For a professional solution, there are quite a few more things required.

I like that direction. Ideas and concepts for this are ready for a long time, but it's also clear that this would be a feature for a very small audience and I suspect that a feature like this can't be delivered without having a different pricing model for it (Disclaimer: not an official statement).

[Thuzad 42]

Yes and this is not really a high availability solution, it's more like distributing the transcoding load on a server.

I fully understand that this is quite complex to implement in a software in general when it hasn't been designed for that.

I'll keep digging into my problem and see if I can do something about it.

Thanks for your feedback @softworkz

[softworkz 3325]

35 minutes ago, Floflobel said:

Yes and this is not really a high availability solution, it's more like distributing the transcoding load on a server.

I fully understand that this is quite complex to implement in a software in general when it hasn't been designed for that.

Correct, for a real high availability solution with load-balancing and auto-failover, the following steps would be required:

• Support storing the data in an external database 
• You will need two DB servers in a cluster/failover configuration supporting transactional replication
  (bye bye MySQL...)
• Change the server to store all session state in the database
• Introduce "Roles" for the server
  so you can have for example a server that is responsible only for library management and metadata retrieval, servers for servicing user requests (http) and finally servers that are doing transcoding and serving the media to clients.

The ability to offload transcoding to "transcoding slave servers" would surely be a reasonable part here, even though not necessarily required for "High Availability".

On the other side, the ability to offload transcoding alone would be much easier to implement and surely useful for certain deployments.

[Thuzad 42]

Yes, I can see that there is some work to be done before we can have this solution. 
Indeed mysql does not allow that, a database such as postgres could do it and it would be really cool. But I know that there are some requests for new features open on this subject.

For my problem I have the impression that an openssl library named "libcrypto" is missing. I think there is a problem when it tries to execute the ffmpeg remotely through SSH.

I've been doing a lot of research to try to make this solution work for me, but I haven't been able to make any progress. So if anyone has any idea how to unblock it it would be really cool and it would take some of the load off my main server.

Although I understand that this do-it-yourself is not adequate, I would really like to do it.

 

I tried this to allow loading the libraries but it didn't work, I still have the same error:

    # Remote command configuration
    commands:
        # A YAML list of prefixes to the ffmpeg command (e.g. sudo, nice, etc.),
        # one line per space-separated command element.
        pre:
            - "export LD_LIBRARY_PATH=/opt/emby-server/lib"

 

Edited March 4, 2021 by Floflobel

[softworkz 3325]

This looks like you're on Linux - and I think I might have some good new for you: 

The latest Emby installations on Linux are including stub scripts which allow you to execute the ffmpeg delivered with Emby from an external context (e.g. shell).

Just take a look at the /opt/emby-server/bin directory. There you can see these scripts named like 'ffmpeg-emby'. Calling these instead of the binary will allow you to run ffmpeg without having to deal with any environment setup.

Hope this helps. even though I don't see much benefit this way
(other than having some fun trying to get it going)

Good Luck!

[softworkz 3325]

Please note my question here: 

 

[nwilging 0]

Just wanted to chime in as I've been working on setting up rffmpeg as well

I have a nearly identical config to @Floflobel:

rffmpeg:
    # rffmpeg state configuration - YOU SHOULD NOT ALTER THESE
    state:
        # Temporary directory to store state
        tempdir: "/run/shm/rffmpeg"

        # Filename format for state instance files
        filename: "instance_{pid}.pid"

        # Contents of the state instance file
        contents: "{host}"

    # Logging configuration
    logging:
        # Enable or disable file logging
        file: true

        # Log messages to this file - ensure the user running rffmpeg can write to it
        logfile: "/var/log/emby/rffmpeg.log"

    # Remote (SSH) configuration
    remote:
        # A YAML list of remote hosts to connect to; either direct list or name/weight supported
        hosts:
            - localhost
            - name: transcoder-1
              weight: 2   # Relative to any non-weighted hosts which have weight 1

        # The remote SSH user to connect as
        user: emby

        # A YAML list of additional SSH arguments (e.g. private keys),
        # one line per space-separated argument element.
        args: 
            - "-i"
            - "/var/lib/emby/.ssh/id_rsa"


    # Remote command configuration
    commands:
        # By default rffmpeg uses $PATH to find the "ssh" program; use this option to set a full path
        # to an SSH binary if you want to override the default.
        ssh: "/usr/bin/ssh"

        # A YAML list of prefixes to the ffmpeg command (e.g. sudo, nice, etc.),
        # one line per space-separated command element.
        pre:
            - ""

        # The (remote) ffmpeg and ffprobe command binary paths
        ffmpeg: "/usr/bin/ffmpeg"
        ffprobe: "/usr/bin/ffprobe"

        # An optional local fallback ffmpeg and ffprobe, if you wish this to be different from the above paths
        fallback_ffmpeg: "/opt/emby-server/bin/emby-ffmpeg"
        fallback_ffprobe: "/opt/emby-server/bin/emby-ffprobe"

I'm running into the same SSH error, but also have a bit more information I've found (that the devs probably already know).

So in order for emby to transcode anything, it must use the `/opt/emby-server/bin/emby-ffmpeg` binary. I have had no luck whatsoever pointing emby to the `/opt/emby-server/bin/ffmpeg` binary, and I assume that ffmpeg binary is compiled specifically for emby (as noted in it's version), and is intended to be used with the `/opt/emby-server/lib` libraries. As you can see by the exports in `/opt/emby-server/bin/emby-ffmpeg`.

So really the issue here is that SSH does not work with (I think) libcrypto library that is included in emby. And you can't remove the LD library export or else emby won't be able to transcode.

I haven't dug any further into it yet, but I intend to in the near future. Ultimately we just need a way to execute `ssh` from python, using the libraries in `/opt/emby-server/lib`

There are likely more complicated solutions as well. Or maybe less complicated, depending on how difficult it would be to actually get ssh to run. But I was thinking of something as far as raw tcp sockets, a client/server type setup where your transcoding hosts will receive a "ping" from the main host and start transcoding. Obviously with all the same network shares and whatnot so the transcoding temp directory is shared between the main emby install and the transcoders

[Thuzad 42]

@nwilging We have searched a lot with the developer of rffmpeg where the problem could come from but we did not manage to solve it.
You can find all the information about it on the issue I opened:

https://github.com/joshuaboniface/rffmpeg/issues/10

[softworkz 3325]

If there are library load errors when running the Emby-ffmpeg, you need to use the stub-scripts - eg. ffmpeg-emby instead of ffmpeg.

[AndyXH 70]

Hi all, is the HA still part of the future ? 

[softworkz 3325]

2 hours ago, andreaxheli said:

Hi all, is the HA still part of the future ? 

This part has been covered pretty detailed in the following topic, starting from the referenced comment to the end:

On 3/13/2021 at 8:26 AM, softworkz said:

You might also find this recent conversation of interest: