My Load Balancing + SSL + Emby Setup

[NomadCF 15]

I run a few emby servers all using shared or common storage shares and all behind a single IP, but with multiple subdomains. And I figured I'd share my load balancing with SSL setup. 

 

Few Notes:

• I load balance between 3 emby server
• my load balancer is a linux server (Debian) using haproxy
• I use SSL termination instead of setting up and maintain SSL certs on each server
• I use a mix of letsencrypt certs and standard multi year certs.
• All my emby server run over port 80, 443. **I do not configure emby this way, I do this thru haproxy only (although I also have the standard 8096 & 8920 open). I found using 80 & 443 easier for browser access. And keeping the standard open easier for setting up Emby "Apps" (Android,IOS, Roku).
• Each emby server does have it's own subdomain name, and they all share a common one.
• When each server is "down" my local apache server gets the traffic. And displays a "emby server is down page" & info

global
        log /dev/log    local0
        log /dev/log    local1 notice
        chroot /var/lib/haproxy
        stats socket /run/haproxy/admin.sock mode 660 level admin
        stats timeout 30s
        user haproxy
        group haproxy
        daemon
        ca-base /etc/ssl/certs
        crt-base /etc/ssl/private
        ssl-default-bind-ciphers ECDH+AESGCM:DH+AESGCM:ECDH+AES256:DH+AES256:ECDH+AES128:DH+AES:ECDH+3DES:DH+3DES:RSA+AESGCM:RSA+AES:RSA+3DES:!aNULL:!MD5:!DSS
        ssl-default-bind-options no-sslv3

defaults
        log global
        mode tcp
        option tcplog
        option  dontlognull
        timeout connect 250
        timeout client  100
        timeout server  250
        errorfile 400 /etc/haproxy/errors/400.http
        errorfile 403 /etc/haproxy/errors/403.http
        errorfile 408 /etc/haproxy/errors/408.http
        errorfile 500 /etc/haproxy/errors/500.http
        errorfile 502 /etc/haproxy/errors/502.http
        errorfile 503 /etc/haproxy/errors/503.http
        errorfile 504 /etc/haproxy/errors/504.http

listen stats
    bind 0.0.0.0:9000       #Listen on all IP's on port 9000
    mode http
    balance
    timeout client 5000
    timeout connect 4000
    timeout server 30000
    #This is the virtual URL to access the stats page
    stats uri /haproxy_stats
    #Authentication realm. This can be set to anything. Escape space characters with a backslash.
    stats realm HAProxy\ Statistics
    #The user/pass you want to use. Change this password!
    stats auth hapadmin:**********
    #This allows you to take down and bring up back end servers.
    #This will produce an error on older versions of HAProxy.
    stats admin if TRUE

#Application Setup
frontend ContentSwitching
  bind 10.1.97.2:8096
  bind 10.1.97.2:8920 ssl crt /etc/haproxy/ssl/
  bind 10.1.97.2:80
  bind 10.1.97.2:443 ssl crt /etc/haproxy/ssl/
  mode http
  option httplog
  acl media_host_1 hdr(host) -i media.example.com
  acl tv_host_1 hdr(host) -i tv.example.com
  acl anime_host_1 hdr(host) -i anime.example.com
  acl movies_host_1 hdr(host) -i movies.example.com
  use_backend streaming_nodes if media_host_1
  use_backend server_tv_nodes if tv_host_1
  use_backend server_anime_nodes if anime_host_1
  use_backend server_movies_nodes if movies_host_1
  default_backend linux_www_node

backend letsencrypt
  mode http
  server letsencrypt 127.0.0.1:8880

backend server_tv_nodes
  mode http
  compression algo gzip
  compression type text/html text/plain text/css text/js
  balance roundrobin
  stick-table type ip size 200k expire 30m
  stick on src
  server node1 10.1.97.8:8096 check

backend server_anime_nodes
  mode http
  compression algo gzip
  compression type text/html text/plain text/css text/js
  balance roundrobin
  stick-table type ip size 200k expire 30m
  stick on src
  server node1 10.1.97.7:8096 check
  server node1 10.1.97.6:8096 check
  server node1 127.0.0.1:80 check backup
  
backend server_movies_nodes
  mode http
  compression algo gzip
  compression type text/html text/plain text/css text/js
  balance roundrobin
  stick-table type ip size 200k expire 30m
  stick on src
  server node1 10.1.97.11:8096 check
  server node1 10.1.97.12:8096 check
  server node1 127.0.0.1:80 check backup
  
backend streaming_nodes
  mode http
  balance roundrobin
  compression algo gzip
  compression type text/html text/plain text/css text/js
  stick-table type ip size 200k expire 5m
  stick on src
  server node1 10.1.97.12:8096 check
  server node1 10.1.97.11:8096 check
  server node1 10.1.97.8:8096 check 
  server node1 10.1.97.7:8096 check backup
  server node1 10.1.97.6:8096 check 
  server node1 127.0.0.1:80 check backup

backend linux_www_node
  mode http
  compression algo gzip
  compression type text/html text/plain text/css text/js
  balance roundrobin
  stick-table type ip size 200k expire 30m
  stick on src
  server node1 127.0.0.1:80 check

I run/use the standlone webserver with letsencrypt so that I don't need to take down either the emby servers OR apache to install or upadate the certs.

 

letsencrypt --agree-tos --renew-by-default --standalone --standalone-supported-challenges http-01 --http-01-port 8080 -d tv.exmaple.com -d movies.example.com -d anime.example.com -d media.exmaple.com

Then I just condense & place the certs for haproxy to find and use.

find /etc/letsencrypt/live/ -maxdepth 1 -mindepth 1 -type d | while read DIRNAME TRASH; do
 CERTNAME=${DIRNAME##*/}
 cat ${DIRNAME}/{fullchain.pem,privkey.pem} > /etc/haproxy/ssl/${CERTNAME}.pem
done

 

All this allows me to load balance all my server if someone does to media.example.com, encrypt all 443 traffic (without having to configure on emby) and have a "page down" for each server if that server is being referenced directly.

Edited January 6, 2016 by NomadCF

[MSattler 387]

I run a few emby servers all using shared or common storage shares and all behind a single IP, but with multiple subdomains. And I figured I'd share my load balancing with SSL setup. 

 

Few Notes:

• I load balance between 3 emby server
• my load balancer is a linux server (Debian) using haproxy
• I use SSL termination instead of setting up and maintain SSL certs on each server
• I use a mix of letsencrypt certs and standard multi year certs.
• All my emby server run over port 80, 443. **I do not configure emby this way, I do this thru haproxy only (although I also have the standard 8096 & 8920 open). I found using 80 & 443 easier for browser access. And keeping the standard open easier for setting up Emby "Apps" (Android,IOS, Roku).
• Each emby server does have it's own subdomain name, and they all share a common one.
• When each server is "down" my local apache server gets the traffic. And displays a "emby server is down page" & info

global
        log /dev/log    local0
        log /dev/log    local1 notice
        chroot /var/lib/haproxy
        stats socket /run/haproxy/admin.sock mode 660 level admin
        stats timeout 30s
        user haproxy
        group haproxy
        daemon
        ca-base /etc/ssl/certs
        crt-base /etc/ssl/private
        ssl-default-bind-ciphers ECDH+AESGCM:DH+AESGCM:ECDH+AES256:DH+AES256:ECDH+AES128:DH+AES:ECDH+3DES:DH+3DES:RSA+AESGCM:RSA+AES:RSA+3DES:!aNULL:!MD5:!DSS
        ssl-default-bind-options no-sslv3

defaults
        log global
        mode tcp
        option tcplog
        option  dontlognull
        timeout connect 250
        timeout client  100
        timeout server  250
        errorfile 400 /etc/haproxy/errors/400.http
        errorfile 403 /etc/haproxy/errors/403.http
        errorfile 408 /etc/haproxy/errors/408.http
        errorfile 500 /etc/haproxy/errors/500.http
        errorfile 502 /etc/haproxy/errors/502.http
        errorfile 503 /etc/haproxy/errors/503.http
        errorfile 504 /etc/haproxy/errors/504.http

listen stats
    bind 0.0.0.0:9000       #Listen on all IP's on port 9000
    mode http
    balance
    timeout client 5000
    timeout connect 4000
    timeout server 30000
    #This is the virtual URL to access the stats page
    stats uri /haproxy_stats
    #Authentication realm. This can be set to anything. Escape space characters with a backslash.
    stats realm HAProxy\ Statistics
    #The user/pass you want to use. Change this password!
    stats auth hapadmin:**********
    #This allows you to take down and bring up back end servers.
    #This will produce an error on older versions of HAProxy.
    stats admin if TRUE

#Application Setup
frontend ContentSwitching
  bind 10.1.97.2:8096
  bind 10.1.97.2:8920 ssl crt /etc/haproxy/ssl/
  bind 10.1.97.2:80
  bind 10.1.97.2:443 ssl crt /etc/haproxy/ssl/
  mode http
  option httplog
  acl media_host_1 hdr(host) -i media.example.com
  acl tv_host_1 hdr(host) -i tv.example.com
  acl anime_host_1 hdr(host) -i anime.example.com
  acl movies_host_1 hdr(host) -i movies.example.com
  use_backend streaming_nodes if media_host_1
  use_backend server_tv_nodes if tv_host_1
  use_backend server_anime_nodes if anime_host_1
  use_backend server_movies_nodes if movies_host_1
  default_backend linux_www_node

backend letsencrypt
  mode http
  server letsencrypt 127.0.0.1:8880

backend server_tv_nodes
  mode http
  compression algo gzip
  compression type text/html text/plain text/css text/js
  balance roundrobin
  stick-table type ip size 200k expire 30m
  stick on src
  server node1 10.1.97.8:8096 check

backend server_anime_nodes
  mode http
  compression algo gzip
  compression type text/html text/plain text/css text/js
  balance roundrobin
  stick-table type ip size 200k expire 30m
  stick on src
  server node1 10.1.97.7:8096 check
  server node1 10.1.97.6:8096 check
  server node1 127.0.0.1:80 check backup
  
backend server_movies_nodes
  mode http
  compression algo gzip
  compression type text/html text/plain text/css text/js
  balance roundrobin
  stick-table type ip size 200k expire 30m
  stick on src
  server node1 10.1.97.11:8096 check
  server node1 10.1.97.12:8096 check
  server node1 127.0.0.1:80 check backup
  
backend streaming_nodes
  mode http
  balance roundrobin
  compression algo gzip
  compression type text/html text/plain text/css text/js
  stick-table type ip size 200k expire 5m
  stick on src
  server node1 10.1.97.12:8096 check
  server node1 10.1.97.11:8096 check
  server node1 10.1.97.8:8096 check 
  server node1 10.1.97.7:8096 check backup
  server node1 10.1.97.6:8096 check 
  server node1 127.0.0.1:80 check backup

backend linux_www_node
  mode http
  compression algo gzip
  compression type text/html text/plain text/css text/js
  balance roundrobin
  stick-table type ip size 200k expire 30m
  stick on src
  server node1 127.0.0.1:80 check

I run/use the standlone webserver with letsencrypt so that I don't need to take down either the emby servers OR apache to install or upadate the certs.

 

letsencrypt --agree-tos --renew-by-default --standalone --standalone-supported-challenges http-01 --http-01-port 8080 -d tv.exmaple.com -d movies.example.com -d anime.example.com -d media.exmaple.com

Then I just condense & place the certs for haproxy to find and use.

find /etc/letsencrypt/live/ -maxdepth 1 -mindepth 1 -type d | while read DIRNAME TRASH; do
 CERTNAME=${DIRNAME##*/}
 cat ${DIRNAME}/{fullchain.pem,privkey.pem} > /etc/haproxy/ssl/${CERTNAME}.pem
done

 

All this allows me to load balance all my server if someone does to media.example.com, encrypt all 443 traffic (without having to configure on emby) and have a "page down" for each server if that server is being referenced directly.

 

How are you handling playback states, watched content state, etc since there is no ability to have a central DB?

 

Thanks!

[NomadCF 15]

I do "cheat" on this one, I tail the logs and when a user logs into a server I export that users data ever min and then diff & replace that users settings into the other servers. 

 

As a side note, every 30 mins I do a dump from the servers sqlite dbs into mariadb, consolidate the data and push the new setting back out the servers. This way I only need to edit one of them and the others will sync up within the hour. 

 

I used to use an SMB share with strict locking and had zero issues. but then sync'ing became an option I had to stop using that method. 

 

** I really do wish emby supported mariadb (mysql) natively.

[MSattler 387]

I do "cheat" on this one, I tail the logs and when a user logs into a server I export that users data ever min and then diff & replace that users settings into the other servers. 

 

As a side note, every 30 mins I do a dump from the servers sqlite dbs into mariadb, consolidate the data and push the new setting back out the servers. This way I only need to edit one of them and the others will sync up within the hour. 

 

I used to use an SMB share with strict locking and had zero issues. but then sync'ing became an option I had to stop using that method. 

 

** I really do wish emby supported mariadb (mysql) natively.

 

 

@@NomadCF

 

I wonder if we could work around Emby not supporting mariadb natively, by using your work around but in plugin form.  Basically, setup a mariadb install on a linux box, used just for that.  Then have a plugin installed on each Emby server which would do the tailing of logs, exporting to maria db, and then also updating all servers every 30 minutes.  While they are still the exact same steps, turning this into a plugin, would make it easier to manage, and more users could take advantage of this?

 

What do you think?

 

I've gone ahead and installed HAProxy and gotten it up and running, including SSL termination, so that all looks fine.  It's basically the backend DB part now that I want to get right.  If creating a plugin is not doable, can you give more details into the DB scripts, etc?

Edited January 19, 2016 by MSattler

[chiefnerd 18]

It's pretty easy setting up load balancing in Nginx. With the limitation of sqllite lite I've added in read timeouts and upstream errors that mean it's not a great issue. You can use rsync if all the server instances are on seperate machines or in this case it's just one instance using multiple IP's and Nginx uses it as queue which stops any timeouts from writing to sqllite or a multitude of other delays I've picked up in logs. This server has around 1800 users connected on average- without load balancing; errors were everywhere because of the bottleneck, once I quickly popped it in place next to no errors and I can quite happily watch 2 or 300 users watched statuses updating in real time from the dashboard.

 

I'm still tweaking this alot by the way to get the perfect balance but it works out of the box as is. We also use cloudflare- cloudlfare respects any expires headers you set so won't cache anything going to emby but if you;re really anal just create a page rule in cloudflare for the domain to bypass the cloudflare cache. The cache in cloudflare isn't the issue really, it's their 'performance rules' which you can also bypass with page rules. We have the direct paths to the media through cloudflare as well, with media images in their folders so Cloudflare in turn then serves those images instead of the servers to Kodi clients (can choose choose Webp, lossly or lossless). I also usually have it set to cache images from emby direct which works a charm but I'm troubleshooting something so I've removed the snippet from this server block.

 

upstream emby  {

 

   server SERVER1IP:PORT max_fails=2  fail_timeout=5s;

  server SERVER2IP:PORT weight=2;

   server SERVER3IP:PORT weight=4;

        

        keepalive 300; <---- allows 300 keepalive connections then drops older inactive ones to add new ones

 

}

 

server {

listen 443 ssl;

server_name yourdomain;

 

#Very Simple SSL Setup to support all clients

       ssl_session_timeout  10m;

        ssl_certificate /pathtoyour.crt;

        ssl_certificate_key /pathtoyourkey.key;

        ssl_protocols TLSv1.2 TLSv1.1 TLSv1;

        ssl_ciphers EECDH+CHACHA20:EECDH+AES128:RSA+AES128:EECDH+AES256:RSA+AES256:EECDH+3DES:RSA+3DES:!MD5;

 

    location / {

        proxy_pass http://emby;

        proxy_read_timeout 20;

        proxy_next_upstream error timeout; <----- goes to next upstream block on an error or timeout

        proxy_redirect off;

        proxy_buffering off;

        proxy_http_version 1.1;

        proxy_set_header Upgrade $http_upgrade;

        proxy_set_header Connection "upgrade";      

        proxy_set_header Host $host;$port;

        include /etc/nginx/conf.d/cfip; <-------- cloudflare real IP and X-Forwarded to double check

        proxy_set_header X-Forwarded-For $http_x_forwarded_for;

        proxy_set_header X-Real-IP $remote_addr;

        proxy_set_header X-Forwarded-Proto https; <-------- sometimes wasn't writing protocol as https so removing variable forces it to write https

        proxy_set_header X-Forwarded-Protocol https;

        add_header Cache-Control "private"; <------ no caching headers - add different blocks for different file types with different expires

        expires off; <------ no caching headers - add different blocks for different file types with different expires

        sendfile on;

        tcp_nodelay on;

        tcp_nopush on;

        

        }

 

}

[Luke 37007]

Great info, thanks !

[Untoten 295]

Bump, this is incredibly well documented and absolutely great information!

[Untoten 295]

I do "cheat" on this one, I tail the logs and when a user logs into a server I export that users data ever min and then diff & replace that users settings into the other servers. 

 

As a side note, every 30 mins I do a dump from the servers sqlite dbs into mariadb, consolidate the data and push the new setting back out the servers. This way I only need to edit one of them and the others will sync up within the hour. 

 

I used to use an SMB share with strict locking and had zero issues. but then sync'ing became an option I had to stop using that method. 

 

** I really do wish emby supported mariadb (mysql) natively.

Can you document this process a bit more in depth?  I absolutely love this post, anything you can provide, code, etc to help better execute this workaround would be greatly appreciated ))

[ddurdle 75]

Your sql scripts to pull the user data would be a great asset to the community.  I assume you are pulling the data to sync straight from sqlite?

[ebr 14903]

Your sql scripts to pull the user data would be a great asset to the community.  I assume you are pulling the data to sync straight from sqlite?

 

Just be aware things like this could break at any time...

[ddurdle 75]

Just be aware things like this could break at any time...

 

Could this be done via the API?  I was checking into the API yesterday through the API browser.  Some of the calls were missing some details.  I could see methods to delete playback status.  Could this conceivably controlled through the API today (fetch the status for user X from server Y and insert that status on user X from server Z)?

[ebr 14903]

Could this be done via the API?  I was checking into the API yesterday through the API browser.  Some of the calls were missing some details.  I could see methods to delete playback status.  Could this conceivably controlled through the API today (fetch the status for user X from server Y and insert that status on user X from server Z)?

 

Yes but it can also be done with the Server Configuration Backup plug-in.  You can restore any user's watched/favorite/resume status into any other user you wish.

[ddurdle 75]

Yes but it can also be done with the Server Configuration Backup plug-in.  You can restore any user's watched/favorite/resume status into any other user you wish.

This is actually how I'm doing it today, but it is a bit manual since I need to use the UI to activate the import.  Is there an API call to do the restore part?

[numberedthought 1]

Please say this is possible via the builtin or plugin API.  This would make my year.

[ddurdle 75]

Please say this is possible via the builtin or plugin API.  This would make my year.

I can't find one, but I hope I simply overlooked it and hoping someone can confirm affirmative that it is there.

[Luke 37007]

Load balancing is something we're interested in for the future. Thanks.

[ivaat 0]

edit: got working

Edited November 16, 2018 by ivaat

[bpbenich 10]

I do "cheat" on this one, I tail the logs and when a user logs into a server I export that users data ever min and then diff & replace that users settings into the other servers. 

 

As a side note, every 30 mins I do a dump from the servers sqlite dbs into mariadb, consolidate the data and push the new setting back out the servers. This way I only need to edit one of them and the others will sync up within the hour. 

 

I used to use an SMB share with strict locking and had zero issues. but then sync'ing became an option I had to stop using that method. 

 

** I really do wish emby supported mariadb (mysql) natively.

 

 

Is this something you happen to maintain? I plan on upgrading my setup in a month or so and this problem is the only thing preventing me from implementing load balancing.

[cuzz1369 7]

Is this something you happen to maintain? I plan on upgrading my setup in a month or so and this problem is the only thing preventing me from implementing load balancing.

Just bumping this ancient thread. If anyone has an elegant solution to sync this data?

 

Sent from my SM-G965W using Tapatalk

[Flippz 17]

On 13/08/2018 at 19:39, Luke said:

Load balancing is something we're interested in for the future. Thanks.

Any work being done to this?

Edited September 7, 2021 by Flippz

[Luke 37007]

1 hour ago, Flippz said:

Any work being done to this?

Hi, not yet, but it's possible for the future. thanks.

[Flippz 17]

Any news?

[Luke 37007]

On 10/21/2022 at 1:19 PM, Flippz said:

Any news?

Hi, not yet, sorry.