Jump to content

Reverse proxy with SSL, hostname routing, and Emby/OpenVPN port sharing


gstuartj

Recommended Posts

gstuartj

I've seen a few reverse-proxy config files posted and thought I'd share my own setup. Like many, I use Nginx to add SSL, etc to Emby, but I have HAProxy sitting in front of it doing hostname routing. Why?

  • I can host both HTTPS and OpenVPN services on the same port (443/TCP) via TCP proxying. SSH would also work.
  • Hostname routing is done at TCP level via SNI, no decryption required. Maintains end-to-end encryption to each downstream server.
  • Powerful configurability for later needs

I also have my Emby hostname publicly routed through CloudFlare, but on my LAN the hostname goes directly to my local address, bypassing CloudFlare. (Better direct-play performance) I did some juggling to get the correct client IP in both scenarios and pass it to Emby.

In addition, traffic is only allowed to hit the virtual host from the LAN or from CloudFlare (with a client certificate). I did my best to security harden the config.

Please offer any suggestions or ask me if you have any questions on how things work.

haproxy.cfg

global
	log /dev/log	local0
	log /dev/log	local1 notice
	chroot /var/lib/haproxy
	user haproxy
	group haproxy
	daemon


defaults
	log	global
	option	dontlognull
	# Safety first
	option http-server-close
	timeout http-request 5s
        timeout connect 5000
        timeout client  30000
        timeout server  30000


frontend tls_router
	bind *:443
	mode tcp
	option tcpka

	# Define known networks for later use
	acl local src 127.0.0.0/8 192.168.0.0/24
	acl cloudflare src 199.27.128.0/21 173.245.48.0/20 103.21.244.0/22 103.22.200.0/22 103.31.4.0/22 141.101.64.0/18 108.162.192.0/18 190.93.240.0/20 188.114.96.0/20 197.234.240.0/22 198.41.128.0/17 162.158.0.0/15 104.16.0.0/12 172.64.0.0/13 2400:cb00::/32 2606:4700::/32 2803:f800::/32 2405:b500::/32 2405:8100::/32	

	#Rate-limiting measures, in case of DoS
        # Table for connection tracking  
        stick-table type ip size 100k expire 30s store conn_cur

        # Allow known CloudFlare IPs to bypass the rate-limiting
        tcp-request connection accept if cloudflare
        # Reject connection if client has more than 10 open 
        tcp-request connection reject if { src_conn_cur ge 10 }
        tcp-request connection track-sc1 src

	# Max inspection delay for SNI routing
        tcp-request inspect-delay 2s
	# Accept only SSL/TLS traffic
        tcp-request content accept if { req_ssl_hello_type 1 }

	# If SNI hostname is known server AND is from known IP, pass to backend
        acl media_sni req_ssl_sni -i emby.hostname
        use_backend crane_tls if media_sni cloudflare or local

        # Other hostname, allow traffic from all
        acl other_sni req_ssl_sni -i other.hostname
        use_backend otherserver_tls if other_sni

	# Root hostname
        acl root_sni req_ssl_sni -i example.domain
        use_backend crane_tls if root_sni cloudflare

	# If SNI hostname isn't recognized, reject connection. Must come last.
	acl fallback_sni req_ssl_sni
	tcp-request connection reject if fallback_sni

	# If SNI hostname not provided, pass to OpenVPN backend to deal with
	default_backend openvpn


backend crane_tls
	mode tcp
	option tcpka
	# Proxy protocol to preserve client info. Needs to be enabled in nginx
	server server_crane_tls localhost:1443 send-proxy


backend otherserver_tls
    mode tcp
    option tcpka
    # Proxy protocol to preserve client info. Needs to be enabled in nginx
    server server_other_tls 192.168.0.23:1443 send-proxy


backend openvpn
	mode tcp
	server server_ovpn 192.168.0.32:1195

nginx

# emby reverse proxy
#
# Intended to sit downstream from HAProxy, uses proxy protocol.
# Allows traffic from the local network, or from WAN w/ CloudFlare client cert
#

server {
	# Using proxy protocol to get client info passed from HAProxy
	listen localhost:1443 ssl http2 proxy_protocol;
	listen [::1]:1443 ssl http2 proxy_protocol;
        keepalive_timeout 180;

	# Who the hell am I?! Where's my stuff?!
	server_name emby.hostname;
	root /var/www/emby.hostname/public_html;

        # Import some solid TLS settings
        include snippets/solidtls.conf;

        # Certs
        ssl_certificate /etc/nginx/ssl/emby.hostname/fullchain.pem;
        ssl_certificate_key /etc/nginx/ssl/emby.hostname/key.pem;
        ssl_trusted_certificate /etc/nginx/ssl/emby.hostname/fullchain.pem;
        ssl_client_certificate /etc/nginx/ssl/cf-origin-pull-ca.pem;

        # Client cert optional. We'll do our own access control rules in /
        ssl_verify_client optional;

	# Let's get the real client info from upstream proxy
	include snippets/realip-cf-haproxy.conf;

	# Set headers with real client info for downstream app
        proxy_set_header Host $host;
	proxy_set_header X-Real-IP $real_ip;
	proxy_set_header X-Forwarded-For $real_ip;
	proxy_set_header X-Forwarded-Proto $real_ip;
	proxy_set_header X-Forwarded-Protocol $scheme;

        # Turn off buffering for streaming purposes
        proxy_buffering off;
        # No request rewriting
        proxy_redirect off;

        # Send websocket data to the backend, courtesy @[member="Karbowiak"] on Emby forum
        proxy_http_version 1.1;
        proxy_set_header Upgrade $http_upgrade;
        proxy_set_header Connection "upgrade";

        # Test if IP from HAProxy is local
	set $is_local_client false;
        if ( $proxy_protocol_addr ~ 192.168.0.([100-254]) ) {
                set $is_local_client true;
        }


	# Force the use of our custom robots.txt
	location /robots.txt {
                try_files $uri /var/www/emby.hostname/public_html/robots.txt;
        }


	# Catch-all. Performs authentication based on client IP or certificate.
	location / {
		# We'll return 418 later if client is authorized
		error_page 418 = @authorized;

		#Access rules
		# Allow local clients without client cert
		if ( $is_local_client = true ) { return 418; }
		# Does client have valid client certificate?
		if ( $ssl_client_verify = "SUCCESS" ) { return 418; }

		# Deny all others, return 403 unauthorized
		return 403;
	}


	# Handles 418 responses for authorized users. Does the actual proxying.
	location @authorized {
		proxy_pass http://localhost:8096;
	}


	# Needed for Let's Encrypt certificate renewals
	include /etc/nginx/snippets/letsencrypt.conf;

}

snippets/solidtls.conf

# Trustworthy SSL/TLS settings in a box
#

ssl on;
gzip off; # gzip allows for some attacks

# General params
ssl_session_timeout 1d;
ssl_session_cache shared:SSL:10m;
ssl_session_tickets off;

# Always use custom-generated DH params >2048 bits
ssl_dhparam /etc/nginx/ssl/dhparam.pem;

#Intermediate cipher config
ssl_protocols TLSv1 TLSv1.1 TLSv1.2;
ssl_ciphers 'ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-AES256-GCM-SHA384:DHE-RSA-AES128-GCM-SHA256:DHE-DSS-AES128-GCM-SHA256:kEDH+AESGCM:ECDHE-RSA-AES128-SHA256:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA:ECDHE-ECDSA-AES128-SHA:ECDHE-RSA-AES256-SHA384:ECDHE-ECDSA-AES256-SHA384:ECDHE-RSA-AES256-SHA:ECDHE-ECDSA-AES256-SHA:DHE-RSA-AES128-SHA256:DHE-RSA-AES128-SHA:DHE-DSS-AES128-SHA256:DHE-RSA-AES256-SHA256:DHE-DSS-AES256-SHA:DHE-RSA-AES256-SHA:ECDHE-RSA-DES-CBC3-SHA:ECDHE-ECDSA-DES-CBC3-SHA:AES128-GCM-SHA256:AES256-GCM-SHA384:AES128-SHA256:AES256-SHA256:AES128-SHA:AES256-SHA:AES:CAMELLIA:DES-CBC3-SHA:!aNULL:!eNULL:!EXPORT:!DES:!RC4:!MD5:!PSK:!aECDH:!EDH-DSS-DES-CBC3-SHA:!EDH-RSA-DES-CBC3-SHA:!KRB5-DES-CBC3-SHA';

#Modern cipher config
#ssl_protocols TLSv1.1 TLSv1.2;
#ssl_ciphers 'ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-AES256-GCM-SHA384:DHE-RSA-AES128-GCM-SHA256:DHE-DSS-AES128-GCM-SHA256:kEDH+AESGCM:ECDHE-RSA-AES128-SHA256:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA:ECDHE-ECDSA-AES128-SHA:ECDHE-RSA-AES256-SHA384:ECDHE-ECDSA-AES256-SHA384:ECDHE-RSA-AES256-SHA:ECDHE-ECDSA-AES256-SHA:DHE-RSA-AES128-SHA256:DHE-RSA-AES128-SHA:DHE-DSS-AES128-SHA256:DHE-RSA-AES256-SHA256:DHE-DSS-AES256-SHA:DHE-RSA-AES256-SHA:!aNULL:!eNULL:!EXPORT:!DES:!RC4:!3DES:!MD5:!PSK';

#Perfect ssllabs score cipher list. Not super usable, works for CF-only setups
#ssl_protocols TLSv1.2;
#ssl_ciphers AES256+EECDH:AES256+EDH:!aNULL;

# No downgrade attacks
ssl_prefer_server_ciphers on;

# HSTS
add_header Strict-Transport-Security "max-age=63072000; includeSubdomains; preload";
add_header X-Frame-Options DENY;
add_header X-Content-Type-Options nosniff;

# Stapling
ssl_stapling on;
ssl_stapling_verify on;

# DNS resolvers to use for cert verification
resolver 8.8.8.8 8.8.4.4 valid=300s;
resolver_timeout 5s;

snippets/realip-cf-haproxy.conf

#Local traffic, including HAProxy on localhost
set_real_ip_from 127.0.0.0/8;

#CloudFlare
set_real_ip_from 199.27.128.0/21;
set_real_ip_from 173.245.48.0/20;
set_real_ip_from 103.21.244.0/22;
set_real_ip_from 103.22.200.0/22;
set_real_ip_from 103.31.4.0/22;
set_real_ip_from 141.101.64.0/18;
set_real_ip_from 108.162.192.0/18;
set_real_ip_from 190.93.240.0/20;
set_real_ip_from 188.114.96.0/20;
set_real_ip_from 197.234.240.0/22;
set_real_ip_from 198.41.128.0/17;
set_real_ip_from 162.158.0.0/15;
set_real_ip_from 104.16.0.0/12;
set_real_ip_from 172.64.0.0/13;
set_real_ip_from 2400:cb00::/32;
set_real_ip_from 2606:4700::/32;
set_real_ip_from 2803:f800::/32;
set_real_ip_from 2405:b500::/32;
set_real_ip_from 2405:8100::/32;

#Default state, assume local traffic/proxy protocol header
set $forwarded_ip_header proxy_protocol;
set $real_ip $proxy_protocol_addr;

# Is Cloudflare header set? Use that header/addr instead of proxy_protocol
if ($http_cf_connecting_ip) {
	set $forwarded_ip_header CF-Connecting-IP;
	set $real_ip $http_cf_connecting_ip;
}

# Tell Nginx which header to use
real_ip_header $forwarded_ip_header;
Edited by gstuartj
  • Like 6
Link to comment
Share on other sites

  • 1 month later...
jaybroni

+1 Like from me. Nice work, very impressive. Too bad it's way over my head. 

 

If you were only 10% as techy as you are, how would you setup a basic SSL Certificate?

 

I'd like to set one up, using a free SSL from one of those free CA sites online, so that my users don't get red flags when they connect using a 'self-signed cert'. How would I go about doing this? If you want to point me to a tutorial that would be great.

Link to comment
Share on other sites

Cerothen

If you have a Linux box kicking around letsencrypt is the way to go, I tried it out for the first time last week and it was a breeze, just follow the getting started page and after about 5 mins you can have a signed cert with every (sub)domain you could want.

Link to comment
Share on other sites

  • 4 weeks later...
Cerothen

Just remember that the outputs are not emby friendly out of the gate, I added the below into my renew script so that it takes the live certificates after the renew and creates a bundle for HAproxy and a pfx without a password for emby.

# cd /etc/letsencrypt/live/<<domain>>
# cat fullchain.pem privkey.pem > bundle.pem
# openssl pkcs12 -export -out hostcert.pfx -inkey privkey.pem -in cert.pem -certfile chain.pem -passout pass:

 
The cert's aren't valid for all that long (3-ish months) but if you have something that stops emby, overwrites the certificate that is stored in "<<emby root>>/ssl/cert_9c31b7884ea5475c8687970fc5996297.pfx" and then starts emby up again then that should meet your needs as far as changing out the certificate there as well. For me I don'd do this yet since I put all my emby data through HAproxy so I don't have any code for you at this time.
 
Though if emby is on the same machine the following should work if added onto the end of the above:
 

# cp hostcert.pfx /var/lib/emby-server/ssl/cert_9c31b7884ea5475c8687970fc5996297.pfx
# service emby-server restart

The actual file name of the certificate file will likely be different than it is specified here.

 

 

 

EDIT: The place where you copy the file should be whatever is specified under the hosting tab in emby.

Edited by Cerothen
Link to comment
Share on other sites

NomadCF

You could also just have haproxy deal with the encryption (losing the true end to end encryption). But in exchange you gain  the ability for a zero down time update of the ssl certs. 

 

(trimmed example)

global
        log /dev/log    local0
        log /dev/log    local1 notice
        chroot /var/lib/haproxy
        stats socket /run/haproxy/admin.sock mode 660 level admin
        stats timeout 30s
        user haproxy
        group haproxy
        daemon
        # Default SSL material locations
        ca-base /etc/ssl/certs
        crt-base /etc/ssl/private
        # Default ciphers to use on SSL-enabled listening sockets.
        # For more information, see ciphers(1SSL). This list is from:
        #  https://hynek.me/articles/hardening-your-web-servers-ssl-ciphers/
        ssl-default-bind-ciphers ECDH+AESGCM:DH+AESGCM:ECDH+AES256:DH+AES256:ECDH+AES128:DH+AES:ECDH+3DES:DH+3DES:RSA+AESGCM:RSA+AES:RSA+3DES:!aNULL:!MD5:!DSS
        ssl-default-bind-options no-sslv3

defaults
        log     global
        mode tcp
        option tcplog
        #mode   http
        #option httplog
        option  dontlognull
        timeout connect 5000
        timeout client  1000
        timeout server  5000
        errorfile 400 /etc/haproxy/errors/400.http
        errorfile 403 /etc/haproxy/errors/403.http
        errorfile 408 /etc/haproxy/errors/408.http
        errorfile 500 /etc/haproxy/errors/500.http
        errorfile 502 /etc/haproxy/errors/502.http
        errorfile 503 /etc/haproxy/errors/503.http
        errorfile 504 /etc/haproxy/errors/504.http

listen stats
    bind 0.0.0.0:9000       #Listen on all IP's on port 9000
    mode http
    balance
    timeout client 5000
    timeout connect 4000
    timeout server 30000
    #This is the virtual URL to access the stats page
    stats uri /haproxy_stats
    #Authentication realm. This can be set to anything. Escape space characters with a backslash.
    stats realm HAProxy\ Statistics
    #The user/pass you want to use. Change this password!
    stats auth admin:password2bchanged
    #This allows you to take down and bring up back end servers.
    #This will produce an error on older versions of HAProxy.
    stats admin if TRUE

#Application Setup
frontend ContentSwitching
  bind 10.1.97.2:8096
  bind 10.1.97.2:8920 ssl crt /etc/haproxy/ssl/
  bind 10.1.97.2:80
  bind 10.1.97.2:443 ssl crt /etc/haproxy/ssl/
  mode http
  option httplog
  option forwardfor
  timeout client  5000
  option http-server-close
  acl letsencrypt_remote_host path_beg /.well-known/acme-challenge/
  acl letsencrypt_remote_host req.ssl_sni -m end .acme.invalid
  acl mediacenter_host_1 hdr(host) -i tv.example.com
  acl mediacenter_host_1 hdr(host) -i streaming.example.com
  acl mediacenter_host_1 hdr(host) -i mediacenter.example.com
  use_backend letsencrypt_nodes if letsencrypt_remote_host
  use_backend streaming_nodes if mediacenter_host_1
  default_backend linux_www_node

backend letsencrypt_nodes
  mode http
  server letsencrypt 127.0.0.1:8080

backend streaming_nodes
  mode http
  compression algo gzip
  compression type text/html text/plain text/css text/js
  stick-table type ip size 200k expire 30m
  stick on src
  server node1 10.1.97.11:8096 check
  server node1 10.1.97.12:8096 check
  server node1 10.1.97.8:8096 check backup
  server node1 10.1.97.7:8096 check backup
  server node1 10.1.97.6:8096 check backup


backend linux_www_node
  mode http
  compression algo gzip
  compression type text/html text/plain text/css text/js
  balance roundrobin
  stick-table type ip size 200k expire 30m
  stick on src
  server node1 10.1.97.16:80 check
 server node2 10.1.97.17:80 check
 server node3 10.1.97.18:80 check backup
 

letsencrypt auto update cron

/usr/bin/letsencrypt  --agree-tos --renew-by-default --standalone --standalone-supported-challenges http-01 --http-01-port 8080 certonly -d tv.example.com -d home.example.com -d mediacenter.example.com >> /var/log/letsencrypt.log; cat /etc/letsencrypt/live/tv.xample.com/fullchain.pem /etc/letsencrypt/live/tv.example.com/privkey.pem > /etc/haproxy/ssl/home.example.com.pem; service haproxy reload
  • Like 2
Link to comment
Share on other sites

Doonga

Thanks for this. I was able to get this up and running with Cloudflare, haproxy and nginx with a Letsencrypt certificate. A few notes if anyone in tries this in the future and runs into issues...
 
I had to turn on HSTS in Cloudflare and set SSL to Full (Strict). I was getting 522s before this.
 
Also here's some great info on using Letsencrypt with nginx and having it be fully automated with zero downtime to renew the certs: http://blog.thesparktree.com/post/138999997429/generating-intranet-and-private-network-ssl My ISP blocks incoming port 80, so essentially I have a private site as far as Letsencrypt is concerned. Using the info on that page I was able to get it up and running. I was unable to use the environment variables for Lexicon for some reason, so I edited the hook script and put in my username and API token manually.
 
Thanks to @gstuartj for this post. I wouldn't have thought to secure my setup this well otherwise! Plus it gave me a project to do and nerd points. :)

Edited by Doonga
  • Like 1
Link to comment
Share on other sites

dcrdev

Quick question, if you're using cloudflare what's the point in using lets encrypt over a self signed cert? Cloudflare does man in the middle ssl anyway, so surely you'd be better off with a long life self signed cert.

 

Atleast that's the set up I have... I don't use cloudflare for my Emby subdomain though as CF doesn't currently do websockets.

  • Like 2
Link to comment
Share on other sites

gstuartj

Quick question, if you're using cloudflare what's the point in using lets encrypt over a self signed cert? Cloudflare does man in the middle ssl anyway, so surely you'd be better off with a long life self signed cert.

 

Atleast that's the set up I have... I don't use cloudflare for my Emby subdomain though as CF doesn't currently do websockets.

On my LAN clients hit my server directly without going through CloudFlare and Nginx presents my local cert. You're right that a self-signed cert would work easily and just as well, but I already had this machine set up to renew other Let's Encrypt certs, so adding another was as simple as creating a cron job. Renewal is automated, and it's nice that the cert verifies for local clients.

 

Your point about websockets is a good catch and definitely something to be aware of. Emby works without websockets, but it will be better once CloudFlare gets around to supporting them.

Link to comment
Share on other sites

gstuartj

Just as a fun aside, I later realized there's at least one small security issue in my setup. In the cf-realip-haproxy.conf snippet I check for the presence of the CF-Connecting-IP HTTP header, and if it exists I set the real client IP to that value. Technically it would be possible to manipulate the headers, put any value you want there (including text) and it will poison the Emby logs. I haven't thought of a good way to prevent this in my server configuration, though it could be detected by comparing the HAProxy and Emby logs for non-matching IP values. If anyone has any ideas, let me know!

 

I don't think this is a serious issue in this context, I just thought it was an interesting vulnerability.

Edited by gstuartj
Link to comment
Share on other sites

  • 9 months later...
chiefnerd

Just spotted this thread- lovely bit of Nginx tomfoolery there but I'm wondering why you've got proxy buffers off  - for streaming? Proxy buffers are for requests not so much file serving, other things like sendfile & tcp_push will make a difference on that. The proxy buffers are for requests to the proxy thus the smallish size but you also haven't set the backup buffer in case upstream devices are more capable thatn yours (it's only for headers so setting at 1k is fine with buffers off), I don't mind posting my Niginx.conf and specifics domain conf if it helps anyone but I don't use HAproxy or anything, just NGinx and Cloudflare & Doonga if you use any cert outside of cloudflare's own one's you enable that TBH, it's kind of a hit and miss thing in some instances but if it's an external cert to cloudflare use strict.

 

TBH, you;re far better off using cloudflare's certs unless there's a reason you're bang against them- long expiry, very simple to setup

Link to comment
Share on other sites

gstuartj

Just spotted this thread- lovely bit of Nginx tomfoolery there but I'm wondering why you've got proxy buffers off  - for streaming? Proxy buffers are for requests not so much file serving, other things like sendfile & tcp_push will make a difference on that. The proxy buffers are for requests to the proxy thus the smallish size but you also haven't set the backup buffer in case upstream devices are more capable thatn yours (it's only for headers so setting at 1k is fine with buffers off), I don't mind posting my Niginx.conf and specifics domain conf if it helps anyone but I don't use HAproxy or anything, just NGinx and Cloudflare & Doonga if you use any cert outside of cloudflare's own one's you enable that TBH, it's kind of a hit and miss thing in some instances but if it's an external cert to cloudflare use strict.

 

TBH, you;re far better off using cloudflare's certs unless there's a reason you're bang against them- long expiry, very simple to setup

 

Good point about proxy_buffers. I use Nginx, but not extensively. I had put that option into my config in a very early revision without reading enough about it and it just remained throughout my revisions. I've removed it from my current configs.

 

The reason I'm also using Let's Encrypt is because my www-router VM also handles SSL/TLS termination for subdomains I don't want to run through Cloudflare. Since I have functioning Let's Encrypt automation on the host, adding a cert for one more subdomain is as simple as adding a single line to a config file, making it pretty low effort for my use case. I also like being able to route local LAN clients directly to my Emby reverse proxy instead of to Cloudflare, and this way my LAN clients still get a cert that validates. Inside the LAN they get my Let's Encrypt cert, outside the LAN they get Cloudflare's. I do use Cloudflare's strict setting.

 

I agree that for most people just using Cloudflare's cert is easiest and most practical, but I'm a tinkerer and want my www-router to be just so. ;)

Link to comment
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now
×
×
  • Create New...