gstuartj 39 Posted February 5, 2016 Share Posted February 5, 2016 (edited) I've seen a few reverse-proxy config files posted and thought I'd share my own setup. Like many, I use Nginx to add SSL, etc to Emby, but I have HAProxy sitting in front of it doing hostname routing. Why? I can host both HTTPS and OpenVPN services on the same port (443/TCP) via TCP proxying. SSH would also work. Hostname routing is done at TCP level via SNI, no decryption required. Maintains end-to-end encryption to each downstream server. Powerful configurability for later needs I also have my Emby hostname publicly routed through CloudFlare, but on my LAN the hostname goes directly to my local address, bypassing CloudFlare. (Better direct-play performance) I did some juggling to get the correct client IP in both scenarios and pass it to Emby.In addition, traffic is only allowed to hit the virtual host from the LAN or from CloudFlare (with a client certificate). I did my best to security harden the config.Please offer any suggestions or ask me if you have any questions on how things work.haproxy.cfg global log /dev/log local0 log /dev/log local1 notice chroot /var/lib/haproxy user haproxy group haproxy daemon defaults log global option dontlognull # Safety first option http-server-close timeout http-request 5s timeout connect 5000 timeout client 30000 timeout server 30000 frontend tls_router bind *:443 mode tcp option tcpka # Define known networks for later use acl local src 127.0.0.0/8 192.168.0.0/24 acl cloudflare src 199.27.128.0/21 173.245.48.0/20 103.21.244.0/22 103.22.200.0/22 103.31.4.0/22 141.101.64.0/18 108.162.192.0/18 190.93.240.0/20 188.114.96.0/20 197.234.240.0/22 198.41.128.0/17 162.158.0.0/15 104.16.0.0/12 172.64.0.0/13 2400:cb00::/32 2606:4700::/32 2803:f800::/32 2405:b500::/32 2405:8100::/32 #Rate-limiting measures, in case of DoS # Table for connection tracking stick-table type ip size 100k expire 30s store conn_cur # Allow known CloudFlare IPs to bypass the rate-limiting tcp-request connection accept if cloudflare # Reject connection if client has more than 10 open tcp-request connection reject if { src_conn_cur ge 10 } tcp-request connection track-sc1 src # Max inspection delay for SNI routing tcp-request inspect-delay 2s # Accept only SSL/TLS traffic tcp-request content accept if { req_ssl_hello_type 1 } # If SNI hostname is known server AND is from known IP, pass to backend acl media_sni req_ssl_sni -i emby.hostname use_backend crane_tls if media_sni cloudflare or local # Other hostname, allow traffic from all acl other_sni req_ssl_sni -i other.hostname use_backend otherserver_tls if other_sni # Root hostname acl root_sni req_ssl_sni -i example.domain use_backend crane_tls if root_sni cloudflare # If SNI hostname isn't recognized, reject connection. Must come last. acl fallback_sni req_ssl_sni tcp-request connection reject if fallback_sni # If SNI hostname not provided, pass to OpenVPN backend to deal with default_backend openvpn backend crane_tls mode tcp option tcpka # Proxy protocol to preserve client info. Needs to be enabled in nginx server server_crane_tls localhost:1443 send-proxy backend otherserver_tls mode tcp option tcpka # Proxy protocol to preserve client info. Needs to be enabled in nginx server server_other_tls 192.168.0.23:1443 send-proxy backend openvpn mode tcp server server_ovpn 192.168.0.32:1195 nginx # emby reverse proxy # # Intended to sit downstream from HAProxy, uses proxy protocol. # Allows traffic from the local network, or from WAN w/ CloudFlare client cert # server { # Using proxy protocol to get client info passed from HAProxy listen localhost:1443 ssl http2 proxy_protocol; listen [::1]:1443 ssl http2 proxy_protocol; keepalive_timeout 180; # Who the hell am I?! Where's my stuff?! server_name emby.hostname; root /var/www/emby.hostname/public_html; # Import some solid TLS settings include snippets/solidtls.conf; # Certs ssl_certificate /etc/nginx/ssl/emby.hostname/fullchain.pem; ssl_certificate_key /etc/nginx/ssl/emby.hostname/key.pem; ssl_trusted_certificate /etc/nginx/ssl/emby.hostname/fullchain.pem; ssl_client_certificate /etc/nginx/ssl/cf-origin-pull-ca.pem; # Client cert optional. We'll do our own access control rules in / ssl_verify_client optional; # Let's get the real client info from upstream proxy include snippets/realip-cf-haproxy.conf; # Set headers with real client info for downstream app proxy_set_header Host $host; proxy_set_header X-Real-IP $real_ip; proxy_set_header X-Forwarded-For $real_ip; proxy_set_header X-Forwarded-Proto $real_ip; proxy_set_header X-Forwarded-Protocol $scheme; # Turn off buffering for streaming purposes proxy_buffering off; # No request rewriting proxy_redirect off; # Send websocket data to the backend, courtesy @[member="Karbowiak"] on Emby forum proxy_http_version 1.1; proxy_set_header Upgrade $http_upgrade; proxy_set_header Connection "upgrade"; # Test if IP from HAProxy is local set $is_local_client false; if ( $proxy_protocol_addr ~ 192.168.0.([100-254]) ) { set $is_local_client true; } # Force the use of our custom robots.txt location /robots.txt { try_files $uri /var/www/emby.hostname/public_html/robots.txt; } # Catch-all. Performs authentication based on client IP or certificate. location / { # We'll return 418 later if client is authorized error_page 418 = @authorized; #Access rules # Allow local clients without client cert if ( $is_local_client = true ) { return 418; } # Does client have valid client certificate? if ( $ssl_client_verify = "SUCCESS" ) { return 418; } # Deny all others, return 403 unauthorized return 403; } # Handles 418 responses for authorized users. Does the actual proxying. location @authorized { proxy_pass http://localhost:8096; } # Needed for Let's Encrypt certificate renewals include /etc/nginx/snippets/letsencrypt.conf; } snippets/solidtls.conf # Trustworthy SSL/TLS settings in a box # ssl on; gzip off; # gzip allows for some attacks # General params ssl_session_timeout 1d; ssl_session_cache shared:SSL:10m; ssl_session_tickets off; # Always use custom-generated DH params >2048 bits ssl_dhparam /etc/nginx/ssl/dhparam.pem; #Intermediate cipher config ssl_protocols TLSv1 TLSv1.1 TLSv1.2; ssl_ciphers 'ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-AES256-GCM-SHA384:DHE-RSA-AES128-GCM-SHA256:DHE-DSS-AES128-GCM-SHA256:kEDH+AESGCM:ECDHE-RSA-AES128-SHA256:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA:ECDHE-ECDSA-AES128-SHA:ECDHE-RSA-AES256-SHA384:ECDHE-ECDSA-AES256-SHA384:ECDHE-RSA-AES256-SHA:ECDHE-ECDSA-AES256-SHA:DHE-RSA-AES128-SHA256:DHE-RSA-AES128-SHA:DHE-DSS-AES128-SHA256:DHE-RSA-AES256-SHA256:DHE-DSS-AES256-SHA:DHE-RSA-AES256-SHA:ECDHE-RSA-DES-CBC3-SHA:ECDHE-ECDSA-DES-CBC3-SHA:AES128-GCM-SHA256:AES256-GCM-SHA384:AES128-SHA256:AES256-SHA256:AES128-SHA:AES256-SHA:AES:CAMELLIA:DES-CBC3-SHA:!aNULL:!eNULL:!EXPORT:!DES:!RC4:!MD5:!PSK:!aECDH:!EDH-DSS-DES-CBC3-SHA:!EDH-RSA-DES-CBC3-SHA:!KRB5-DES-CBC3-SHA'; #Modern cipher config #ssl_protocols TLSv1.1 TLSv1.2; #ssl_ciphers 'ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-AES256-GCM-SHA384:DHE-RSA-AES128-GCM-SHA256:DHE-DSS-AES128-GCM-SHA256:kEDH+AESGCM:ECDHE-RSA-AES128-SHA256:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA:ECDHE-ECDSA-AES128-SHA:ECDHE-RSA-AES256-SHA384:ECDHE-ECDSA-AES256-SHA384:ECDHE-RSA-AES256-SHA:ECDHE-ECDSA-AES256-SHA:DHE-RSA-AES128-SHA256:DHE-RSA-AES128-SHA:DHE-DSS-AES128-SHA256:DHE-RSA-AES256-SHA256:DHE-DSS-AES256-SHA:DHE-RSA-AES256-SHA:!aNULL:!eNULL:!EXPORT:!DES:!RC4:!3DES:!MD5:!PSK'; #Perfect ssllabs score cipher list. Not super usable, works for CF-only setups #ssl_protocols TLSv1.2; #ssl_ciphers AES256+EECDH:AES256+EDH:!aNULL; # No downgrade attacks ssl_prefer_server_ciphers on; # HSTS add_header Strict-Transport-Security "max-age=63072000; includeSubdomains; preload"; add_header X-Frame-Options DENY; add_header X-Content-Type-Options nosniff; # Stapling ssl_stapling on; ssl_stapling_verify on; # DNS resolvers to use for cert verification resolver 8.8.8.8 8.8.4.4 valid=300s; resolver_timeout 5s; snippets/realip-cf-haproxy.conf #Local traffic, including HAProxy on localhost set_real_ip_from 127.0.0.0/8; #CloudFlare set_real_ip_from 199.27.128.0/21; set_real_ip_from 173.245.48.0/20; set_real_ip_from 103.21.244.0/22; set_real_ip_from 103.22.200.0/22; set_real_ip_from 103.31.4.0/22; set_real_ip_from 141.101.64.0/18; set_real_ip_from 108.162.192.0/18; set_real_ip_from 190.93.240.0/20; set_real_ip_from 188.114.96.0/20; set_real_ip_from 197.234.240.0/22; set_real_ip_from 198.41.128.0/17; set_real_ip_from 162.158.0.0/15; set_real_ip_from 104.16.0.0/12; set_real_ip_from 172.64.0.0/13; set_real_ip_from 2400:cb00::/32; set_real_ip_from 2606:4700::/32; set_real_ip_from 2803:f800::/32; set_real_ip_from 2405:b500::/32; set_real_ip_from 2405:8100::/32; #Default state, assume local traffic/proxy protocol header set $forwarded_ip_header proxy_protocol; set $real_ip $proxy_protocol_addr; # Is Cloudflare header set? Use that header/addr instead of proxy_protocol if ($http_cf_connecting_ip) { set $forwarded_ip_header CF-Connecting-IP; set $real_ip $http_cf_connecting_ip; } # Tell Nginx which header to use real_ip_header $forwarded_ip_header; Edited February 5, 2016 by gstuartj 6 Link to comment Share on other sites More sharing options...
gstuartj 39 Posted February 5, 2016 Author Share Posted February 5, 2016 Edit screwed up the configs. Reposted. Link to comment Share on other sites More sharing options...
jaybroni 4 Posted March 19, 2016 Share Posted March 19, 2016 +1 Like from me. Nice work, very impressive. Too bad it's way over my head. If you were only 10% as techy as you are, how would you setup a basic SSL Certificate? I'd like to set one up, using a free SSL from one of those free CA sites online, so that my users don't get red flags when they connect using a 'self-signed cert'. How would I go about doing this? If you want to point me to a tutorial that would be great. Link to comment Share on other sites More sharing options...
Cerothen 89 Posted March 19, 2016 Share Posted March 19, 2016 If you have a Linux box kicking around letsencrypt is the way to go, I tried it out for the first time last week and it was a breeze, just follow the getting started page and after about 5 mins you can have a signed cert with every (sub)domain you could want. Link to comment Share on other sites More sharing options...
jaybroni 4 Posted April 14, 2016 Share Posted April 14, 2016 I do have a Linux box kicking around, so I'll look into just what you said. Thank you very much! Link to comment Share on other sites More sharing options...
Cerothen 89 Posted April 14, 2016 Share Posted April 14, 2016 (edited) Just remember that the outputs are not emby friendly out of the gate, I added the below into my renew script so that it takes the live certificates after the renew and creates a bundle for HAproxy and a pfx without a password for emby. # cd /etc/letsencrypt/live/<<domain>> # cat fullchain.pem privkey.pem > bundle.pem # openssl pkcs12 -export -out hostcert.pfx -inkey privkey.pem -in cert.pem -certfile chain.pem -passout pass: The cert's aren't valid for all that long (3-ish months) but if you have something that stops emby, overwrites the certificate that is stored in "<<emby root>>/ssl/cert_9c31b7884ea5475c8687970fc5996297.pfx" and then starts emby up again then that should meet your needs as far as changing out the certificate there as well. For me I don'd do this yet since I put all my emby data through HAproxy so I don't have any code for you at this time. Though if emby is on the same machine the following should work if added onto the end of the above: # cp hostcert.pfx /var/lib/emby-server/ssl/cert_9c31b7884ea5475c8687970fc5996297.pfx # service emby-server restart The actual file name of the certificate file will likely be different than it is specified here. EDIT: The place where you copy the file should be whatever is specified under the hosting tab in emby. Edited April 14, 2016 by Cerothen Link to comment Share on other sites More sharing options...
NomadCF 15 Posted April 14, 2016 Share Posted April 14, 2016 You could also just have haproxy deal with the encryption (losing the true end to end encryption). But in exchange you gain the ability for a zero down time update of the ssl certs. (trimmed example) global log /dev/log local0 log /dev/log local1 notice chroot /var/lib/haproxy stats socket /run/haproxy/admin.sock mode 660 level admin stats timeout 30s user haproxy group haproxy daemon # Default SSL material locations ca-base /etc/ssl/certs crt-base /etc/ssl/private # Default ciphers to use on SSL-enabled listening sockets. # For more information, see ciphers(1SSL). This list is from: # https://hynek.me/articles/hardening-your-web-servers-ssl-ciphers/ ssl-default-bind-ciphers ECDH+AESGCM:DH+AESGCM:ECDH+AES256:DH+AES256:ECDH+AES128:DH+AES:ECDH+3DES:DH+3DES:RSA+AESGCM:RSA+AES:RSA+3DES:!aNULL:!MD5:!DSS ssl-default-bind-options no-sslv3 defaults log global mode tcp option tcplog #mode http #option httplog option dontlognull timeout connect 5000 timeout client 1000 timeout server 5000 errorfile 400 /etc/haproxy/errors/400.http errorfile 403 /etc/haproxy/errors/403.http errorfile 408 /etc/haproxy/errors/408.http errorfile 500 /etc/haproxy/errors/500.http errorfile 502 /etc/haproxy/errors/502.http errorfile 503 /etc/haproxy/errors/503.http errorfile 504 /etc/haproxy/errors/504.http listen stats bind 0.0.0.0:9000 #Listen on all IP's on port 9000 mode http balance timeout client 5000 timeout connect 4000 timeout server 30000 #This is the virtual URL to access the stats page stats uri /haproxy_stats #Authentication realm. This can be set to anything. Escape space characters with a backslash. stats realm HAProxy\ Statistics #The user/pass you want to use. Change this password! stats auth admin:password2bchanged #This allows you to take down and bring up back end servers. #This will produce an error on older versions of HAProxy. stats admin if TRUE #Application Setup frontend ContentSwitching bind 10.1.97.2:8096 bind 10.1.97.2:8920 ssl crt /etc/haproxy/ssl/ bind 10.1.97.2:80 bind 10.1.97.2:443 ssl crt /etc/haproxy/ssl/ mode http option httplog option forwardfor timeout client 5000 option http-server-close acl letsencrypt_remote_host path_beg /.well-known/acme-challenge/ acl letsencrypt_remote_host req.ssl_sni -m end .acme.invalid acl mediacenter_host_1 hdr(host) -i tv.example.com acl mediacenter_host_1 hdr(host) -i streaming.example.com acl mediacenter_host_1 hdr(host) -i mediacenter.example.com use_backend letsencrypt_nodes if letsencrypt_remote_host use_backend streaming_nodes if mediacenter_host_1 default_backend linux_www_node backend letsencrypt_nodes mode http server letsencrypt 127.0.0.1:8080 backend streaming_nodes mode http compression algo gzip compression type text/html text/plain text/css text/js stick-table type ip size 200k expire 30m stick on src server node1 10.1.97.11:8096 check server node1 10.1.97.12:8096 check server node1 10.1.97.8:8096 check backup server node1 10.1.97.7:8096 check backup server node1 10.1.97.6:8096 check backup backend linux_www_node mode http compression algo gzip compression type text/html text/plain text/css text/js balance roundrobin stick-table type ip size 200k expire 30m stick on src server node1 10.1.97.16:80 check server node2 10.1.97.17:80 check server node3 10.1.97.18:80 check backup letsencrypt auto update cron /usr/bin/letsencrypt --agree-tos --renew-by-default --standalone --standalone-supported-challenges http-01 --http-01-port 8080 certonly -d tv.example.com -d home.example.com -d mediacenter.example.com >> /var/log/letsencrypt.log; cat /etc/letsencrypt/live/tv.xample.com/fullchain.pem /etc/letsencrypt/live/tv.example.com/privkey.pem > /etc/haproxy/ssl/home.example.com.pem; service haproxy reload 2 Link to comment Share on other sites More sharing options...
Doonga 17 Posted April 17, 2016 Share Posted April 17, 2016 (edited) Thanks for this. I was able to get this up and running with Cloudflare, haproxy and nginx with a Letsencrypt certificate. A few notes if anyone in tries this in the future and runs into issues... I had to turn on HSTS in Cloudflare and set SSL to Full (Strict). I was getting 522s before this. Also here's some great info on using Letsencrypt with nginx and having it be fully automated with zero downtime to renew the certs: http://blog.thesparktree.com/post/138999997429/generating-intranet-and-private-network-ssl My ISP blocks incoming port 80, so essentially I have a private site as far as Letsencrypt is concerned. Using the info on that page I was able to get it up and running. I was unable to use the environment variables for Lexicon for some reason, so I edited the hook script and put in my username and API token manually. Thanks to @gstuartj for this post. I wouldn't have thought to secure my setup this well otherwise! Plus it gave me a project to do and nerd points. Edited April 17, 2016 by Doonga 1 Link to comment Share on other sites More sharing options...
dcrdev 251 Posted April 17, 2016 Share Posted April 17, 2016 Quick question, if you're using cloudflare what's the point in using lets encrypt over a self signed cert? Cloudflare does man in the middle ssl anyway, so surely you'd be better off with a long life self signed cert. Atleast that's the set up I have... I don't use cloudflare for my Emby subdomain though as CF doesn't currently do websockets. 2 Link to comment Share on other sites More sharing options...
gstuartj 39 Posted April 17, 2016 Author Share Posted April 17, 2016 Quick question, if you're using cloudflare what's the point in using lets encrypt over a self signed cert? Cloudflare does man in the middle ssl anyway, so surely you'd be better off with a long life self signed cert. Atleast that's the set up I have... I don't use cloudflare for my Emby subdomain though as CF doesn't currently do websockets. On my LAN clients hit my server directly without going through CloudFlare and Nginx presents my local cert. You're right that a self-signed cert would work easily and just as well, but I already had this machine set up to renew other Let's Encrypt certs, so adding another was as simple as creating a cron job. Renewal is automated, and it's nice that the cert verifies for local clients. Your point about websockets is a good catch and definitely something to be aware of. Emby works without websockets, but it will be better once CloudFlare gets around to supporting them. Link to comment Share on other sites More sharing options...
gstuartj 39 Posted April 17, 2016 Author Share Posted April 17, 2016 @@Doonga, thanks, glad it was helpful! Link to comment Share on other sites More sharing options...
gstuartj 39 Posted April 17, 2016 Author Share Posted April 17, 2016 (edited) Just as a fun aside, I later realized there's at least one small security issue in my setup. In the cf-realip-haproxy.conf snippet I check for the presence of the CF-Connecting-IP HTTP header, and if it exists I set the real client IP to that value. Technically it would be possible to manipulate the headers, put any value you want there (including text) and it will poison the Emby logs. I haven't thought of a good way to prevent this in my server configuration, though it could be detected by comparing the HAProxy and Emby logs for non-matching IP values. If anyone has any ideas, let me know! I don't think this is a serious issue in this context, I just thought it was an interesting vulnerability. Edited April 17, 2016 by gstuartj Link to comment Share on other sites More sharing options...
chiefnerd 18 Posted January 24, 2017 Share Posted January 24, 2017 Just spotted this thread- lovely bit of Nginx tomfoolery there but I'm wondering why you've got proxy buffers off - for streaming? Proxy buffers are for requests not so much file serving, other things like sendfile & tcp_push will make a difference on that. The proxy buffers are for requests to the proxy thus the smallish size but you also haven't set the backup buffer in case upstream devices are more capable thatn yours (it's only for headers so setting at 1k is fine with buffers off), I don't mind posting my Niginx.conf and specifics domain conf if it helps anyone but I don't use HAproxy or anything, just NGinx and Cloudflare & Doonga if you use any cert outside of cloudflare's own one's you enable that TBH, it's kind of a hit and miss thing in some instances but if it's an external cert to cloudflare use strict. TBH, you;re far better off using cloudflare's certs unless there's a reason you're bang against them- long expiry, very simple to setup Link to comment Share on other sites More sharing options...
gstuartj 39 Posted January 25, 2017 Author Share Posted January 25, 2017 Just spotted this thread- lovely bit of Nginx tomfoolery there but I'm wondering why you've got proxy buffers off - for streaming? Proxy buffers are for requests not so much file serving, other things like sendfile & tcp_push will make a difference on that. The proxy buffers are for requests to the proxy thus the smallish size but you also haven't set the backup buffer in case upstream devices are more capable thatn yours (it's only for headers so setting at 1k is fine with buffers off), I don't mind posting my Niginx.conf and specifics domain conf if it helps anyone but I don't use HAproxy or anything, just NGinx and Cloudflare & Doonga if you use any cert outside of cloudflare's own one's you enable that TBH, it's kind of a hit and miss thing in some instances but if it's an external cert to cloudflare use strict. TBH, you;re far better off using cloudflare's certs unless there's a reason you're bang against them- long expiry, very simple to setup Good point about proxy_buffers. I use Nginx, but not extensively. I had put that option into my config in a very early revision without reading enough about it and it just remained throughout my revisions. I've removed it from my current configs. The reason I'm also using Let's Encrypt is because my www-router VM also handles SSL/TLS termination for subdomains I don't want to run through Cloudflare. Since I have functioning Let's Encrypt automation on the host, adding a cert for one more subdomain is as simple as adding a single line to a config file, making it pretty low effort for my use case. I also like being able to route local LAN clients directly to my Emby reverse proxy instead of to Cloudflare, and this way my LAN clients still get a cert that validates. Inside the LAN they get my Let's Encrypt cert, outside the LAN they get Cloudflare's. I do use Cloudflare's strict setting. I agree that for most people just using Cloudflare's cert is easiest and most practical, but I'm a tinkerer and want my www-router to be just so. Link to comment Share on other sites More sharing options...
Recommended Posts
Create an account or sign in to comment
You need to be a member in order to leave a comment
Create an account
Sign up for a new account in our community. It's easy!
Register a new accountSign in
Already have an account? Sign in here.
Sign In Now