Emby via reverse proxy and internal network settings

[pir8radio 1292]

OK - more weirdness. I had posted another thread in the Android TV section about my client not playing a bunch of media.

 

However, if I connect directly to the Emby server, bypassing the Reverse Proxy, everything works perfectly.

 

There must be an issue with my HAProxy configuration. Anyone have any thoughts, pulling my hair out here

 

Sorry I'm of no use when it comes to HAProxy,  If you are not against trying out nginx I can be of help        I'm not even sure who all uses haproxy on here to recommend someone that can help you.

[Tur0k 143]

OK - more weirdness. I had posted another thread in the Android TV section about my client not playing a bunch of media.

 

However, if I connect directly to the Emby server, bypassing the Reverse Proxy, everything works perfectly.

 

There must be an issue with my HAProxy configuration. Anyone have any thoughts, pulling my hair out here

I had this problem when I first installed HAproxy on my PFSENSE firewall.

 

I have a feeling it was tied to passing the client IP to the backend server. You can do this globally or by backend server. I think I:

1. Disabled the global pass through.

2. Created separate backend servers for either

A. Internal access with client ip pass through disabled.

B. External backend server with client ip pass through enabled.

 

To the best of my recollection the problem was tied to the server trying to communicate directly with the client on the LAN and was bypassing the reverse proxy.

 

Your other option would be to put Emby server on a separate subnet with the reverse proxy as the only gateway between Emby and the clients. Then you could pass through all IPs.

 

What are you deploying HAProxy on?

 

 

Sent from my iPhone using Tapatalk

Edited October 4, 2017 by Tur0k

[FunkadelicRelic 10]

I had this problem when I first installed HAproxy on my PFSENSE firewall.

 

I have a feeling it was tied to passing the client IP to the backend server. You can do this globally or by backend server. I think I:

1. Disabled the global pass through.

2. Created separate backend servers for either

A. Internal access with client ip pass through disabled.

B. External backend server with client ip pass through enabled.

 

To the best of my recollection the problem was tied to the server trying to communicate directly with the client on the LAN and was bypassing the reverse proxy.

 

Your other option would be to put Emby server on a separate subnet with the reverse proxy as the only gateway between Emby and the clients. Then you could pass through all IPs.

 

What are you deploying HAProxy on?

 

 

Sent from my iPhone using Tapatalk

@@Tur0k - thanks, this sounds promising.

 

I too am using HAProxy on pfSense. I do actually already have Emby running in a separate subnet, as are all my internet accessable services.

 

I will try disabling the forwardfor option this evening when I get home. I had originally enabled it to get internet clients to correctly determine they were coming from the outside world and prompt for a password, but I turned it on for both internal and external listeners. I've temporarily disabled the external listener so I don't have to worry about exposing my server while I'm trying to sort this.

 

Do you have a copy of your HAProxy config that you wouldn't mind sharing?

[Tur0k 143]

@@Tur0k - thanks, this sounds promising.

 

I too am using HAProxy on pfSense. I do actually already have Emby running in a separate subnet, as are all my internet accessable services.

 

I will try disabling the forwardfor option this evening when I get home. I had originally enabled it to get internet clients to correctly determine they were coming from the outside world and prompt for a password, but I turned it on for both internal and external listeners. I've temporarily disabled the external listener so I don't have to worry about exposing my server while I'm trying to sort this.

 

Do you have a copy of your HAProxy config that you wouldn't mind sharing?

My network is a bit complex, I use VLANS and firewall rules to split up my network by type. I will pull a config and scrub it.

 

 

Sent from my iPhone using Tapatalk

[FunkadelicRelic 10]

No dice @@Tur0k - tried removing the forwardfor option but the problem still persists.

 

However - I did notice something strange in my logs as I was researching that. In almost all instances in my server.log, the hostname is referenced as the internal 'backend' server name and port. Except one. When I try to play a file that fails, it actually uses a different port. See below:

2017-10-04 19:22:20.613 Info App: Profile: Android-Exo, Path: /media/TV/Teenage Mutant Ninja Turtles (2012)/Season 01/Teenage Mutant Ninja Turtles (2012) - 1x05 - I Think His Name Is Baxter Stockman.mkv, isEligibleForDirectPlay: True, isEligibleForDirectStream: True
2017-10-04 19:22:20.613 Info App: Profile: Android-Exo, No direct play profiles found for Path: /media/TV/Teenage Mutant Ninja Turtles (2012)/Season 01/Teenage Mutant Ninja Turtles (2012) - 1x05 - I Think His Name Is Baxter Stockman.mkv
2017-10-04 19:22:20.621 Info HttpServer: HTTP Response 200 to 10.8.50.254. Time: 20ms. http://emby.domain.com:8096/emby/Items/82954e26dffb00cc94688871799a759c/PlaybackInfo?format=json 
2017-10-04 19:22:21.069 Info HttpServer: HTTP POST http://emby.domain.com:8096/emby/Sessions/Playing. UserAgent: Dalvik/2.1.0 (Linux; U; Android 6.0.1; BRAVIA 4K 2015 Build/MMB29V.S35)
2017-10-04 19:22:21.123 Info HttpServer: HTTP Response 204 to 10.8.50.254. Time: 54ms. http://emby.domain.com:8096/emby/Sessions/Playing 
2017-10-04 19:22:21.501 Info HttpServer: HTTP GET http://emby.domain.com:8096/emby/Videos/82954e26dffb00cc94688871799a759c/stream.mkv?DeviceId=429c5e4a30396037&Static=true&Tag=d15a1f01898e26a7f36f1cfc587a9413&MediaSourceId=82954e26dffb00cc94688871799a759c. Host=emby.domain.com:443, Accept=*/*, Accept-Language=en_US, User-Agent=VLC/3.0.0-git LibVLC/3.0.0-git, Range=bytes=0

Check out line 6 - the Host= section. It references my server on port 443. This happens everytime I play a file that fails. I haven't specified port 443 anywhere in the configuration of Emby, the only place it is referenced is the Frontend configuration of HAProxy. Could Emby be misinterpreting the hostname based on that? This certainly could cause some sort of loop...

[Guest plexman]

At least to it right!

 

Public key pinning, brave guy!

[FunkadelicRelic 10]

Public key pinning, brave guy!

 

Do you mean me? My server isn't open to the outside yet (plus SSL Labs won't detect my server yet as I have GeoIP blocking in place and haven't excluded it temporarily yet). Once I get this working I will be opening up external access and I'm sure with the correct ciphers I'll be able to achieve that rating - my others are at A+ - but one thing at a time - need to get this sorted internally first

[gstuartj 39]

Just wanted to chime in that I'm also seeing a strange problem with the Android mobile app when connecting to Emby through HAProxy. If I connect directly all is fine, but when I connect via HAProxy and attempt to play media the client gets stuck in an endless loop of attempting to play it and failing. (Note that the FireTV app, which should be similar to the Android TV app, does work for me.)

 

I haven't taken time to investigate much because none of my users really use the mobile clients and other clients seem to work. I used an Nginx reverse proxy a few months ago and all was okay, so unless something has changed with Emby I'm kind of at a loss for why the issue only crops up with HAProxy. I am interested in tracking down a fix, though.

[Luke 37060]

Check and see if you have a sub-directory and that is causing some of the urls to wind up with /emby/emby/xxx

 

the double /emby would be a problem.

[gstuartj 39]

Check and see if you have a sub-directory and that is causing some of the urls to wind up with /emby/emby/xxx

 

the double /emby would be a problem.

I'm proxying a subdomain directly to the Emby backend and not adding anything to the path.

[FunkadelicRelic 10]

Same here. No sub directory for me. The issue sounds the same though. Android TV and endless loop.

[Luke 37060]

i would suggest double checking and make sure you're not seeing that in your emby server logs.

[FunkadelicRelic 10]

Definitely not seeing that in my logs. I've attached them in this thread. I'll triple check just in case I missed something though.

[gstuartj 39]

I just grep'd for "emby/emby" in [logdir]/* and turned up nothing. I attempted to play an item in the Android app first to ensure there would be recent log data.

Edited October 5, 2017 by gstuartj

[FunkadelicRelic 10]

@@gstuartj - in your servers log, do you see the same issue I describe in post 31? The different Host= URL?

[gstuartj 39]

@@gstuartj - in your servers log, do you see the same issue I describe in post 31? The different Host= URL?

 

I believe that's correct and expected behavior. It's logging the headers in the HTTP request, and one of those is the Host header, which is preserved by the reverse proxy. So the URL at the beginning is the actual proxied GET request location that Emby responds to, and the Host header indicates the name and port of the server where the request was sent by the client.

Edited October 5, 2017 by gstuartj

[FunkadelicRelic 10]

Ah OK. I was hoping that was the issue but I get not.

 

I've been looking at this for days and unfortunately no closer to a solution. If you happen to find a fix @@gstuartj please could you let me know, it would be greatly appreciated.

[gstuartj 39]

I pulled and attached some relevant server logs; one is a sample of broken playback when connecting through the proxy, the other is working playback from a direct connection. Both are the same client.

What's interesting is that in the broken playback log sample the server never receives a GET request for the actual stream data from libVLC.

Something else I just noticed is that the setting under "Advanced" which I think used to be something like "Report HTTPS as external address" is now labeled "Require https for external connections" and can't be enabled without defining local certificates in Emby. (Which isn't needed when offloading TLS to a reverse proxy.) I used to have the old setting enabled with Emby behind a proxy.

@@Luke , is it possible that the wrong protocol or port number is sometimes being passed to libVLC for proxied connections, causing a request failure? Or do you have any other ideas? Edit: This definitely isn't the problem and libVLC is inheriting the connection scheme correctly. Details to come.

Edited October 5, 2017 by gstuartj

[pir8radio 1292]

 

Something else I just noticed is that the setting under "Advanced" which I think used to be something like "Report HTTPS as external address" is now labeled "Require https for external connections" and can't be enabled without defining local certificates in Emby. (Which isn't needed when offloading TLS to a reverse proxy.) I used to have the old setting enabled with Emby behind a proxy.

 

 

That shouldn't be needed if the reverse proxy is doing https.   Remember the link between your RP and emby is (usually) just HTTP and the RP takes care of making sure to change http to https if its setup correctly.   Think of the reverse proxy as your only web server, all the reverse proxy does is acts like a user to emby and relays that info to an internet user as the web server.   I do not have this box checked and I force HTTPS.  I have no issues with clients.  My home clients are android based, with no issues.  I would dig deeper into your configs, or consider just testing nginx (sorry I'm a nginx fan).

 

[FunkadelicRelic 10]

Thanks @@pir8radio - Indeed, in my case the proxy is terminating the SSL and sending requests on plain HTTP on port 8096. I haven't set any advanced options in Emby because as you say, the Reverse Proxy should be handling it.

 

Still cannot get this working though. I have a very basic config for Emby in the reverse proxy, and the fact everything else works - only the Android TV client fails - makes me think this isn't necessarily the RP config. Happy to be proved wrong on that though - anything to get this working

[gstuartj 39]

That shouldn't be needed if the reverse proxy is doing https.   Remember the link between your RP and emby is (usually) just HTTP and the RP takes care of making sure to change http to https if its setup correctly.

Yeah, forcing HTTPS on the backend is undesirable and the setting as it is now is unnecessary. The way it was worded previously makes it sound like it just did canonical URL reporting to the client, which could actually be useful in some edge cases.

 

That shouldn't be needed if the reverse proxy is doing https.   Remember the link between your RP and emby is (usually) just HTTP and the RP takes care of making sure to change http to https if its setup correctly.   Think of the reverse proxy as your only web server, all the reverse proxy does is acts like a user to emby and relays that info to an internet user as the web server.   I do not have this box checked and I force HTTPS.  I have no issues with clients.  My home clients are android based, with no issues.  I would dig deeper into your configs, or consider just testing nginx (sorry I'm a nginx fan).

Nginx is great and is an important part of my stack, and they have some feature overlap, but HAProxy is also a mature and powerful tool that serves some different purposes. So if there's a weird interaction between it and specific Emby clients I'd really like to track down the root cause and know why. I'd also have to re-architect things a bit and run Emby on a different port, as Nginx's support for proxying raw TCP streams is fairly new and not as powerful as HAProxy and I'm doing some tricky things with TCP proxying.

Edited October 5, 2017 by gstuartj

[gstuartj 39]

Alright, @@FunkadelicRelic, now that I've had some sleep I found my problem and I'm sorry to say it's not the same as yours. As the saying goes, never attribute to a bug what you can attribute to configuration. (This is where @@pir8radio gets to gloat. )

 

I thought it was weird that the GET request for stream.mkv was never hitting Emby, so I enabled some very detailed logging in HAProxy and started seeing that request turn up, it just wasn't getting to Emby. What I found is that libVLC, unlike the rest of the client, is sending the port number in the Host header, and I was matching the whole header against just the hostname in my ACL, so the ACL was returning false only when the request came from the video player (libVLC). This is unexpected, but technically not incorrect behavior. I changed the ACL to match the beginning of the header and streams started working for me again.

 

It looks like in your config you're already matching the beginning of the Host header and not the whole thing, and it also looks like the request for stream.mkv is getting through in that log sample you posted, so I'm unsure of what your problem is. Post a bigger log sample from Emby with "Debug logging" turned on, and maybe also an HAProxy log with info elevated to err. Maybe we can spot something. Also, has your configuration changed at all since post #25?

 

At least we know it's almost certainly not some obscure low-level bug, which would have been unlikely but really cool to find.

Edited October 5, 2017 by gstuartj

[pir8radio 1292]

Thanks @@pir8radio - Indeed, in my case the proxy is terminating the SSL and sending requests on plain HTTP on port 8096. I haven't set any advanced options in Emby because as you say, the Reverse Proxy should be handling it.

 

Still cannot get this working though. I have a very basic config for Emby in the reverse proxy, and the fact everything else works - only the Android TV client fails - makes me think this isn't necessarily the RP config. Happy to be proved wrong on that though - anything to get this working

 

Well, I know it seems a little sketchy, but if you want to PM me your server, maybe create a temporary account with access to some not important media, I can see if I see anything.  

[FunkadelicRelic 10]

Thanks @@gstuartj - I'll attempt to capture a debug log this afternoon and post the results.

 

@@pir8radio - Thanks for the offer - I might take you up on that if all other options are exhausted. As mentioned above, I've completely disabled outside access at the moment while I troubleshoot this, but could re-enable if it turns out we are still having issues.

 

To everyone else - I just wanted to say thanks so far for all your help. I'm new here and really want to make the switch from Plex and the way this community has responded so far makes me all that much more sure that I have made the right decision - even though I still can't get it working through my reverse proxy

[FunkadelicRelic 10]

Some more testing reveals:

 

On a file that works, when it starts playing, I get the HTTP GET in the log with the Host= field set to emby.domain.com. The file plays and works perfectly.

 

When I play a file that fails, I get the HTTP GET in the log with the Host= field set to emby.domain.com:443.

 

Any idea where this field is getting its host from? It seems odd that it is different. Unsure as to why the port is being appended.