Failed logins from 127.0.0.1?

[Q-Droid 989]

It's going through my WAN IP (NAT loopback) which is the masked 99.99.99.81 address. I'm running Chrome on a Windows desktop and the Emby beta server is a docker container on my Linux server. I've also tried the same test over ProtonVPN using both IP address and domain name, the results are the same.

With "x-forwarded-for" header spoofing the IP that is the only client IP shown. When I remove the header it goes back to showing a real client IP though in the loopback case that's the WAN IP.

 

 

[softworkz 5066]

22 minutes ago, Q-Droid said:

When I remove the header it goes back to showing a real client IP though in the loopback case that's the WAN IP.

Where does it show a real client IP? At the same place which I marked in yellow?

In your log there's also a request without the x-forwarded-for header and it's still showing 127.0.0.1:

 

[Q-Droid 989]

The logs are for tests with the header on, other tests are not in these logs.

Re: Why it doesn't show the header for all requests your guess is as good as mine. As far as I can tell from the browser dev tools all of the requests are including that header.

 

 

 

[Q-Droid 989]

One more thing that I just realized but hadn't tried yet - Doh! moment. I'm switching ISPs so I have both services active right now. No need for NAT loopback for VPNs - same results when I go from one ISP to the other.

 

[softworkz 5066]

Unfortunately, the output cannot be fully trusted, it's doing its own things. I've PMed you something for additional logging to shed some more light on this...

[zefritz 0]

7 hours ago, softworkz said:

@zefritz- The interesting question is: How did you make your Emby server accessible from outside? How did you publish it to the internet?

Your log is showing that the request is actually coming from 127.0.0.1 (at the network level), so the question is how does that come? It would not be like that when you would have published the server through a router (which is the typical case).

When you don't have a reverse proxy and are publishing your server through a router, then the X-FORWARDED-FOR header doesn't have any effect.

The only case where Emby is taken X-FORWARDED-FOR and X-REALIP headers into account is when the request at the network level is coming from the local machine.
In that case, Emby makes the assumption that you are running a reverse proxy (with proper configuration) and only then it is using those headers.

Maybe that (automatic internal) assumption is a bit too keen. The question is though: Why does the request come from 127.0.0.1 even though you don't have a reverse proxy in place?

 

On my home router I have port forwarding for 80->8096 and 443->8920.  I have a domain I own pointing to my IP, and use a lets encrypt certificate for the https connection. I don't use UPnP anywhere and have it turned off on my router, I checked the box for 'Allow remote connections to this Emby Server', and my 'Secure connection mode' is set to Preferred, but not required. My emby version is 4.7.11.0.  To my knowledge it shouldn't be possible for any requests to come from 127.0.0.1, which is what alarmed me when I saw all those alerts yesterday evening. After looking at the alerts/logs, and then checking the server for any unusual activity and not seeing anything, I made the post here to see if anyone could help explain what's happening.

[softworkz 5066]

Update

I can confirm that this issue exists and I share the view that this is a vulnerability in the current implementation of handling incoming HTTP requests - of moderate severity at least.

What's happening?

The irony is that this vulnerability has been introduced in order to support setups with reverse proxies for publishing Emby endpoints to the internet. Certain features and behaviors in Emby Server are dependent on the ability to detect whether requests are coming 

1. From the public internet (external)
2. From the local network
3. From the local machine

This distinction can be made easily in case of a regular setup where Emby services are published through a router. In that case, Emby sees the source IP address of the requests and can categorize based on that source IP address.

When running a reverse proxy, it is typically either on the local machine or on another host on the local network, and as Emby gets the requests from the reverse proxy in that case, it sees the requests either coming from localhost or from the local network, which means that it would categorize ALL requests coming through the reverse proxy as either #2 or #3.
To inform the target about the actual origin of a request, reverse proxies are adding either the X-FORWARDED-FOR or the X-REALIP header to the request when forwarding it to Emby server. And Emby Server in turn needs to rely on these headers being correct in order to determine where a requests comes from.

The Cause

When requests are not coming through a remote proxy, Emby shouldn't actually trust such headers, but when a reverse proxy is in place, it's mandatory that Emby uses and relies on those headers. And that's a bit of an unfortunate contradiction. Without any configuration switch being in place, Emby cannot know whether requests come from a reverse proxy or directly from the internet.
Currently, Emby always trusts those headers if present, which is the cause of the vulnerability.

Mitigations

For Setups without Reverse Proxy

• None
• A fix is currently in development (see below)

For Setups with Reverse Proxy

• Make sure to drop X-FORWARDED-FOR and X-REALIP headers from requests coming from the public internet
• Important
  • Do not accidentally suppress the proxy's generation of those headers when forwarding requests to Emby Server
  • The proxy needs to generate at least an X-FORWARDED-FOR or X-REALIP header for Emby to know the actual origin of a request
• These instructions apply, in either case - with or without the fix being installed
   

Emby Server Fix

A preliminary fix has been prepared and tested with help from @Q-Droidand is pending internal review.

Please PM me if you would be interested in testing the fix (which requires to have Server Beta 4.8.0.30 installed).

[pwhodges 2012]

1 hour ago, softworkz said:

For Setups with Reverse Proxy

• Make sure to drop X-FORWARDED-FOR and X-REALIP headers from requests coming from the public internet

Note that Caddy drops these headers by default; it will only preserve their values (and add its own to them) if you go to the trouble of adding statements specifying what upstream proxies are to be trusted.

Paul

Edited April 21, 2023 by pwhodges

[Spaceboy 2573]

yep, i use haproxy and you tell it what headers you want to pass through, nothing enabled by default

[pir8radio 1312]

1 hour ago, softworkz said:

Update

I can confirm that this issue exists and I share the view that this is a vulnerability in the current implementation of handling incoming HTTP requests - of moderate severity at least.

What's happening?

The irony is that this vulnerability has been introduced in order to support setups with reverse proxies for publishing Emby endpoints to the internet. Certain features and behaviors in Emby Server are dependent on the ability to detect whether requests are coming 

1. From the public internet (external)
2. From the local network
3. From the local machine

This distinction can be made easily in case of a regular setup where Emby services are published through a router. In that case, Emby sees the source IP address of the requests and can categorize based on that source IP address.

When running a reverse proxy, it is typically either on the local machine or on another host on the local network, and as Emby gets the requests from the reverse proxy in that case, it sees the requests either coming from localhost or from the local network, which means that it would categorize ALL requests coming through the reverse proxy as either #2 or #3.
To inform the target about the actual origin of a request, reverse proxies are adding either the X-FORWARDED-FOR or the X-REALIP header to the request when forwarding it to Emby server. And Emby Server in turn needs to rely on these headers being correct in order to determine where a requests comes from.

The Cause

When requests are not coming through a remote proxy, Emby shouldn't actually trust such headers, but when a reverse proxy is in place, it's mandatory that Emby uses and relies on those headers. And that's a bit of an unfortunate contradiction. Without any configuration switch being in place, Emby cannot know whether requests come from a reverse proxy or directly from the internet.
Currently, Emby always trusts those headers if present, which is the cause of the vulnerability.

Mitigations

For Setups without Reverse Proxy

• None
• A fix is currently in development (see below)

For Setups with Reverse Proxy

• Make sure to drop X-FORWARDED-FOR and X-REALIP headers from requests coming from the public internet
• Important
  • Do not accidentally suppress the proxy's generation of those headers when forwarding requests to Emby Server
  • The proxy needs to generate at least an X-FORWARDED-FOR or X-REALIP header for Emby to know the actual origin of a request
• These instructions apply, in either case - with or without the fix being installed
   

Emby Server Fix

A preliminary fix has been prepared and tested with help from @Q-Droidand is pending internal review.

Please PM me if you would be interested in testing the fix (which requires to have Server Beta 4.8.0.30 installed).

You can’t drop those headers the reverse proxy will then default to assigning those headers the local proxy ip.  Which will make users look local.  You need to pass them to preserve the client ip.    But ignore local ip’s in those headers.  

[softworkz 5066]

23 minutes ago, pir8radio said:

You can’t drop those headers the reverse proxy will then default to assigning those headers the local proxy ip.  Which will make users look local.  You need to pass them to preserve the client ip.    But ignore local ip’s in those headers.  

I don't think that's correct. When a client makes a request, that request never contains such headers. No client is creating such headers normally. It's the proxy which adds those headers before forwarding the request to Emby Server.

Only a malicious client would add such headers in order to make it appear to Emby as if the request has gone through a proxy (while that malicious client hopes that there's no proxy in-between). That's why a reverse proxy must drop such headers coming from a client and add its own headers which are pointing to the actual IP address from which it has received the request.

[softworkz 5066]

Only exception would be when you are chaining proxies, but in that case, you need to configure a secondary proxy in the way that it only accepts headers from known (trusted) proxies but not from direct client requests (= unknown sources).

Edited April 21, 2023 by softworkz

[rbjtech 5284]

14 hours ago, pir8radio said:

You can’t drop those headers the reverse proxy will then default to assigning those headers the local proxy ip.  Which will make users look local.  You need to pass them to preserve the client ip.    But ignore local ip’s in those headers.  

I've had a look at my nginx config (derived from your original post) and believe the following should be removed :-

proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;  

This is only to be used in the case of appending the forwarded IP - it does not remove the original IP - which I believe is the danger being highlighted here ?

By removing it - the remote IP is still passed to emby (via set_header X-Real-IP) from my limited testing from a vpn and 4g mobile.

@Q-Droid- Could you kindly add nginx rp to your testing please ?

I'm happy to ping your my public url over PM if that's ok but I'll also need your public IP as it's also geo locked, but can open it up for a short time if needs be - thanks ?

Edited April 22, 2023 by rbjtech

[Q-Droid 989]

2 hours ago, rbjtech said:

I've had a look at my nginx config (derived from your original post) and believe the following should be removed :-

proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;  

This is only to be used in the case of appending the forwarded IP - it does not remove the original IP - which I believe is the danger being highlighted here ?

By removing it - the remote IP is still passed to emby from my limited testing from a vpn and 4g mobile.

@Q-Droid- Could you kindly add nginx rp to your testing please ?

I'm happy to ping your my public url over PM if that's ok but I'll also need your public IP as it's also geo locked, but can open it up for a short time if needs be - thanks ?

DM'd. As mentioned before Caddy does not append to that header and replaces instead. I'm not sure what the purpose would be for maintaining a trail of IPs like that in a simple RP setup. Perhaps if there was a need to traverse and track multiple zones/subnets in the application but as far as I can tell Emby only cares about the difference between local and remote.

 

[rbjtech 5284]

To update - with nginx with XFF removed - @Q-Droidsaw normal behaviour forcing a local header (ie presented with a login page).

I've put it back to how it was (ie XFF append) and lets see if that changes things ..

edit

So even with XFF enabled - it still does not appear to pass the 127.0.0.1 to emby - presenting the login page as above.

So I'll be removing it again (as it passes the remote IP via X-Real-IP) - as it's not needed in my setup (I only have one RP) but it appears its not causing any issues by leaving it.

Thanks @Q-Droidfor the testing. 

Edited April 22, 2023 by rbjtech

[TMCsw 249]

I’m a little confused as this thread started as (I think?) about a vulnerability with using emby ‘Out Of the Box’?
But then evolved into reverse proxies or not ?

PLEASE Clarify…

I use nginx as a reverse proxy to my emby server…

https://securityheaders.com/
Gives me an A+

regardless of if my location {} contains either/or/not

  proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
  proxy_set_header X-Real-IP $remote_addr;

Also my nginx logs don’t seem to be different?

Clarify? about then RISK please?

[rbjtech 5284]

11 hours ago, TMCsw said:

I’m a little confused as this thread started as (I think?) about a vulnerability with using emby ‘Out Of the Box’?
But then evolved into reverse proxies or not ?

PLEASE Clarify…

I use nginx as a reverse proxy to my emby server…

https://securityheaders.com/
Gives me an A+

regardless of if my location {} contains either/or/not

  proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
  proxy_set_header X-Real-IP $remote_addr;

Also my nginx logs don’t seem to be different?

Clarify? about then RISK please?

See - 

 

In summary, as you are using nginx - there are no known vulnerability issues relating to this topic.

The 'fix' (I believe) was for users using emby as their public facing web server.  @softworkzcould you kindly confirm ?  Thanks.

 

Edited April 24, 2023 by rbjtech

[rbjtech 5284]

Fixed in .31 beta

Fix regression with realtime monitor causing events to not be triggered
Fix regression causing web app to not load when data explorer plugin is installed
Fix 172.X addresses always being considered private
Don't allow local network addresses to be specified in x-forwarded-for and x-real-ip
Various fixes with keep up to number of recordings option
Avoid refreshing metadata when files have a date modified timestamp of 0

 

[softworkz 5066]

14 hours ago, rbjtech said:

The 'fix' (I believe) was for users using emby as their public facing web server.  @softworkzcould you kindly confirm ?  Thanks.

Correct.

[pir8radio 1312]

On 4/22/2023 at 6:38 AM, Q-Droid said:

DM'd. As mentioned before Caddy does not append to that header and replaces instead. I'm not sure what the purpose would be for maintaining a trail of IPs like that in a simple RP setup. Perhaps if there was a need to traverse and track multiple zones/subnets in the application but as far as I can tell Emby only cares about the difference between local and remote.

 

seeing a trail would tell you if someone had a man in the middle reverse proxy setup, or if you pass through a company proxy, cloudflare proxy, load balancer, etc, keeps the logs accurate if you are log analyzing.  like all headers it can be spoofed.   but more info more control.