WAN Access treated like LAN when "X-Forwarded-For" header is set, allowing login w/o password

[pwhodges 1531]

Fair enough.

Paul

[softworkz 3335]

7 hours ago, rbjtech said:

A professional change management platform such as Github would have simply highlighted the 'Issue' as it would be given a formal Issue number, tagged as high priority/security and tracked as such.

The problem in this case was an incorrect risk assessment, and the way how you are tracking issues doesn't matter much when you assign a wrong severity to the issue.

7 hours ago, rbjtech said:

Logging a security issue, bug or any other important detail in a forum is just hit and miss if you get a response - I'd imagine it certainly isn't tracked beyond best efforts.

I appreciate everyone is not going to have or want a Github account - but the Mods can raise issues they deem worthy if necessary - and then provide a link in the forum.  Github then takes over the management of the issue raised, with all tracking, reporting, comments etc coming from a tool designed to do just that.

I agree that the forums are very unsuitable for tracking issues. On the other side, if our internal tracking would be public, it would be difficult to understand and we would need to permanently explain entries and have an additional layer for non-public discussions.

At one time I was wondering whether there might be some plugin for the forums which provides some tracking capabilities but I haven't come to pitch the idea and research on it..

[rbjtech 4265]

3 hours ago, ebr said:

FYI We use GitHub issues internally.

Sure - I was just answering the question from @cptloresabout how a formal change system has many advantages over a forum - but I know you know this.. as all the embysupport stuff is on Git ..    

[pr3dict 33]

Yeah, so this sucks... Im like 35% good at my debian security life. I installed this when logged in as root but it runs as a service under the emby username/group. Am I screwed and should be reinstalling all my linux VM's? 

I'm also confused how this happened and would like some more explanation as to what specifically happened. Words like "Exposed to the internet" is too vague. My install was behind a reverse proxy that had ACL's for lan/wan access. Yes WAN access was allowed but with a robust admin password this doesnt explain how an actor was able to get in. From there what did they use in the emby programming to install malware? 

[pr3dict 33]

I also had SSL cert private keys stored on that machine. Should I be going back to my CA and having new keys and certificates generated? This seems like a bigger issue then the attention it's being given. 

[softworkz 3335]

1 hour ago, pr3dict said:

Yeah, so this sucks... Im like 35% good at my debian security life. I installed this when logged in as root but it runs as a service under the emby username/group. Am I screwed and should be reinstalling all my linux VM's? 

I'm also confused how this happened and would like some more explanation as to what specifically happened. Words like "Exposed to the internet" is too vague. My install was behind a reverse proxy that had ACL's for lan/wan access. Yes WAN access was allowed but with a robust admin password this doesnt explain how an actor was able to get in. From there what did they use in the emby programming to install malware? 

Has your system been shut down and prevented from re-starting with log entries pointing to the forums here?

[Carlo 4330]

2 hours ago, pr3dict said:

Yeah, so this sucks... Im like 35% good at my debian security life. I installed this when logged in as root but it runs as a service under the emby username/group. Am I screwed and should be reinstalling all my linux VM's? 

I'm also confused how this happened and would like some more explanation as to what specifically happened. Words like "Exposed to the internet" is too vague. My install was behind a reverse proxy that had ACL's for lan/wan access. Yes WAN access was allowed but with a robust admin password this doesnt explain how an actor was able to get in. From there what did they use in the emby programming to install malware? 

Did you get a message in your logs or console that your system had been breached or was your server shutdown?

I'm not trying to deflect here, but the "X-Forwarded-For" issue is only 1 part of 3 that would have let a hacking in.
The "X-Forwarded-For" part in this is allowing a remote person to look local. That's all it really does.

Part 2 is the Admin of the system didn't have the top or bottom option enabled for an account with management permissions.

The admin might have thought it was ok to show the admin account on the local network for quick login on different devices. With the header exploit the hacker can now appear to be a local IP address and would see different local logins which at least one of them had management permissions.  The hacker would probably not be able to tell a normal from admin account until named "manager", "sysop", "admin", "god", "root" or similar but it really doesn't matter as the hacker will try to login to each account shown to him.

At this point he had a valid login (maybe with management permissions).

Part 3 is the final needed part of this.  When setting up any admin account there was no password created OR had this option enabled on the admin account.

With either the Don't require or no password set, the hacker does nothing but click on the user to gain access to the system.

Take away is don't use your admin account for daily media playback but reserve it for admin use only.
Set a good password on admin accounts.
Use the HIDE options so admin accounts do not show up for quick log ins.

Options:
Make sure all accounts have passwords or at least PINs (normal accounts).
Don't use your admin account for general media playback but reserve it for management tasks.  Assign yourself a 2nd account used for general use.
Consider setting the admin account so it can only be used by a couple Emby Theater installations you have setup.

If allowing a login without PIN or PASSWORD on a local account (for use by young kids, grandparents, babysitter, guest) make sure they have no destructive permissions such as being able to delete media.  Assume this account can be easily breached as anyone on your LAN can use it.
 

 

 

Edited June 12, 2023 by Carlo

[pr3dict 33]

Yes - I was affected and did the clean-up as explained in the advisory. I have not changed my passwords yet for all the users but am keeping the server off for now. This is the only publicly available service that I have that does not rely on the proxy to obey the ACL's for "admin usage". The application here was the only line of defense. 

 

I totally understand the "good security practice" post you just made. However, it does not answer the question regarding why it was said that 

Quote

Emby Server systems configured to allow optional remote Internet access that are also configured to allow local admin account usage without a required password that also do not employ a properly configured reverse proxy server or CDN could be exploited to gain remote admin access without a password.

I believe I have a properly configured reverse proxy that handle ACLs based on forwarded-for. Unless you're saying thats the wrong way for a reverse proxy to be secure and that anyone can bypass the ACL's for any local resource by spoofing their IP? 

Edited June 12, 2023 by pr3dict

[darkassassin07 423]

16 minutes ago, pr3dict said:

I totally understand the "good security practice" post you just made. However, it does not answer the question regarding why it was said that 

I believe I have a properly configured reverse proxy that handle ACLs based on forwarded-for. Unless you're saying thats the wrong way for a reverse proxy to be secure and that anyone can bypass the ACL's for any local resource by spoofing their IP? 

It is possible to configure a reverse proxy to not set but simply pass these headers as given by the client. It is also possible to configure a reverse proxy to not pass the client ip at all, while itself is also within the same lan as the emby instance. Both of these would allow WAN clients to appear as if they are LAN clients either intentionally or simply by default due to the misconfiguration.

 

It's unlikely that either are the case, but they are a possibility. If you had been directly affected by the attack, your server would not have started after installing update 4.7.13.0.

[pr3dict 33]

14 minutes ago, darkassassin07 said:

It is possible to configure a reverse proxy to not set but simply pass these headers as given by the client. It is also possible to configure a reverse proxy to not pass the client ip at all, while itself is also within the same lan as the emby instance. Both of these would allow WAN clients to appear as if they are LAN clients either intentionally or simply by default due to the misconfiguration.

 

It's unlikely that either are the case, but they are a possibility. If you had been directly affected by the attack, your server would not have started after installing update 4.7.13.0.

It would have started after installing te update if the issue was fixed... I deleted the dll's and config files and it started back up. Was that not the expected result?

[softworkz 3335]

The shutdown does not depend on the server version.

[softworkz 3335]

5 minutes ago, pr3dict said:

I deleted the dll's and config files and it started back up. Was that not the expected result?

Yes it was.

[rbjtech 4265]

12 hours ago, pr3dict said:

Yes - I was affected and did the clean-up as explained in the advisory. I have not changed my passwords yet for all the users but am keeping the server off for now. This is the only publicly available service that I have that does not rely on the proxy to obey the ACL's for "admin usage". The application here was the only line of defense. 

 

I totally understand the "good security practice" post you just made. However, it does not answer the question regarding why it was said that 

I believe I have a properly configured reverse proxy that handle ACLs based on forwarded-for. Unless you're saying thats the wrong way for a reverse proxy to be secure and that anyone can bypass the ACL's for any local resource by spoofing their IP? 

Maybe post that portion of your Reverse Proxy config ?

In my NGINX config as an example, I only use X-Real-IP $remote_addr - no other X-Forwarded-For header lines are needed or required - as I believe these potentially do forward client requests but I'm not an NGINX expert by any means.  Emby works (and shows the WAN IP) perfectly well with just the X-Real-IP config.

To note, if your server was compromised, then they passed through your 1st line of defense - so you need to review that configuration asap - as that vulnerability/weakness is still there - it is not related to emby.

Edited June 12, 2023 by rbjtech

[pr3dict 33]

2 hours ago, rbjtech said:

Maybe post that portion of your Reverse Proxy config ?

In my NGINX config as an example, I only use X-Real-IP $remote_addr - no other X-Forwarded-For header lines are needed or required - as I believe these potentially do forward client requests but I'm not an NGINX expert by any means.  Emby works (and shows the WAN IP) perfectly well with just the X-Real-IP config.

To note, if your server was compromised, then they passed through your 1st line of defense - so you need to review that configuration asap - as that vulnerability/weakness is still there - it is not related to emby.

I don't know necessarily if my first line of defense is a defense in the way that it was setup. my reverse proxy uses ACL's based on IP address to allow access to certain servers. This is in place so that I can use a full.domain.name with one SSL port and one SSL cert for all my different servers. The ONLY server I have that is not behind a firewall rule blocking outside traffic is emby - For obvious reasons.

I'm currently checking with PFSense on their implementation of forward-for headers on HAProxy. HAProxy obviously knows what the actual IP address of a connecting client is vs whatever is provided in the header. I'm just not sure if there is a default to just forward what is provided vs inserting the real IP. 

 

anyway my last question regarding this is - Could this have been possible through emby-connect?

[rbjtech 4265]

3 minutes ago, pr3dict said:

I don't know necessarily if my first line of defense is a defense in the way that it was setup. my reverse proxy uses ACL's based on IP address to allow access to certain servers. This is in place so that I can use a full.domain.name with one SSL port and one SSL cert for all my different servers. The ONLY server I have that is not behind a firewall rule blocking outside traffic is emby - For obvious reasons.

ok - without seeing the reverse proxy setup its difficult to fully understand what you mean here - but any WAN/Public access would have had to pass though a NAT and firewall and maybe a WAF to even get to the RP.    If your RP is not behind a firewall, then I think you may need to review your security design, but I may be confused with your wording above.

10 minutes ago, pr3dict said:

anyway my last question regarding this is - Could this have been possible through emby-connect?

I don't use it myself, but from my understanding - emby connect is effectively a DDNS service that re-directs users to the local public IP address.  Once re-directed, no traffic is passed through the service as the client then connects directly.  For further details, you'll need to get an answer from others I'm afraid as that is as much as I know about 'emby-connect'..

[pr3dict 33]

8 minutes ago, rbjtech said:

ok - without seeing the reverse proxy setup its difficult to fully understand what you mean here - but any WAN/Public access would have had to pass though a NAT and firewall and maybe a WAF to even get to the RP.    If your RP is not behind a firewall, then I think you may need to review your security design, but I may be confused with your wording above.

So the way the setup is, and I think should be is.

 

Client ---> my.website.com:443 --> DNS comes back with 300.302.23.10

Client ---> 300.302.23.10 (Firewall/NAT) with IP Header that says they were looking for my.website.com:443

Firewall says, anything originating from WAN destined for WAN Address of 300.302.23.10 on port 443 goes to reverse proxy. 

Reverse proxy says thank you and looks at destination was supposed to be my.website.com and finds the backend server that its supposed to go to. 

NOW - talking about what network interface the proxy is actually listening /communicating on is a great question. The Reverse proxy is definitely the gate keeper as the firewall allows all 443 traffic to move forward to it. 

 

[Carlo 4330]

20 hours ago, pr3dict said:

I totally understand the "good security practice" post you just made. However, it does not answer the question regarding why it was said that 

"Emby Server systems configured to allow optional remote Internet access that are also configured to allow local admin account usage without a required password that also do not employ a properly configured reverse proxy server or CDN could be exploited to gain remote admin access without a password."

What I meant is that your reverse proxy is your first line of defense to fight of things of this nature. Configured correctly your Proxy would never allow any traffic through it that has any type of spoofed LAN IP. Not only should it not let the traffic through, but you may want to ban the real IP as well for a period of time.

[softworkz 3335]

8 hours ago, rbjtech said:

In my NGINX config as an example, I only use X-Real-IP $remote_addr - no other X-Forwarded-For header lines are needed or required - as I believe these potentially do forward client requests but I'm not an NGINX expert by any means.  Emby works (and shows the WAN IP) perfectly well with just the X-Real-IP config.

You need to be careful and ensure that your RP does not copy the X-FORWARDED-FOR header to the internal connection, because Emby Server will use X-FORWARDED-FOR first and only when absent it will use X-REAL-IP.

[pr3dict 33]

Yeah it seems that the implementation of HAProxy on PFSense does not strip the provided IP in the header before sending to the backend server. HAProxy can be configured to take that provided IP off and only put the real IP but I don't have any way to test it. 

However, as Carlo said earlier I probably should have had an IDS or something setup beforehand to block traffic that was trying to provide an IP that wasn't its own. Not that I'll know if this will work but I just added snort to my WAN interface haha. It's a first step I guess. Im not even sure if it'll work as designed because the only port that was open was 443. Without remembering my IP layers, etc, I don't know what value this will be.

[rbjtech 4265]

10 hours ago, pr3dict said:

However, as Carlo said earlier I probably should have had an IDS or something setup beforehand to block traffic that was trying to provide an IP that wasn't its own. Not that I'll know if this will work but I just added snort to my WAN interface haha. It's a first step I guess. Im not even sure if it'll work as designed because the only port that was open was 443. Without remembering my IP layers, etc, I don't know what value this will be.

It will be of significant value.   An IPS/IDS is inspecting packet headers and patterns that still exist even in encrypted traffic.  Obviously it cannot inspect the actual packet contents... On my IPS, I see a lot of hits/drops on 443 - mainly port scans but also a lot of bot nets attempting to gain zombies..  

Watch for performance hits though - if you are running low end hardware, then this may reduce your max throughput.

I would also add Geoblocking and known blacklist IP's if you can (all available in PFSense I think) - if all your clients are in the same country, then this will significantly cut down on scans from know 'intenet unfriendly' countries..

Edited June 13, 2023 by rbjtech