Emby Blog

20 minutes ago, xe` said:

I did and I have reread it but I can still see no details other than notes on the decision to take the action.

It should not be taken as a given that an addon system has this level of system control especially when the majority of addons are 3rd party. Consider that one of the addons in this event was malicious. Did it have access to the the same levels of system control that were used for "legitimate" remote shutdown? I would like to see specifics of how this was achieved and conceptual plans for limiting control on the future.

I believe there will be many internal follow up's, recommendations and changes made - but the most important priority is systems that were compromised have been shutdown to limit further damage.  The current phase, is getting those systems back up with the threat removed.   We would all like to see 'the next steps' but I think that is an reasonable ask at this time.

17 minutes ago, HyphenZhao said:

1. It seems that if the server was not publicly available or used different ports than standard (8096, 8920), the server should be not compromised at all (Level I Penetration)2

While the standard emby ports are the obvious target - ANY port on the Release version is advertising it is 'emby' in the http headers.  This is also the reason for the 'Google' phishing site reports.  .12 and the last couple Beta's have resolved this, by using invididual / non-emby headers - so a query no longer comes back with any indication it's emby.   So by now using 'emby' ports (and not 'common 80/443) you are actually at a disadvantage..

The other 'layers' are correct - but the key one is if the user CHANGED the default 'need a password for local login' to 'do not need a password' on the ADMIN account.  I emphasing on the ADMIN account.  If this was set on normal users, then it was not and is still not a problem as the user does not have permissions to install Plugins etc.   If they unfortunately did this, then remote Admin access has been granted without the need for the password due to the combination of the local/wan spoofing vulnerability and the above password setting.

Edited May 26, 2023 by rbjtech

17 minutes ago, rbjtech said:

I believe there will be many internal follow up's, recommendations and changes made - but the most important priority is systems that were compromised have been shutdown to limit further damage.  The current phase, is getting those systems back up with the threat removed.   We would all like to see 'the next steps' but I think that is an reasonable ask at this time.

I can partially agree here, but will also say that the staff of Emby needed to fully understand what this compromise represented before they took action like was done.

 

In other words... The process that was used to deem this "critical enough" to shut down everyone's system would ALSO have afforded a lot more detail that can now be shared with users.

Hi,

 

this all is a bit scary. Makes me feel a bit uncomfortable 

 

was this 

 

""Proxy Header Vulnerability"

 

possible with an revers proxy in front?

Cause I have one and I was as far as I can see not harmed. (Hopefully I am right)

Just now, ember1205 said:

I can partially agree here, but will also say that the staff of Emby needed to fully understand what this compromise represented before they took action like was done.

All I can say is they did - a significant amount of investigative analysis and testing was done before the action was taken and then subsequent follow up communication.  Clearly it would not be in the best interests to communicate first and apply the action later - for obvious reasons..

1 minute ago, ember1205 said:

In other words... The process that was used to deem this "critical enough" to shut down everyone's system would ALSO have afforded a lot more detail that can now be shared with users.

It is up to emby what they wish to share, and lets be honest here, while there are no longer any immediate threats, emby have some security 'concerns' they need to address asap.  imo openly listing these is not in the best interests of the users and is just providing information for hackers to find the next exploit. 

To be clear here, I'm not defending emby due to the lack of acknowledgement/action on the original disclosure but I am very much hoping this is a big wake up call for them to put security at the top of their priority list and keep it there .. 

13 minutes ago, schamane said:

Hi,

 

this all is a bit scary. Makes me feel a bit uncomfortable 

 

was this 

 

""Proxy Header Vulnerability"

 

possible with an revers proxy in front?

Cause I have one and I was as far as I can see not harmed. (Hopefully I am right)

A RP will re-write it's own headers - so it's highly unlikely that vulnerability passed through your RP.  

42 minutes ago, HyphenZhao said:

Thanks! I also read Andrewds' threads over this. It is pretty helpful. I didn't have any admin user to be able to log in without passwords in LAN but I do have normal users be able to log in without passwords in LAN. I checked my plugins and saw there is only EmbyHelper.dll but no helper.dll. I assume I have got some luck but not all. I think my privacy data (such as photos with family, and videos of my wedding day) has been exposed to the hacker but not other credentials (e.g. passwords) . 

To my understanding, the hacker needs these steps to fully penetrate the whole system:

(A) Access Emby through public ports ->  (B) Pretend to be local login -> (C) Login as admin -> (D) Download helper.dll for further penetration -> (E) Completely gain access to the server

1. It seems that if the server was not publicly available or used different ports than standard (8096, 8920), the server should be not compromised at all (Level I Penetration)

2. If the local users could be seen but not be able to logged in at all, then only your username is compromised (Level II Penetration)

3. If the local users could be logged in without passwords, but no admin users could be logged in, then it would be my situation. That is saying your privacy data (all media and its related metadata) is compromised. (Level III Penetration)

4. If the helper.dll has been downloaded, but the system user of Emby does not have enough level to create other things, then the server is partially compromised depending on the privileges the Emby system user had (read, write, execute, etc). (Level IV Penetration)

5. If the Emby user had all access, then that mean the server is basically transparent to the hacker. And the following things would just be simply out of control. (Level V Penetration)

I was panicked yesterday and I now got a breath, although not fully relieved. Would just like to share my thoughts so that you can understand your situation. Nevertheless, if there is anything I listed that was wrong, please let me know. I may have to panic again if you tell me I am more compromised than I think but l will have to face it anyway. I thank the service provided by Emby these years and I appreciate the efforts of Emby's team to mitigate this cyber-issue. Nevertheless, I am very disappointed that the team didn't pay attention to this security issue which has been discussed thoroughly three years ago. 

I had an additional question. I didn't have my admin users to be accessible through LAN. However, I still see EmbyHelper.dll. Does that mean my server was still compromised somehow (As Level 5 Penetration I mentioned above)? Is EmbyHelper.dll essentially the same as helper.dll?

15 hours ago, KMBanana said:

Do we know if Jellyfin is vulnerable to the same cause (Proxy headers able to identify as local IP addresses and a setting allowing passwordless access from local IP addresses)?

If so, were their devs given a heads up or are they going to be scrambling?

No. Jellyfin is not currently vulnerable to this particular issue. It was fixed in our 10.7.0 release in March 2021.

Also no. The Emby team has never communicated any known security issues to us. 

3 minutes ago, HyphenZhao said:

I had an additional question. I didn't have my admin users to be accessible through LAN. However, I still see EmbyHelper.dll. Does that mean my server was still compromised somehow (As Level 5 Penetration I mentioned above)? Is EmbyHelper.dll essentially the same as helper.dll?

As per the Article - if any or both of those two DLL's are in the Plugins directory, then consider them and your system compromised.

14 minutes ago, rbjtech said:

A RP will re-write it's own headers - so it's highly unlikely that vulnerability passed through your RP.  

Just as an addendum, a properly configured reverse proxy should do this, but it's entirely possible to have a poorly configured one.  Without adding those headers its possible Emby would see every remote connection as coming from the local address of the reverse proxy itself.  

FYI

https://emby.media/community/index.php?/topic/118955-full-disclosure-data-collection-in-the-process-of-botnet-takedown/

Closing loop - backup data, reset Asustor NAS, reinstalled with 4.7.12 package.   Emby is back.   Still rebuilding and re-configuring.

I do not know how to access all these files on synology nas to change them, nor should i have to. I'm paying for a product that is obviously faulty and now i have to fix it. I read that 4.7.12 will fix the problem but it doesn't. wasted my money for a premium subscription for a faulty product. Are they going to come out with an update that actually fixes this problem that they created?

12 minutes ago, hotwheels26180 said:

I do not know how to access all these files on synology nas to change them, nor should i have to. I'm paying for a product that is obviously faulty and now i have to fix it. I read that 4.7.12 will fix the problem but it doesn't. wasted my money for a premium subscription for a faulty product. Are they going to come out with an update that actually fixes this problem that they created?

Please name a piece of software that you have had to purchase (directly or via subscription) that was completely bug-free and never required you to install updates to it.

1 hour ago, hotwheels26180 said:

I do not know how to access all these files on synology nas to change them, nor should i have to. I'm paying for a product that is obviously faulty and now i have to fix it. I read that 4.7.12 will fix the problem but it doesn't. wasted my money for a premium subscription for a faulty product. Are they going to come out with an update that actually fixes this problem that they created?

I get it where you are coming from. I was a total knewby when it came to Linux, but because I run QNAPs I had to bite the bullet and get a minimum education on how to deal with Linux. That included installing Putty, learning how to work in a command prompt window, plus get to know a few commands. Minimum is cd (Change directory), ls (List Directory), rm (Remove Files), vi (Visual Editor to modify files like the hosts one). All this is available with Google searches and the forum for you Linux device.

Didn't want to do this but since I run media servers on UnRaid and QNAP NAS devices it was a must. This is part and parcel of being in the digital world and it will not go away.

Now to your problem Synology is no different then my QNAP and if you follow the "Action to Take" instructions from the Alert you will be fine since it corrected my problem.
1. Install Putty (Windows device)
2. Navigate to the Plugin Directory (remember the directory ,qpkg may not show up, but if you type .q and then hit the TAB key it will auto complete the entry)
3. delete the DLLs
4. Edit the hosts file
5. Delete the two files in the plugin configuration directory.

Before you start stop the Emby server from your Synology App section where you manually installed it. REMEMBER in Linux all names are case sensitive. Emby is different then emby and will not be recognized as the same.

All the best to get back to normal as it sucks to have ones routines interrupted.

the full path to the plugin directory is
/share/CACHEDEV1_DATA/.qpkg/EmbyServer/programdata/plugins/

Edited May 26, 2023 by One2Go

5 hours ago, ember1205 said:

Additionally, what EXACTLY was put at risk with a compromised server? Was information stolen off of the machines? Was the machine simply used as part of a botnet? It seems to me that there was a data exfiltration component here that no one is talking about...

Anything could have theoretically happened to a compromised server.  If the Emby service was being run as a user with admin privileges on the box, the hacker could have used the scripting plugin to download and execute literally anything including remote access trojans (RAT) that would have given them the same level of access as a user sitting in front of the computer with a keyboard and mouse.

Participating in botnets, stealing crypto private keys, stealing login information for websites being accessed from the system, harvesting credentials from a password manager that was logged into, etc.  This was a full blown RCE with all relevant implications.

4 minutes ago, moviefan said:

Anything could have theoretically happened to a compromised server.  If the Emby service was being run as a user with admin privileges on the box, the hacker could have used the scripting plugin to download and execute literally anything including remote access trojans (RAT) that would have given them the same level of access as a user sitting in front of the computer with a keyboard and mouse.

Participating in botnets, stealing crypto private keys, stealing login information for websites being accessed from the system, harvesting credentials from a password manager that was logged into, etc.  This was a full blown RCE with all relevant implications.

Other comments indicating that extensive investigation was done, so it isn't unreasonable to believe that at least SOME examples of data exfiltration are know and could be shared as things others may have reported.

I do understand that the likelihood of being able to nail it down to an explicit list is almost impossible, but it would be useful to at least share some of the things that are known to have occurred.

Ok, so let's get this straight because the more I think about it the more it irks me:

1)The vulnerability was KNOWN, REPORTED AND ACKNOWLEDGED by the Emby team for 3 years.  It was 3 years out there in the open in this very same forum and they did not patch it until someone just decided to mess things up.  Jellyfin, their free open source competitor had this patched a long, long, long time ago.

2)They had to make the choice of forcing an update to clients that will break their setups just to do damage control, which in an of itself is a big issue because tons of users (and paid customers might I add) have no idea how to fix this themselves, as we can see in the posts here.  Don't get me wrong, I get the lesser evil thing, but I honestly don't like the idea of the devs being able to decide to shut down my system

3)As of today, no e-mail was sent, no communication other than the security update (which you can easily miss if you are not checking), the forum post (which, again, I don't visit the website every day) and the broken setup (which you can easily miss if you are not using Emby daily, if you are on holidays, etc).  There might be people out there with compromised systems and credentials that have not found out yet because they are not super regular users, or systems that might be turned off but when turned on will have an outdated an still insecure version of Emby. 

4)Again, and this is crazy, they frame this as a good thing "We saved you from the evil BotNets!". No! You had a huge vulnerability exposed, out in the open on your forums go unpatched for 3 YEARS. I have not seen anyone from the dev team either apologize or explain this.  A post explaining how did this happen and what steps will be taken so that security vulnerabilities are taken seriously AND patched is definitely in order.

 

The issue is not that there was a security vulnerability, that's just the cost of developing software, but that after it was disclosed it was not fixed (again, 3 years, I can't state this enough times), it was mishandled and it was not properly disclosed to ALL users, but rather they were more worried on getting their own spin on the news than on actually alerting everyone their systems might be compromised.

 

PLEASE, send a damn e-mail letting people know what happened, NOT EVERY EMBY USER MIGHT NOW AND THERE'S PROBABLY VULNERABLE SYSTEMS UNPATCHED OUT THERE WITH PEOPLE'S DATA.

For anyone having trouble elevating their permissions to root to delete these files, I finally found some helpful info how to do it. You need both Putty and WinSCP. Here is the link. I may have finally been able to delete these files! 

https://easycodeforall.com/winscp-root-access-login.html

1 hour ago, ember1205 said:

Please name a piece of software that you have had to purchase (directly or via subscription) that was completely bug-free and never required you to install updates to it.

I don't have a problem with updating to fix it. I have a problem with having to go in and fix by adding and deleting system files. That should be done with an update. I have never had windows shut my system down. They have always fixed it with an update.

I was finally able to remove the files and get my server access back. Now I'm having two issues if anyone could please assist me with step by step.

1. I'm fairly confident I mistyped my new password for my admin account because it tells me my pw is incorrect. Need help resetting.

2. I don't know how to update emby to the latest version, if anyone could walk me through that.

26 minutes ago, hotwheels26180 said:

I don't have a problem with updating to fix it. I have a problem with having to go in and fix by adding and deleting system files. That should be done with an update. I have never had windows shut my system down. They have always fixed it with an update.

Not sure what flavor of windows you are running, but mine is shut down on an almost monthly basis.  All in the name of installing security updates.

26 minutes ago, AP123 said:

I was finally able to remove the files and get my server access back. Now I'm having two issues if anyone could please assist me with step by step.

1. I'm fairly confident I mistyped my new password for my admin account because it tells me my pw is incorrect. Need help resetting.

2. I don't know how to update emby to the latest version, if anyone could walk me through that.

You should have a link on the login page to recover / reset a password. Click that and it will write a file to disk that contains the new URL to use as well as the PIN you'll need to enter... Once you have that sorted, you can focus on software updates.

2 minutes ago, ember1205 said:

You should have a link on the login page to recover / reset a password. Click that and it will write a file to disk that contains the new URL to use as well as the PIN you'll need to enter... Once you have that sorted, you can focus on software updates.

Again, I appreciate the help, but if I was smart enough to find that file and where on the server it was saved then I'd just be going to do it. I got the message saying "a file as been created on your server" but have no idea where it is. 

Just now, AP123 said:

Again, I appreciate the help, but if I was smart enough to find that file and where on the server it was saved then I'd just be going to do it. I got the message saying "a file as been created on your server" but have no idea where it is. 

Oh... On mine, is specifically tells me the directory and file name that was written. What OS are you running?