Emby Blog

4 minutes ago, MrSaumon said:

I agree on the fixing it quick, but I'm a bit disappointed in the way they did get the word out.
English is not my first language, so I maybe did get the tone wrong, but when I read :

"How we took down a BotNet of 1200 hacked Emby Servers within 60seconds"
For me it sounds like a wholesome story of a white-hat hacker group that saved people from evil guys, not the story of a company that had a security hole in their software (using the x-forwarded-for and x-real-ip headers as the truth).
Also, in the advisory :
"Starting Mid-May 2023, a hacker managed to infiltrate private user-hosted instances of Emby Server which were accessible via public internet and had an insecure configuration for administrative user accounts."
This sounds like the insecure configuration part is the users problem, but it's a checkbox in the standard configuration panel that allowed the security hole to be exploited.

I'm also not a huge fan of the remote software shutdowns, even if I understand the urgency on mitigating the issue, I don't really want every software on my server to be able to be remotely stopped by the companies. I maintain my server myself and try to keep up with the updates, but it's my server.

Anyway, I like the software, was not impacted by the hack and will continue to use it, just my honest opinion on this.

Described in open forum over 3 years ago.

6 minutes ago, HyphenZhao said:

Thanks for your quick action. As a software developer myself, I know that cyber-issue is inevitable. 

It would be much helpful if the team could list out what exactly the hack script did so that the admin could at least have a clue of which part to be more cautious about. A simply reset of the system is not possible for a server with lots of other online services. 

To my understanding, the script somehow was injected into my server and logged in every account which does not have local login password. And then what? Did they try to acquire administration privilege and use the API provided by Emby and try to retrieve the text-based passwords? If only the cyphered passwords, it should be alright. But somehow, they could get text-based passwords, that would be a disaster. So what exactly did the script do? And under what condition, like the local no-password accounts had admin privilege, would be more compromised? A clarification of the hack script functions would be very helpful and urgent.

Thanks.

According to this, the issue was that emby used the attacker's browser headers information to see if this is a local IP and considered every 172. addresses as local, which is wrong.

If you had the "enable admin login without a password if you're in the LAN" (or something like that) option enabled in your emby server, the attacker could login as admin without a password and do everything you can do on your server. In this case install a malicious  plugin that allowed full access to your server (limited by the access rights of emby, I guess, if there was no escalation vulnerability).

Not 100% sure about all of this, but it's my understanding of the vulnerability so far.

3 minutes ago, andrewds said:

Review the first post in this thread for additional context about the attack.

This as well.

Ouch... Thanks for the link, this was an interesting read (the second link gives me a 403, it was truncated for some reason)

3 minutes ago, MrSaumon said:

Ouch... Thanks for the link, this was an interesting read (the second link gives me a 403, it was truncated for some reason)

I corrected the link, sorry about that.

The directions are very clear that one has to change all users' passwords BEFORE starting the server back up... can anyone explain to me how that gets done? Since the entire user database is store ON the Emby server, it isn't accessible UNLESS the server is running.

 

The directions for cleaning a server leave a lot to be desired... So does the title of the thread, though - it should be "How we took down a botnet in 60 seconds AFTER it has been running for WEEKS"

28 minutes ago, MrSaumon said:

This sounds like the insecure configuration part is the users problem, but it's a checkbox in the standard configuration panel that allowed the security hole to be exploited.

The "insecure configuration" is a huge convenience.  From what I understand, that is the feature that allows to bypass the password on your local network.  I, and probably many others do this because who wants to type in a password through an on-screen keyboard every time they select themselves to login?  Passwords aren't supposed to be easy or reused either.  I certainly don't want to type in a 12 digit complex password just to switch between me and the wife's account. 

The steps they took seemed reasonable to me.  Shutting down your server (in most cases they shutdown someone's docker container) doesn't stop you from turning it back on.  It was done to get your attention, and it did.  We know there are people who don't update anything for months at a time, and if they didn't do that (and let it spread further), the forum would have been full of people saying they should have done more after they found all their media deleted one day...

Like I said, I'm not saying it isn't frustrating, but I think they did pretty good at shutting this down.  Let's not forget that many much larger (and better funded) companies have had hacks, so they can't "just plug holes" to fix everything.  It doesn't work that way in the connected world.  Bugs upstream, convenience vs. security, bad user practices...  It all plays a role.  They could make the servers "bulletproof", but you wouldn't be able to remotely access your media either.

I can remember when Kronos, who does payroll for half the world, had their entire infrastructure encrypted with ransomware making people go back to punch cards for six months.  I'm just reminding people that this isn't a huge corporation, but they handled it better than many with more resources do in similar situations.  We should all update, review our server security, and sit back and watch a movie...

2 minutes ago, HouseOfCards said:

The steps they took seemed reasonable to me.  Shutting down your server (in most cases they shutdown someone's docker container) doesn't stop you from turning it back on.  It was done to get your attention, and it did.  We know there are people who don't update anything for months at a time, and if they didn't do that (and let it spread further), the forum would have been full of people saying they should have done more after they found all their media deleted one day...

Like I said, I'm not saying it isn't frustrating, but I think they did pretty good at shutting this down. 

How so? At a MINIMUM, mine required being updated to a (yet another unsigned) new version of the software before it would start.

Why no email to let me know that it was offline? How many OTA recordings were missed because TV isn't my life and I didn't -know- it was down?

There are quite a lot of posts on this topic so far so excuse if I missed this but is it documented how Emby remotely shut down services on customer premises?

3 minutes ago, ember1205 said:

How so? At a MINIMUM, mine required being updated to a (yet another unsigned) new version of the software before it would start.

Why no email to let me know that it was offline? How many OTA recordings were missed because TV isn't my life and I didn't -know- it was down?

I don't know what you think could have been done differently?  I'm not trying to fight with you, I just think these cyber-threats have a lot of moving parts, and they took the needed steps to stop it.  I updated the two servers I have running, and it took less than five minutes.

Keep in mind, the moment the team starts sending notifications and posting on the forums about the steps they are taking, they are also telling those conducting the attack what they plan to do.  In this case, they shut it down, then dealt with getting people updated.  Seems reasonable is what I'm saying.  If you were affected, they shut your system down before any more damage could be done, like deleting your media...

I'm good with that. 

16 minutes ago, xe` said:

There are quite a lot of posts on this topic so far so excuse if I missed this but is it documented how Emby remotely shut down services on customer premises?

Did you read the Advisory ? (you can't miss it as it's an alert on the top of the forum)  - all details are in there.

https://emby.media/support/articles/advisory-23-05.html

Edited May 26, 2023 by rbjtech

After nearly a full day of trying, I'm completely stuck. I am unable to delete the .dll or xml files as for whatever reason WinSCP is telling me I need admin privileges. I am the only user on my pc, and I know what the log in info is once I SSH into my ras pi so it's impossible for me not to have admin privileges. If anyone knows what else I can do, as I was unable to find the files using Putty but did find them using WinSCP, I'd really appreciate the help. I'd love to avoid delete emby from yacht and portainer and avoid rebuilding if possible, but at this point I cannot remedy the situation

7 minutes ago, AP123 said:

After nearly a full day of trying, I'm completely stuck. I am unable to delete the .dll or xml files as for whatever reason WinSCP is telling me I need admin privileges. I am the only user on my pc, and I know what the log in info is once I SSH into my ras pi so it's impossible for me not to have admin privileges. If anyone knows what else I can do, as I was unable to find the files using Putty but did find them using WinSCP, I'd really appreciate the help. I'd love to avoid delete emby from yacht and portainer and avoid rebuilding if possible, but at this point I cannot remedy the situation

Are you ssh onto your rpi via 'root' or another account ?

If another account - then you can ssh onto the pi and then use sudo to elevate that account to 'root' privilages for the delete commands - that is probably what you need to do.

Out the box, Unix systems will very rarely let you use 'root' as a 'user' account.

15 minutes ago, HouseOfCards said:

I don't know what you think could have been done differently?  I'm not trying to fight with you, I just think these cyber-threats have a lot of moving parts, and they took the needed steps to stop it.  I updated the two servers I have running, and it took less than five minutes.

Keep in mind, the moment the team starts sending notifications and posting on the forums about the steps they are taking, they are also telling those conducting the attack what they plan to do.  In this case, they shut it down, then dealt with getting people updated.  Seems reasonable is what I'm saying.  If you were affected, they shut your system down before any more damage could be done, like deleting your media...

I'm good with that. 

I had specifically asked in the post you quoted why no emails were sent. I agree that their first effort shouldn't be to queue up emails and alert to the vulnerability and such prior to taking action, but I work for a software company - you send emails. They get sent in batches, not individually, and it isn't hard to do. Don't want to send to everyone? Fine. Send it to those that have Emby Premium (another benefit of paying).

I also would have appreciated clearer, more precise directions. They tell you to change all user password BEFORE starting the server back up, but that is literally impossible to do (unless I am completely missing something). Additionally, what EXACTLY was put at risk with a compromised server? Was information stolen off of the machines? Was the machine simply used as part of a botnet? It seems to me that there was a data exfiltration component here that no one is talking about...

9 minutes ago, rbjtech said:

Are you ssh onto your rpi via 'root' or another account ?

If another account - then you can ssh onto the pi and then use sudo to elevate that account to 'root' privilages for the delete commands - that is probably what you need to do.

Out the box, Unix systems will very rarely let you use 'root' as a 'user' account.

I’m sorry but I don’t know how to do this. Someone set up my pi for me. This is all new to me. I know the log in info tho. How can I give myself permissions? If you can give a clear step by step I’d appreciate it. Also now sure how to use sudo in winscp since basically I’m just going thru folders and clicking delete. I’m not typing command lines. I’m using it like explorer. 

Edited May 26, 2023 by AP123

18 minutes ago, AP123 said:

After nearly a full day of trying, I'm completely stuck. I am unable to delete the .dll or xml files as for whatever reason WinSCP is telling me I need admin privileges. I am the only user on my pc, and I know what the log in info is once I SSH into my ras pi so it's impossible for me not to have admin privileges. If anyone knows what else I can do, as I was unable to find the files using Putty but did find them using WinSCP, I'd really appreciate the help. I'd love to avoid delete emby from yacht and portainer and avoid rebuilding if possible, but at this point I cannot remedy the situation

On a Linux system (I am supposing that's what you have), you are not the only user... there are a lot of them. One of them in particular is "emby" and will own all of the content. Your user account won't have write permission to most of the files.

Use PuTTY or similar to access the machine via SSH (which is what WinSCP is using). Once logged in, you will need to change users to the root user or similar. THEN navigate the filesystem... Which underlying OS you are running will dictate exactly how you become the superuser. My guess would be that you're running Ubuntu and the process to directly become the superuser is to execute the command "sudo su". Be Careful. This user can literally do whatever they want, including completely crash the system.

 

6 minutes ago, AP123 said:

I’m sorry but I don’t know how to do this. Someone set up my pi for me. This is all new to me. I know the log in info tho. How can I give myself permissions? If you can give a clear step by step I’d appreciate it. Also now sure how to use sudo in winscp since basically I’m just going thru folders and clicking delete. I’m not typing command lines. I’m using it like explorer. 

so you need to elevate WinSCP to run Sudo - for guidance try -

 https://winscp.net/eng/docs/faq_su

51 minutes ago, ember1205 said:

The directions are very clear that one has to change all users' passwords BEFORE starting the server back up... can anyone explain to me how that gets done? Since the entire user database is store ON the Emby server, it isn't accessible UNLESS the server is running.

 

The directions for cleaning a server leave a lot to be desired... So does the title of the thread, though - it should be "How we took down a botnet in 60 seconds AFTER it has been running for WEEKS"

The instructions could be reordered for clarity. Until they are approach the action plan like a punch list. Complete the items you can and come back to the ones you can't due to dependencies like a running server. And for everyone the very first step should be to disable router port forwarding, firewall rules, gateway rules that allow the remote access.

17 minutes ago, ember1205 said:

On a Linux system (I am supposing that's what you have), you are not the only user... there are a lot of them. One of them in particular is "emby" and will own all of the content. Your user account won't have write permission to most of the files.

Use PuTTY or similar to access the machine via SSH (which is what WinSCP is using). Once logged in, you will need to change users to the root user or similar. THEN navigate the filesystem... Which underlying OS you are running will dictate exactly how you become the superuser. My guess would be that you're running Ubuntu and the process to directly become the superuser is to execute the command "sudo su". Be Careful. This user can literally do whatever they want, including completely crash the system.

 

I've been trying to upgrade my user to root and nothing works. I've read every suggestion on google so far for WinSCP to elevate to root, and it still doesn't work.

9 minutes ago, rbjtech said:

so you need to elevate WinSCP to run Sudo - for guidance try -

 https://winscp.net/eng/docs/faq_su

Thanks. I found this article. It didn't work.

1 minute ago, AP123 said:

I've been trying to upgrade my user to root and nothing works. I've read every suggestion on google so far for WinSCP to elevate to root, and it still doesn't work.

WinSCP won't allow you to elevate privs. You have to use a direct SSH client like PuTTY.

33 minutes ago, rbjtech said:

Did you read the Advisory ? (you can't miss it as it's an alert on the top of the forum)  - all details are in there.

https://emby.media/support/articles/advisory-23-05.html

I did and I have reread it but I can still see no details other than notes on the decision to take the action.

It should not be taken as a given that an addon system has this level of system control especially when the majority of addons are 3rd party. Consider that one of the addons in this event was malicious. Did it have access to the the same levels of system control that were used for "legitimate" remote shutdown? I would like to see specifics of how this was achieved and conceptual plans for limiting control on the future.

Just now, ember1205 said:

WinSCP won't allow you to elevate privs. You have to use a direct SSH client like PuTTY.

Ok great. Thanks for that. I've tried using putty using the sudo cmd prompts and I wasn't even able to find the dll or xml files when I ran the search, let alone delete them. If you can give some clear steps on how to do it then I'd try it. But as a very novice person trying to do this, nonspecific answers are just very hard for me to follow. I appreciate the help.

1 minute ago, AP123 said:

Ok great. Thanks for that. I've tried using putty using the sudo cmd prompts and I wasn't even able to find the dll or xml files when I ran the search, let alone delete them. If you can give some clear steps on how to do it then I'd try it. But as a very novice person trying to do this, nonspecific answers are just very hard for me to follow. I appreciate the help.

Once you've assumed superuser, try "cd /var/lib/emby/plugins" to change to where the DLL files -should- be. You can also locate them all with something like "find / -name *Helper.dll" (which will print out the full path of every file that matches by name. You're looking for Helper.dll and EmbyHelper.dll

2 minutes ago, AP123 said:

Ok great. Thanks for that. I've tried using putty using the sudo cmd prompts and I wasn't even able to find the dll or xml files when I ran the search, let alone delete them. If you can give some clear steps on how to do it then I'd try it. But as a very novice person trying to do this, nonspecific answers are just very hard for me to follow. I appreciate the help.

Let's take it over here and I can try to help you.

 

 

1 hour ago, MrSaumon said:

According to this, the issue was that emby used the attacker's browser headers information to see if this is a local IP and considered every 172. addresses as local, which is wrong.

If you had the "enable admin login without a password if you're in the LAN" (or something like that) option enabled in your emby server, the attacker could login as admin without a password and do everything you can do on your server. In this case install a malicious  plugin that allowed full access to your server (limited by the access rights of emby, I guess, if there was no escalation vulnerability).

Not 100% sure about all of this, but it's my understanding of the vulnerability so far.

Thanks! I also read Andrewds' threads over this. It is pretty helpful. I didn't have any admin user to be able to log in without passwords in LAN but I do have normal users be able to log in without passwords in LAN. I checked my plugins and saw there is only EmbyHelper.dll but no helper.dll. I assume I have got some luck but not all. I think my privacy data (such as photos with family, and videos of my wedding day) has been exposed to the hacker but not other credentials (e.g. passwords) . 

To my understanding, the hacker needs these steps to fully penetrate the whole system:

(A) Access Emby through public ports ->  (B) Pretend to be local login -> (C) Login as admin -> (D) Download helper.dll for further penetration -> (E) Completely gain access to the server

1. It seems that if the server was not publicly available or used different ports than standard (8096, 8920), the server should be not compromised at all (Level I Penetration)

2. If the local users could be seen but not be able to logged in at all, then only your username is compromised (Level II Penetration)

3. If the local users could be logged in without passwords, but no admin users could be logged in, then it would be my situation. That is saying your privacy data (all media and its related metadata) is compromised. (Level III Penetration)

4. If the helper.dll has been downloaded, but the system user of Emby does not have enough level to create other things, then the server is partially compromised depending on the privileges the Emby system user had (read, write, execute, etc). (Level IV Penetration)

5. If the Emby user had all access, then that mean the server is basically transparent to the hacker. And the following things would just be simply out of control. (Level V Penetration)

I was panicked yesterday and I now got a breath, although not fully relieved. Would just like to share my thoughts so that you can understand your situation. Nevertheless, if there is anything I listed that was wrong, please let me know. I may have to panic again if you tell me I am more compromised than I think but l will have to face it anyway. I thank the service provided by Emby these years and I appreciate the efforts of Emby's team to mitigate this cyber-issue. Nevertheless, I am very disappointed that the team didn't pay attention to this security issue which has been discussed thoroughly three years ago.