Emby Blog

17 hours ago, Luke said:

Hi, apologies for the disruption. Please see the attention banner at the top of the community for more info on how to get back up and running. Thanks.

Do we know if Emby Connect credentials were compromised with this? As far as I can see, the only 'full admin' account on my server had emby connect (I've made everyone else in the house make an account so I could get rid of local logins entirely) and while I was trying to reboot my docker container (yes, I've been using Hotio's dockerized Emby server) I kept getting reinfected. It appears I got it cleaned up after I also deleted all of the recently updated plugins (DLNA, MovieDB - which seems to be the vector you used to shut down servers if I read logs right?, OMDB and TVDB). One plugin that didn't come back after I purged recent ones (glad I took a screenshot) was 'EmbyHelper.dll', which was installed May 11th on my server. Not sure if that was a problematic plugin, or if it didn't reinstall for some other reason

This might be something other users are going to run into that are using Docker, because to my knowledge, we don't have a way to disable network access and still be able to access the Web GUI

Edited May 26, 2023 by sera_denoir

24 minutes ago, jackthejerk said:

I just wanted to share what our firewall had caught since the 11th (when this all began) from what I can tell.

I'm guessing they published at least my server info somewhere as I have since been getting multiple attempts to find more ways in. shown in the attached log.These are attempts and blocked, I do not know what may have gotten through. if anyone might see these or like addresses and attacks may be able to narrow down the source and targets

Same here I noticed on May 11th that several times Emby Scriper-X plugin was installed then the next entry was a failed login attempt. This followed by uninstalling the plugin only to be repeated 7 times with the last Alert entry being Emby Scripter-X 4.0.0.8 has been uninstalled and that entry was May 13th.

I believe that nothing serious has happened as the Emby server not starting was because the Emby team shut all of the infected ones down. Went and took the action as outlined and server started back up.

QUESTION: When will version 4.7.12 be available in a qpkg for a QNAP NAS?

11 minutes ago, sera_denoir said:

Do we know if Emby Connect credentials were compromised with this? As far as I can see, the only 'full admin' account on my server had emby connect (I've made everyone else in the house make an account so I could get rid of local logins entirely) and while I was trying to reboot my docker container (yes, I've been using Hotio's dockerized Emby server) I kept getting reinfected. It appears I got it cleaned up after I also deleted all of the recently updated plugins (DLNA, MovieDB - which seems to be the vector you used to shut down servers if I read logs right?, OMDB and TVDB). One plugin that didn't come back after I purged recent ones (glad I took a screenshot) was 'EmbyHelper.dll', which was installed May 11th on my server. Not sure if that was a problematic plugin, or if it didn't reinstall for some other reason

This might be something other users are going to run into that are using Docker, because to my knowledge, we don't have a way to disable network access and still be able to access the Web GUI

Since I can't edit anymore, a bit more of a post-mortem on my end for others who may have similar issues of reinfection:

EmbyHelper.dll showing up the same day Emby Scripter-X was installed and uninstalled (in 3 seconds, when I was asleep) strongly suggests that it was the vector used to install the EmbyHelper.dll plugin, which when I deleted that file seemed to prevent my server being reinfected with XML data (which it was doing even with the HOSTS edit to my docker config) and crashing the server. Since then, I've gotten the newly pushed security update.

The big concern I have is this: only my user account had 'manage the server' permissions. And the server noted a failed login but no successful login. Also, like I said before, my login was set up with Emby Connect already, so I had no local login. Does this mean that Connect's auth process were also compromised?

2 minutes ago, One2Go said:

Same here I noticed on May 11th that several times Emby Scriper-X plugin was installed then the next entry was a failed login attempt. This followed by uninstalling the plugin only to be repeated 7 times with the last Alert entry being Emby Scripter-X 4.0.0.8 has been uninstalled and that entry was May 13th.

I believe that nothing serious has happened as the Emby server not starting was because the Emby team shut all of the infected ones down. Went and took the action as outlined and server started back up.

QUESTION: When will version 4.7.12 be available in a qpkg for a QNAP NAS?

My dockerized instance just got the update about 10 minutes ago. I'd just check the server dashboard?

As far as the server not starting because all the infected ones went down is because Emby devs did that on purpose. How long the botnet was active I'm not seeing any confirmation on so far, but if you and I both had strange activity and failed logins on the 11th, I'm thinking parts of the botnet may have been running for two weeks

7 minutes ago, sera_denoir said:

My dockerized instance just got the update about 10 minutes ago. I'd just check the server dashboard?

As far as the server not starting because all the infected ones went down is because Emby devs did that on purpose. How long the botnet was active I'm not seeing any confirmation on so far, but if you and I both had strange activity and failed logins on the 11th, I'm thinking parts of the botnet may have been running for two weeks

Definitely May 11th through 13th for me was when all hell broke loose these are the Alerts that showed up in my Dashboard.

 

27 minutes ago, sera_denoir said:

Do we know if Emby Connect credentials were compromised with this?

No. Emby Connect (and thus forum account) credentials are not impacted. Only local Emby account credentials, and only if the system was compromised in the first place.

2 minutes ago, seanbuff said:

No. Emby Connect (and thus forum account) credentials are not impacted. Only local Emby account credentials, and only if the system was compromised in the first place.

How do you know if the system was compromised? I did find the EmbyHelper.dll in the plugin directory does tthat mean it was compromised? look at the above alerts from the dashboard, does that look like being compromised. The user ID that they were trying to use, had a very difficult password.

Thanks
O2G

2 minutes ago, One2Go said:

I did find the EmbyHelper.dll in the plugin directory does tthat mean it was compromised? look at the above alerts from the dashboard, does that look like being compromised

Yes it appears your system has been compromised. Have you shut down Emby and undertaken the steps covered in "Actions to Take" ?

6 minutes ago, One2Go said:

How do you know if the system was compromised? I did find the EmbyHelper.dll in the plugin directory does tthat mean it was compromised? look at the above alerts from the dashboard, does that look like being compromised. The user ID that they were trying to use, had a very difficult password.

Thanks
O2G

I'm not sure how you could know the difficulty of my password, but as it was the only admin account on the server, I changed my Emby Connect password anyway, as the whole thing is making me concerned.

Edit: Never mind, I confused myself because I thought you were talking to me, but now I think you were talking about your obfuscated user

Edited May 26, 2023 by sera_denoir

30 minutes ago, seanbuff said:

Yes it appears your system has been compromised. Have you shut down Emby and undertaken the steps covered in "Actions to Take" ?

I went through the whole "Actions to Take" bit. Shut the server down from my QNAP Apps section. Then deleted the one DLL I found (EmbyHelper.dll), added the entry to the hosts file. deleted the two files in the configuration directory. After that I started the server back up and I could log in and operation was just as normal.

What puzzles me is this, that the User they tried to use failed every time and it looks almost like they tried to log in with no password with that User as I have not used that User ID for months. When I started the server back up I got an alert "Security change applied" which said that the User ID that was being used and failed had no password. Very weird since I had a difficult password associated with it.

In any occasion that User I deleted and changed the password on the one User that can manage the server. I must say I find the Emby connect difficult to fathom as I do want an account that I can be sure to use when traveling and my ISP decided to give my router a different IP address.

1 hour ago, One2Go said:

How do you know if the system was compromised? I did find the EmbyHelper.dll in the plugin directory does tthat mean it was compromised? look at the above alerts from the dashboard, does that look like being compromised. The user ID that they were trying to use, had a very difficult password.

Thanks
O2G

You would know because the server would not be starting at all.

5 hours ago, katbyte said:

@Lukefrom the banner:

What does this mean? is it that the hack only worked on instances that didn't have a secure password on an admin account?

ie what precisely was the attack vector?

I am still reading through this thread but based on what I've read so far ... my assumption is that the hacker figured out a way through forging headers to convince Emby that the requests being sent from outside of the network were actually originating from a local IP address.  Since some people have their Emby configuration setup to automatically login to the system when on local network without requiring a password, and some of the accounts that allowed this were administrator accounts, the hacker was able to leverage this exploit to gain administrative access remotely.  This likely allowed them to have privileges on the Emby server under whichever account the Emby service was running as.

I was not impacted by this at least going by the fact that my server was not shutdown even though I did have an admin with local login privileges sans password.  I do run the Emby application with a dedicated non-admin account on my Windows system which only has access to the media files necessary to run Emby so hopefully that would have helped slightly in the event that someone were able to take it over.

I also run Emby on a non-standard port and I restrict network access at the firewall to only a few select countries.

Edited May 26, 2023 by moviefan

2 hours ago, jackthejerk said:

I just wanted to share what our firewall had caught since the 11th (when this all began) from what I can tell.

 

I'm guessing they published at least my server info somewhere as I have since been getting multiple attempts to find more ways in. shown in the attached log.

These are attempts and blocked, I do not know what may have gotten through. if anyone might see these or like addresses and attacks may be able to narrow down the source and targets

2008R210Today.txt 12.53 kB · 24 downloads

Looking really quickly through this data, I can see that all of these attempts appear to be from outside of the U.S.  If you are in the U.S., suggest restricting access to external countries ASAP.  If you are in another country, and don't need to allow access beyond that one, look to block everywhere else.

25 minutes ago, Luke said:

You would know because the server would not be starting at all.

I could stop and start the server from the QNAP app section I was just not able to bring up the dashboard it just said we are having trouble connecting and I believe that was because the Emby team shut down the server and prevented further startup. In addition I did not use 8096 as the port for administration.

I just take it that everything is back to normal after following through with the "Action to take" suggestions. When will the 4.7.12 qpck for QNAP be released? Any ETA?

Just now, One2Go said:

I could stop and start the server from the QNAP app section I was just not able to bring up the dashboard it just said we are having trouble connecting and I believe that was because the Emby team shut down the server and prevented further startup. In addition I did not use 8096 as the port for administration.

I just take it that everything is back to normal after following through with the "Action to take" suggestions. When will the 4.7.12 qpck for QNAP be released? Any ETA?

Correct.  The new release is already up. 

Thanks just updated to 4.7.12 and I am forgetting this day as I am sure many on the Emby team likewise want to forget this day.

Thanks for being diligent and helpful.

33 minutes ago, Luke said:

Correct.  The new release is already up. 

Is it supposed to be up for Windows too?  Not seeing the option to update on my server.

I went straight to the download part of the emby.media site and got the release from there. It didn't show up on the dashboard yet.

I'm so glad I didn't use any of the default port numbers - I'm sure I would have been hit if I did. 

@LukeA clarifying question regarding the attack vector. Did one dodge a bullet, if Emby has only been used locally (that is, no Emby Connect usage at all)?

There is nothing in the changelog attached to:

• https://github.com/MediaBrowser/Emby.Releases/releases/tag/4.7.12.0

to signify this is a critical security release.

I would suggest that as a matter of standard practice any security fixes be tagged as such so that the community and upstream who follow only Github for releases are aware.

I would add that with all the "cyber-issues" out there in the world, the Emby team did a great job getting the word out on this, and fixing it quick.  It's easy to get "upset" about this kind of stuff, but don't forget that software is complicated, and all we can do is react to new threats as they happen.  They did that.  Just practice good security measures, and keep on truckin'...

I woke up to a notification that an update was available, and done...

Thanks guys!

Thanks for your quick action. As a software developer myself, I know that cyber-issue is inevitable. 

It would be much helpful if the team could list out what exactly the hack script did so that the admin could at least have a clue of which part to be more cautious about. A simply reset of the system is not possible for a server with lots of other online services. 

To my understanding, the script somehow was injected into my server and logged in every account which does not have local login password. And then what? Did they try to acquire administration privilege and use the API provided by Emby and try to retrieve the text-based passwords? If only the cyphered passwords, it should be alright. But somehow, they could get text-based passwords, that would be a disaster. So what exactly did the script do? And under what condition, like the local no-password accounts had admin privilege, would be more compromised? A clarification of the hack script functions would be very helpful and urgent.

Thanks.

22 minutes ago, HouseOfCards said:

I would add that with all the "cyber-issues" out there in the world, the Emby team did a great job getting the word out on this, and fixing it quick.  It's easy to get "upset" about this kind of stuff, but don't forget that software is complicated, and all we can do is react to new threats as they happen.  They did that.  Just practice good security measures, and keep on truckin'...

I woke up to a notification that an update was available, and done...

Thanks guys!

I agree on the fixing it quick, but I'm a bit disappointed in the way they did get the word out.
English is not my first language, so I maybe did get the tone wrong, but when I read :

"How we took down a BotNet of 1200 hacked Emby Servers within 60seconds"
For me it sounds like a wholesome story of a white-hat hacker group that saved people from evil guys, not the story of a company that had a security hole in their software (using the x-forwarded-for and x-real-ip headers as the truth).
Also, in the advisory :
"Starting Mid-May 2023, a hacker managed to infiltrate private user-hosted instances of Emby Server which were accessible via public internet and had an insecure configuration for administrative user accounts."
This sounds like the insecure configuration part is the users problem, but it's a checkbox in the standard configuration panel that allowed the security hole to be exploited.

I'm also not a huge fan of the remote software shutdowns, even if I understand the urgency on mitigating the issue, I don't really want every software on my server to be able to be remotely stopped by the companies. I maintain my server myself and try to keep up with the updates, but it's my server.

Anyway, I like the software, was not impacted by the hack and will continue to use it, just my honest opinion on this.

Review the first post in this thread for additional context about the attack.

This as well.

Edited May 26, 2023 by andrewds