FULL DISCLOSURE: May 2023 Security Incident Report

[softworkz 3338]

As promised before, here's our final report, detailing the events and measures taken around the security incident and attack on Emby Server installations in May 2023:

Emby Malware Incident Report 2023-05.pdf

I hope, this will help to address remaining questions. We don't have all answers about origin, background and intentions of the attackers, but we tried to transparently document what we did when and how and why.

softworkz

[crusher11 854]

16 hours ago, softworkz said:

I hope, this will help to address remaining questions...we tried to transparently document what we did when and how and why.

Quote

Disclosure has been postponed. 

Uh huh. 

[softworkz 3338]

Amendment to section 1.3.1-2

The reason for postponing the disclosure of the mentioned vulnerability is that there is still a number of Emby Servers online where users haven't updated yet. As long as that is the case, we will not disclose the vulnerability. 

Edited June 21, 2023 by softworkz

[crusher11 854]

There's a sentence fragment pointing to footnote 16 that repeats three times at random locations in the document.

I'll admit I misread that section I quoted earlier, my initial takeaway was that it was saying the first, identified vulnerability wasn't used, when it's actually saying that of the second, unidentified vulnerability. So it's not the concern I thought it was, although as someone who was until last night stuck on a vulnerable version due to the DSM 6 issues it's not great. 

[adrianwi 238]

A few questions that I don't believe the report addresses:

• Why wasn't the thread below addressed in the following 3 years, which may have avoided this situation?
• What is being done to make sure things like this can't be missed in the future?
• What changes are being made to the plugin system to make this kind of exploit impossible/more difficult in the future?
• What changes will be made to emby to improve user security?

Thanks

[softworkz 3338]

@adrianwiThanks for some very good questions.

 

19 hours ago, adrianwi said:

Why wasn't the thread below addressed in the following 3 years, which may have avoided this situation?

I think the initial assessment of risk and severity was incorrect and further on, the lack of any kind of exploitation might have reinforced that classification.
What probably played a role as well, is that there weren't any precedent incidents of similar impact and severity in the history of Emby.

 

19 hours ago, adrianwi said:

What is being done to make sure things like this can't be missed in the future?

The incident has shown that security related issues need to be treated with highest priority, and some lessons were surely learned.

 

19 hours ago, adrianwi said:

What changes are being made to the plugin system to make this kind of exploit impossible/more difficult in the future?

The vulnerability was not specific to the plugin system, it was about gaining (emby-)administrative access - which would have also allowed to do other things than installing a plugin.

But there are considerations about adding extra guards for changes to "high-risk" configuration settings - both in Emby Server as well as in plugins - which applies to those kinds of settings where code or script execution is controlled. 

 

19 hours ago, adrianwi said:

What changes will be made to emby to improve user security?

A fundamental change in response to this incident is that we are dropping the conceptual distinction between "local network" and "non-local network", while the more challenging part is to provide a similar level of convenience like before but in a secure way.

There are more security-related changes in the works and in planning. These will arrive in subsequent beta releases and will be explained alongside.

Best regards,

softworkz

 

Edited June 23, 2023 by softworkz

[adrianwi 238]

Thanks for the detailed response.  I really do hope security is taken much more seriously and not just dismissed as a user/administrator issue.  Sure, it is ultimately the user/administrator responsibility to secure their server, but emby should be providing tools to help as a matter of priority.  No security suggestions in the forum should go unchecked and ignored for months or in some cases years.

 

 

[TheTabman 4]

On 6/23/2023 at 6:15 PM, softworkz said:

What probably played a role as well, is that there weren't any precedent incidents of similar impact and severity in the history of Emby.

Can we assume that this kind of thinking is no longer in effect? Will instead the Emby team from now on address security issues as soon as they are brought to their attention?

[Gilgamesh_48 943]

9 hours ago, TheTabman said:

Can we assume that this kind of thinking is no longer in effect? Will instead the Emby team from now on address security issues as soon as they are brought to their attention?

In this case I actually think that Emby has learned a valuable lesson. I also hope that they do not over react as other companies have done in the past. There should be tighter security going forward but I hope those us that do not use the remote access features at all will not be too inconvenienced in the long run.

[justinrh 174]

This seems disingenuous.  Shouldn't this say a "few years"?  Besides, no outsider can know in a few days if the developers decide to fix something (if it had just been reported).

 

And, wait, the nature/name of the vulnerability is just now being mentioned on page 18?

 

And what is this about?  It is under the section "Replicating the Hack".  This is not a hack, just poor user credential management.  So what is this telling us besides the login mechanism works.

 

Where does the report tell how the malicious DLLs were installed on the server?  Is this done after doing the header spoof and getting Emby admin access?

Where is the acknowledgment to user pse for reporting the vulnerability three years ago?

Did anyone review this report?  This report needs to be reworked for clarity, organization, and grammatical errors.  (I've never a "cross-size origin" header.  )

[softworkz 3338]

1 hour ago, justinrh said:

This seems disingenuous.  Shouldn't this say a "few years"?  Besides, no outsider can know in a few days if the developers decide to fix something (if it had just been reported).

Section 3.2.4 starts by mentioning the date of registration of the hacker's domain and the date of the forums discussions around our fixing of the vulnerability in the beta.
The "few days" term is referring to a part of the time span between the two events.

 

1 hour ago, justinrh said:

Where is the acknowledgment to user pse for reporting the vulnerability three years ago?

Valid point. I'll add appropriate acknowledgement to the CVE. Thanks.

 

1 hour ago, justinrh said:

[...] grammatical errors.  (I've never a "cross-size origin" header.  )

I've never grammatical error.

[acnp77 61]

I find data privacy and independency very important, and I am not a fan of Emby being able to push packages to my server without me requesting them. And yes, there are still many issues to fix in emby server. But in this case all I can say is thanks a lot for all your work and transparency. CHEERS

[darkassassin07 427]

1 hour ago, acnp77 said:

I am not a fan of Emby being able to push packages to my server without me requesting them.

You and all the rest of us did request new packages be pushed to our systems; by enabling auto-update. That's it's entire purpose and the mechanism that was used here.

 

Same goes for any other software with auto update enabled.

Edited July 10, 2023 by darkassassin07

[Spaceboy 2494]

i'm not certain thats quite true. i believe there is a long reported and still outstanding "bug" that sets emby to auto update following any server update

[ebr 14925]

Automatic updates are a part of app life these days.  They are really necessary to keep systems running properly and to have any chance at all at supporting them.

Our terms of use do state we have the right to modify the software at any time (remember, your license with any software is one of use, not ownership).  This is really necessary for any successful software operation.

[softworkz 3338]

Besides that, no automatic update of Emby Server has been performed.
We didn't update or change anybody's Emby Server installation.
It was an update of a single plugin with a minimal addition to save users' systems.

[acnp77 61]

20 hours ago, darkassassin07 said:

You and all the rest of us did request new packages be pushed to our systems; by enabling auto-update

This is incorrect. I did not enable any kind of auto-update. I am talking about the plugins that you can not prevent from being updated automatically. But I also know that this a necessary mechanism these days. Important here are the intentions of the owner/vendor. In case of emby I am confident the intentions are good.

[acnp77 61]

@softworkz Please excuse the stupid question: Has it been ensured that users do not have the malicious code on their systems anymore? Or in other words: Do users that were not affected due to having pw's set for all admin accounts in LAN and WAN, and therefore were not flaged by emby, need to take any measures? Thanks,

 

[softworkz 3338]

39 minutes ago, acnp77 said:

@softworkz Please excuse the stupid question: Has it been ensured that users do not have the malicious code on their systems anymore? 

First of all, there are still infected systems being detected, but the curve is steadily going down and it seems were close to an end. Last stats (5 days old) were like 30 infections over the preceding 14 days.

As explained, we are not manipulating the file system on machines and we do not remove the malware plugin. What we do is write a message into the log, shutdown Emby and prevent it from being started again without having followed our instructions.
Manipulating users' machines was considered a no-go, possibly even illegal in certain countries and we also didn't want to interfere with or possibly erase any forensic evidence.
Any actions taken to deal with the infection must and can only be conducted by the owner/administrator of the server, so we needed a guaranteed way to gain the attention of the responsible person. Also, we didn't want to give the hackers a chance to continue their plans and adapt to the situation, possibly doing more damage to users' servers as time proceeds.
Shutting down the infected servers was the best way to achieve both of these goals.

 

39 minutes ago, acnp77 said:

 Do users that were not affected due to having pw's set for all admin accounts in LAN and WAN, and therefore were not flaged by emby, need to take any measures?,

Yes. Upgrade to the latest stable version as soon as possible.

Thanks

[acnp77 61]

1 hour ago, softworkz said:

Yes. Upgrade to the latest stable version as soon as possible.

Thanks. Yes, this goes without saying. Done.

I see 4 cases:

#1 Malware plugin not installed, sufficient security config

#2 Malware plugin not installed, insufficient security config

#3 Malware plugin installed, sufficient security config

#4 Malware plugin installed, insufficient security config

 

Security config has been taken care of by your meassures. What about the plugin removal? How do users that were not at risk (#1 or #3) know if they have the plugin installed? What is the name of the plugin? Is simply removing it enough?

 

 

[softworkz 3338]

14 minutes ago, acnp77 said:

What about the plugin removal?

14 minutes ago, acnp77 said:

What is the name of the plugin? Is simply removing it enough?

The steps to be taken are described here:

Advisory: https://emby.media/support/articles/advisory-23-05.html

15 minutes ago, acnp77 said:

How do users that were not at risk (#1 or #3) know if they have the plugin installed?

The server would shut down in case of infection.

[rbjtech 4284]

Is there an ETA for some of the basic security improvements ?

I believe Emby advised aftert the incident that things like brute force lockout and password strength were now a high priority ?

I removed Scripter-X from my install as part of the initial incident investigation - but I used it to log all 'access' to a file which was then scraped for IPBan.   So this hasn't been functional since the incident back in May...

Ideally, I'd like emby to do this internally - but if it's not coming within a month or so, then I'll have to organise alternative arrangements as the login process for emby remains extremly weak...

Thanks.

[Luke 37099]

4 hours ago, rbjtech said:

Is there an ETA for some of the basic security improvements ?

I believe Emby advised aftert the incident that things like brute force lockout and password strength were now a high priority ?

I removed Scripter-X from my install as part of the initial incident investigation - but I used it to log all 'access' to a file which was then scraped for IPBan.   So this hasn't been functional since the incident back in May...

Ideally, I'd like emby to do this internally - but if it's not coming within a month or so, then I'll have to organise alternative arrangements as the login process for emby remains extremly weak...

Thanks.

Lockouts after several consecutive failed password attempts are coming soon.

[darkassassin07 427]

10 hours ago, rbjtech said:

I removed Scripter-X from my install

This is unnecessary. Scripter-X was not used to gain access, nor is it compromised in some way; it was just a tool used once administrative access had already been gained.

If a future hacker gained access (through some other means, the hole used here is closed) and wanted to use scripter-x in a similar way, they'd just install it just as they did here.

[rbjtech 4284]

5 hours ago, darkassassin07 said:

This is unnecessary. Scripter-X was not used to gain access, nor is it compromised in some way; it was just a tool used once administrative access had already been gained.

If a future hacker gained access (through some other means, the hole used here is closed) and wanted to use scripter-x in a similar way, they'd just install it just as they did here.

Yep, aware of that  - as per my post, I had already installed Scripter-X for other duties (producing dedicated activity logs) and the plugin also has a questionable function of deploying 'packages' from a remote site that may also have been compromised as part of the incident - at that point in time (working with @softworkzbefore it was publicaly released as an incident) - it was wise to remove all forms of potential entry/risk.  None of my systems were ever compromised - this was all preventative.

imo the scripter-x package deploy feature should NOT be allowed by emby - or restricted as much as they can - as this is then allowing this sort of thing to potentially happen again.   'Plugins' that do 'Administrative' changes need to be much better controlled - but I suspect that needs many more emby framework changes as there is only 1 level of 'Admin' atm.  Some of this work has already started as I'm involved with a few plugins myself.  

@LukeDo you have a better ETA than 'coming soon' - as that could be next week or it could be next year ...   This month or next month would suffice .. thanks.

 

Edited August 1, 2023 by rbjtech