FULL DISCLOSURE: Data Collection in the Process of BotNet Takedown

[ember1205 23]

Just now, Racerprose said:

It seems like you are more upset with the inconvenience of your server being turned off than your system being compromised.

I agree only in that I do not think they went far enough to getting everyone notified about what was happening. I think they acted appropriately in turning off the systems affected. This happens all the time in other compromised situations where you may be temporarily logged out of your account whereby you will need to reset credentials.

No... I'm mad that they believe that turning my server off served was completely within their right to do AND served as sufficient "notification" of this issue. The fact that I even FOUND the information posted on this site was completely by luck and the direct result of being fortunate for deciding to search for a very specific error message that I was able to locate in my logs only when I was repeatedly trying to start the server up.

With regard to being logged out of compromised accounts... that's happened. And do you know how I know? I get an email telling me what happened, telling me my account needs its password to be reset, and a link to go do it.

Even with the banner here, the directions were incomplete, had directions to do things that were actually impossible, and had steps in the wrong order. I'm a pretty seasoned Linux admin with a fairly elaborate home lab and it took me almost an hour to bring my server back on line with the directions they provided.

[Racerprose 23]

1 minute ago, ember1205 said:

No... I'm mad that they believe that turning my server off served was completely within their right to do

But it was their right. Its their server software you are using. Just like your ISP can turn off your internet if they so chose. You do not own it, you are simply using it. And just because you are paying for it doesn't mean you own it.

By turning off the affected systems they were able to limit the spread. Again it sounds like you are just more annoyed they have the power to turn off servers in special situations like this. I don't really understand the logic of allowing the systems to go on being compromised and letting the server owners decide what to do. I'm fairly certain most would turn off their server and clean the infection and process the update file.

[softworkz 3335]

8 minutes ago, ember1205 said:

No... I'm mad that they believe that turning my server off served was completely within their right to do AND served as sufficient "notification" of this issue. The fact that I even FOUND the information posted on this site was completely by luck and the direct result of being fortunate for deciding to search for a very specific error message that I was able to locate in my logs only when I was repeatedly trying to start the server up.

With regard to being logged out of compromised accounts... that's happened. And do you know how I know? I get an email telling me what happened, telling me my account needs its password to be reset, and a link to go do it.

Even with the banner here, the directions were incomplete, had directions to do things that were actually impossible, and had steps in the wrong order. I'm a pretty seasoned Linux admin with a fairly elaborate home lab and it took me almost an hour to bring my server back on line with the directions they provided.

I understand your position. I can assure you that the decision hasn't been made light-heartedly. It had been clear to us that this kind of intervention will be seen controversial, but there were reasons for why we chose to go that far and shutdown infected servers as the ultimate last resort.

I wrote an article which is detailing the decision process, I hope it will be published soon.

[rstat1 1]

I am in no way a fan of how this was handled. My server was not compromised, and yet for some reason I find myself locked out of my admin account because Emby for some strange reason has the ability to change my password, as well as other settings on an instance that is self-hosted. I as the ADMIN, should be the only one with those abilities on a SELF-HOSTED server.

My only question is how do I block or otherwise stop such interference from occurring in the future? This is not OK.

[Gilgamesh_48 943]

30 minutes ago, ember1205 said:

Let me put this a different way... What the hell right does the Emby team have to shut down my server? I understand that their "hearts were in the right place", but that is 100% -MY- property and they have absolutely no right to do ANYTHING to render it inoperable. EVER. I paid for a lifetime Premiere license. So long as I stay within what I am licensed for and licensed to do, NO ONE has a right to touch my property in any way. Do I -want- to be participating in a botnet? No. But, instead of writing code and pushing it to my server to crash it, they should have actually NOTIFIED ME of the issue and let ME decide how and when to deal with it.

I mostly agree with you statements but there is one thing: (I am NOT defending Emby by this statement it is just a fact,)
It is not "your" server. It is Emby that owns the server software. You own the hardware and, probably, you own you media. But you do not own the server software. You simply have license to use it. Just like almost every piece of software the exists. I do not know exactly where the ownership of operating systems like Linux (BTW: open source does mean the there are not owners out there.)
Microsoft owns Windows and Apple owns whatever the call their OS at this time and so on.

Even "purchasing: a lifetime pass/license does not convey ownership.

If Emby had done something to render your server hardware inoperable then you would have a point but all they have done is turned off their service. Sometime people need to read license agreements and privacy statements. It would amaze people just exactly how much they don't own of what they think they bought.

I have not read every document associated with the software I use but I just assume that I own no software I use, I just control it, most of the time. The only software I am pretty sure I own is software I wrote and then many compilers have clauses, usually unenforced, that say that anything written using that compiler is owned by the company that wrote the compiler. Of course that would be extremely hard to enforce.

Is this "right"? I do not believe it is BUT I also believe that situations like we have with Emby it is expected that they continue to maintain and further develop it and for that to happen Emby must have a way to force servers offline to protect the users or Emby itself.

Could this have been handled better? Of course it could! But, at this point, what they have done with and to their server.

My biggest hope at this point is that Emby does not go overboard and make my use of Emby harder.

[andrewds 93]

2 minutes ago, rstat1 said:

My only question is how do I block or otherwise stop such interference from occurring in the future? This is not OK.

There are solutions that will allow you to island Emby Server and truly deny it any access beyond your network, but the immediately obvious trade-off will be that the Emby DRM features will no longer work and therefore your premiere key, if you are a premiere subscriber, will no longer validate and therefore premiere features will be unavailable to you.

[softworkz 3335]

26 minutes ago, rstat1 said:

I am in no way a fan of how this was handled. My server was not compromised, and yet for some reason I find myself locked out of my admin account because Emby for some strange reason has the ability to change my password, as well as other settings on an instance that is self-hosted. I as the ADMIN, should be the only one with those abilities on a SELF-HOSTED server.

I can assure you that your password hasn't been changed.

Two adjustments were made:

• If "don't require password in local network" was enabled for any admin account, it was disabled
  This option will be removed in future Emby Server versions - at least for admin accounts
• If there were admin accounts without password, those accounts were changed to not be shown on logon pages
  No change to the password, though
  In future Emby Server versions, admin accounts without password will no longer be allowed

These were exactly those two ways how the hackers gained access and we did only that to secure all servers from getting infected.

[andrewds 93]

2 minutes ago, softworkz said:

I can assure you that your password hasn't been changed.

Two adjustments were made:

• If "don't require password in local network" was enabled for any admin account, it was disabled
  This option will be removed in future Emby Server versions - at least for admin accounts
• If there were admin accounts without password, those accounts were changed to not be shown on logon pages
  No change to the password, though
  In future Emby Server versions, admin accounts without password will no longer be allowed

These were exactly those two ways how the hackers gained access and we did only that to secure all servers from getting infected.

Thanks @softworkzfor the explanation. To confirm then you don't have any evidence of a brute force attack against a properly passworded admin account or an attack leveraging a weak password? Just having a password on the admin account doesn't eliminate the risk it just makes the attack more difficult.

[softworkz 3335]

34 minutes ago, rstat1 said:

I as the ADMIN, should be the only one with those abilities on a SELF-HOSTED server.

You can't imagine how much I hated having to do it this way. It regularly upsets me that Chrome, Edge and Firefox are installing Scheduled Tasks on Windows which allow them to execute things with elevation. Same goes for all other updates which install without asking for consent.

The problem in this case was that it had to happen all at once for all Emby servers in the world. That was the only way to take that BotNet down - by giving them no time to adapt and take measures.

34 minutes ago, rstat1 said:

My only question is how do I block or otherwise stop such interference from occurring in the future? This is not OK.

You can disable automatic restart after plugin updates.

[softworkz 3335]

3 minutes ago, andrewds said:

To confirm then you don't have any evidence of a brute force attack against a properly passworded admin account or an attack leveraging a weak password?

No such evidence.

The failed login attempts that can be seen in some logs were intentional, because they used ScripterX to execute actions (before their own dll was in place). And they used the ScripterX action "OnFailedAuthentication" - this means, they already had admin access and generated failed logins only to trigger the ScripterX actions.
This had confused me a bit initially  

[rstat1 1]

17 minutes ago, softworkz said:

I can assure you that your password hasn't been changed.

Two adjustments were made:

• If "don't require password in local network" was enabled for any admin account, it was disabled
  This option will be removed in future Emby Server versions - at least for admin accounts
• If there were admin accounts without password, those accounts were changed to not be shown on logon pages
  No change to the password, though
  In future Emby Server versions, admin accounts without password will no longer be allowed

These were exactly those two ways how the hackers gained access and we did only that to secure all servers from getting infected.

Are you sure? I have an Activity Log entry that says otherwise, it was not me who changed it, and this server is not accessible to anyone outside of my local network (you'll get a nice big "You've been blocked" error page if you try).

The log entry timestamp matches when I tried to log in which is what lead me to believe it was reset by someone/thing that was not me.

 

EDIT: If I may suggest only disabling the password free local logins if a user has remote access enabled.

Edited May 27, 2023 by rstat1

[softworkz 3335]

1 hour ago, Racerprose said:

But it was their right. Its their server software you are using

I totally disagree that it is "our right" to do such things in general. We have no right to decide when servers start or shutdown.

What I think is right, though, is when after a software update, the server doesn't start anymore while a malware is present.

[softworkz 3335]

6 minutes ago, rstat1 said:

EDIT: If I may suggest only disabling the password free local logins if a user has remote access enabled

That "remote access" enablement is not a precise discrimination. It's possible to allow remote access (through proxy or VPN) even without "enabling remote access".

[andrewds 93]

8 minutes ago, rstat1 said:

EDIT: If I may suggest only disabling the password free local logins if a user has remote access enabled.

This exploit would have worked even if remote access were disabled in the server settings, and on users with remote access disabled.

Generally I like this idea though.

[softworkz 3335]

8 minutes ago, rstat1 said:

Are you sure? I have an Activity Log entry that says otherwise, it was not me who changed it, and this server is not accessible to anyone outside of my local network (you'll get a nice big "You've been blocked" error page if you try).

The log entry timestamp matches when I tried to log in which is what lead me to believe it was reset by someone/thing that was not me.

When it would have been done by our update. then you would have seen one of those messages on the dashboard:

[rstat1 1]

Just now, softworkz said:

When it would have been done by our update. then you would have seen one of those messages on the dashboard:

I only had the 2nd Alert. Not the first. My admin account already had a decently secure password on it. My only issue here, is that it was randomly reset by NOT me, which would've locked me out entirely had I not also been logged in on my phone.

[softworkz 3335]

3 minutes ago, andrewds said:

This exploit would have worked even if remote access were disabled in the server settings, and on users with remote access disabled.

Generally I like this idea though.

The whole feature complex around discriminating local network vs. non-local network is flawed by design.

We will probably get away from this. Passwordless login is something that needs to be handled by the clients, not by compromising server security.

[softworkz 3335]

2 minutes ago, rstat1 said:

I only had the 2nd Alert. Not the first. My admin account already had a decently secure password on it. My only issue here, is that it was randomly reset by NOT me, which would've locked me out entirely had I not also been logged in on my phone.

I think when it would have come from our update, then you wouldn't be the only one to report it, because this has been applied to a really large figure of servers.

Edited May 27, 2023 by softworkz

[rstat1 1]

5 minutes ago, softworkz said:

I think when it would have come from our update, then you wouldn't be the only one to report it, because this has been applied to a really large figure of servers.

I haven't done the server update yet. Only updates I've gotten recently are for plugins.

Which if plugins can change privileged server settings and user passwords, than that's a much bigger issue.

[Gibberish 1]

If they're going to require a password for local network admin accounts, they should add another user option to allow the refreshing of metadata, especially episode metadata. Not editing, just refresh. That'll probably take care of most people's need to be an administrator on their daily use account and make it more household friendly to put.

[Painkiller8818 203]

It's a bit funny how people are complaining about Emby trying to find a Solution to this situation, and they needed to decide quick this is nothing they had time to plan it over weeks.

And now people are complaining about turning off their servers and write a info in the emby server log, because they are not able to notify everyone per mail because free users don't need to provide any contact informations.

Think about it had to happen fast, and this is the way they decided to do it.

From my perspective as an IT Guy, i know you need to do things really quick sometimes and normally if the emby server is not starting up anymore, i look at the emby logs, server logs or go to their support site, so this is where people will find the information.

But as i know there are so many people with that less tech knowledge we even could be happy they are able to turn on their computers every day, they now start to complain about it.

I would like to see your posts if all your files has been gone because the emby devs wouldn't have done anything and all your files got deleted.

I am pretty sure, if this would happen, it would be more than enough for you to prevent it by stopping your server from starting up.

 

So think about you haven't lost any data and now life goes on instead of you have 0 movies, shows and music at all because they just don't react because "IT IS YOUR SERVER AND THEY HAVE NO RIGHT TO DECIDE WHAT HAPPENS TO YOUR STUFF"


I really would see what you guys would have done in this situation and if the first users start to flood your forum because all files are gone 

Then they can give you the answer you wanna hear from them now: It is your server, we are not responsive for your files and the security on your server. 

Edited May 27, 2023 by Painkiller8818

[Dickydodah! 128]

There is a saying that you can't please all of the people all of the time and this is a good example of that. I have had to deal with similar security issues in systems much larger and somewhat more important than a media server and the potential for a witch hunt after the system has been protected is always there. I always found that in the commercial world the customer always wanted a witch hunt and someone to blame but in the sector I dealt with they were much more prosaic as they had much more important sh1t to deal with .

I'm sure the Emby team will have a good washup after this and hopefully have an AAR (after action report) with recommendations of how to deal with this sort of situation in the future. I don't expect full disclosure of this report as it should be treated as "commercial in confidence" and kept away from the prying eyes of the hackers who find this sort of thing fun. It would be nice to have an outline policy published which could/should form part of the conditions of use of the software.

Considering that 1200 systems were found to be affected and the number of complaints is only 1 - 2% of that figure I would consider this a JOB WELL DONE  

Some big lessons to be learnt and I'm sure they have been 

 

I do have a general question for all. What was the point of this attack? was it just a "I can so I will" by some idiot who thinks it's fun or was there actually any gain to be had?

[pwhodges 1531]

The most likely theory I've seen expressed is that this attack was building a botnet for later exploitation, perhaps initially being sold on to more directly malignant actors.  This is supported by the lack of any evidence (so far) of effects that reached beyond Emby itself.

Paul

[Dickydodah! 128]

I suppose we can all at least be satisfied that all the effort put in by the hacker was wasted  Still a PITA though.

[softworkz 3335]

13 minutes ago, pwhodges said:

This is supported by the lack of any evidence (so far) of effects that reached beyond Emby itself.

...and by the APIs and functionality that the malware exposes. It doesn't do anything by itself, but you can do everything with it when you have control, also it hides itself and has functions to hide its activities and clean the logs from it in a targeted way.