FULL DISCLOSURE: Data Collection in the Process of BotNet Takedown

[darkassassin07 432]

11 minutes ago, Painkiller8818 said:

Because even if you have a Password and someone really wants to hack you, this is really simple with bruteforcing a PIN or using a dictionary attack for more complex passwords. Using the GPU for this will allow to try 60K+ passwords a second and having wordlists with 20GB+ can have a lot passwords.

I agree with the sentiment; just wanted to note this kind of speed would require direct access to the stored passwords to try against directly. Ie, the password db has already been leaked from the machine in its encrypted form and is being worked on seprate from the web server being attacked. Once carcked, taking those passwords to the web server and logging in.

 

You'd be vastly limited by how quickly the web server will process login attempts otherwise and you'd constantly be throwing 'failed login' errors in the dashboard. You may still get in, but it'll take much much longer than '60k+/second' and an admin will more than likely notice by then.

 

Rate limiting failed login attempts is still a good idea to increase this time further though.

Edited May 29, 2023 by darkassassin07

[Painkiller8818 203]

6 minutes ago, darkassassin07 said:

You'd be vastly limited by how quickly the web server will process login attempts otherwise and you'd constantly be throwing 'failed login' errors in the dashboard. You may still get in, but it'll take much much longer than '60k+/second' and an admin will more than likely notice by then.

 

yeah you're right, i just took the same speed like for hacking wifi passwords

[CharlieMurphy 73]

1 hour ago, Painkiller8818 said:

Because even if you have a Password and someone really wants to hack you, this is really simple with bruteforcing a PIN or using a dictionary attack for more complex passwords. Using the GPU for this will allow to try 60K+ passwords a second and having wordlists with 20GB+ can have a lot passwords.

I've considered the brute force possibility before any of this happened, but they would also have to guess the usernames. In the process of guessing it would would be apparent from the Emby dashboard. In all my years with Emby I've never had any attempts that weren't me so far.
But I agree, the brute force potential is there and I assume we could use fail2ban if someone were trying to exploit it. 

Edited May 29, 2023 by CharlieMurphy

[darkassassin07 432]

23 hours ago, Gibberish said:

 

• it seemed like a waste of resources to have the reverse proxy constantly working for the LAN Emby clients.
• If something happens to the reverse proxy server, the family still has access to the Emby server at home without changing the server address.

I've just never really been concerned about resource consumption, I have plenty of overhead space.

 

The second point I actually see the other way: if the main server machine is offline for whatever reason, the proxy serves a 'service unavailable, please try again later' page vs just timing out the connection. (same for the rest of my services as well) let's me know just the emby server is offline vs a much larger issue.

 

 

We've definitely all got some interesting differences in our setups.

[rbjtech 4304]

4 hours ago, Painkiller8818 said:

One important question, so lets say you have a password/PIN protection, and a hacker tries to gain access, does emby has the ability for a lockout policy? EG 3 failed tries in 5 mins and locking the account for x mins or permanently?


Because even if you have a Password and someone really wants to hack you, this is really simple with bruteforcing a PIN or using a dictionary attack for more complex passwords. Using the GPU for this will allow to try 60K+ passwords a second and having wordlists with 20GB+ can have a lot passwords.

So maybe some kind of lockout policy for security would be great.

Thanks

So short answer is no - emby currently has no modern password management at all.  Not complexity/entropy check, nor any form of lockout.   Anybody with any cyber security knowledge is using reverse proxies with fail2ban - these can also be used on the emby web-server - I believe @chefeven developed one as a Plugin.

Remember if you use http, then the password is transmitted in the clear any way, so it doesn't really matter how strong it is, it can be snooped.  Also remember the recent vulnerability didn't even need a password.

As has been said many many times, the security basics need addressing, then you can start adding layers of security on top.

I'm unsure where basic password management is on the list, but I suspect its now been promoted to near the top ..

Edited May 29, 2023 by rbjtech

[darkassassin07 432]

44 minutes ago, rbjtech said:

I'm unsure where basic password management is on the list, but I suspect its now been promoted to near the top ..

I'd certainly hope an audit of embys security practices is underway given recent events...

[justinrh 175]

21 hours ago, arrbee99 said:

I can even remember it - so far.

How about today, @arrbee99?

[nuke11 21]

3 hours ago, rbjtech said:

So short answer is no - emby currently has no modern password management at all.  Not complexity/entropy check, nor any form of lockout.   Anybody with any cyber security knowledge is using reverse proxies with fail2ban - these can also be used on the emby web-server - I believe @chefeven developed one as a Plugin.

Remember if you use http, then the password is transmitted in the clear any way, so it doesn't really matter how strong it is, it can be snooped.  Also remember the recent vulnerability didn't even need a password.

As has been said many many times, the security basics need addressing, then you can start adding layers of security on top.

I'm unsure where basic password management is on the list, but I suspect its now been promoted to near the top ..

What I would like to have looked at is how plug-ins are loaded. Since I have not read a full report of what happened, the ease at which a non-approved plugin was loaded is a major problem.

I have no idea of the environment in Emby for creating plugin's, but maybe a revamp of that sub-system might be in order?  Are Emby plugins digital signed at all by Emby LLC? Is there a way to prevent un-signed plugins from being loaded without a special developer password to enable that ability on the server? This doesn't have to be an Emby LLC managed password, but a password everyone sets up on there own server that is only used if they develop plug-ins and are testing.

Edited May 29, 2023 by nuke11

[arrbee99 1561]

29 minutes ago, justinrh said:

How about today, @arrbee99?

Yep, its still in there. Wow, I'm really on a roll...

[justinrh 175]

On 5/26/2023 at 7:08 AM, softworkz said:

we used some auto-updating capabilities built into Emby Server, and so it was your server who initiated the connection to report back the information we needed.

I wanted to make sure I understood how you knew how many machines were potentially part of a botnet.  It is because of the logs sent from the users' servers, right?  If you received 100 logs with EmbyHelper.dll in them, then you could say that 100 servers were part of the botnet, right?

And, if I followed this thread correctly, the 'update' did its work after detecting the hack w/o any round trips to you (Emby HQ) - it was a single delivery to the user's machine to do all the work needed.  It was not like: user's server does a 'check for updates', update code sends a report to Emby HQ, then Emby sends a script to shutdown server with updated log, right?

 

[softworkz 3341]

11 minutes ago, justinrh said:

It is because of the logs sent from the users' servers, right?

No logs were sent. It was just that single piece of information which is documented in the first post of this conversation.

[justinrh 175]

21 minutes ago, softworkz said:

No logs were sent. It was just that single piece of information

Right - in the context of this thread/issue, that could be considered a log file to you.  I did not mean the user's server log file.

No comment on the other questions?  Then who can answer?

Edited May 30, 2023 by justinrh

[justinrh 175]

@Lukeplease update the advisory to be easier to digest and less scatter-brained (sorry, only description I can think to use right now, not meant to be derogatory).  It is illogically written and does not inspire confidence for the system cleanup.

[softworkz 3341]

25 minutes ago, justinrh said:

And, if I followed this thread correctly, the 'update' did its work after detecting the hack w/o any round trips to you (Emby HQ) - it was a single delivery to the user's machine to do all the work needed.  It was not like: user's server does a 'check for updates', update code sends a report to Emby HQ, then Emby sends a script to shutdown server with updated log, right?

Correct. There was never any direct interaction with a user's server. 

There were two phases: The first one on Mon/Tue. This was purely passive without any actions. It served for us to get a clear picture how many machines are affected and also as a "dry-run" to see whether the detection works and make sure that no unnecessary shutdown would be made.
After that phase we adjusted the detection (because we realized that in most cases the plugin hid itself - opposed to the analyzed case). We also saw that in half of the cases, an empty admin password allowed them to get in while we had previously assumed that it's always the "don't require password in local network" setting.

In the second phase there was the same reporting as previously (one report after server start) plus another report right in the moment of shutdown or security tightening. In all cases it's been the same report like described in the first post.

Edited May 30, 2023 by softworkz

[softworkz 3341]

5 minutes ago, justinrh said:

@Lukeplease update the advisory to be easier to digest and less scatter-brained (sorry, only description I can think to use right now, not meant to be derogatory).  It is illogically written and does not inspire confidence for the system cleanup.

And it exactly shall not do that: "inspire confidence".

The conclusion must be that you either analyze your server in an exhaustive way - until you are confident
or you decide to re-install.

A malware infection is a serious incident and spreading confidence is the most inappropriate thing we could do in that situation.

It's not like "haha, we shut your server down and here are the instructions to get it start again". We did the shutdown because the situation is that serious. And when you are lacking confidence in the security of your server, then that's good and adequate to the situation.

[rdhardi 19]

2 hours ago, nuke11 said:

What I would like to have looked at is how plug-ins are loaded. Since I have not read a full report of what happened, the ease at which a non-approved plugin was loaded is a major problem.

I have no idea of the environment in Emby for creating plugin's, but maybe a revamp of that sub-system might be in order?  Are Emby plugins digital signed at all by Emby LLC? Is there a way to prevent un-signed plugins from being loaded without a special developer password to enable that ability on the server? This doesn't have to be an Emby LLC managed password, but a password everyone sets up on there own server that is only used if they develop plug-ins and are testing.

 

2 hours ago, nuke11 said:

What I would like to have looked at is how plug-ins are loaded. Since I have not read a full report of what happened, the ease at which a non-approved plugin was loaded is a major problem.

I have no idea of the environment in Emby for creating plugin's, but maybe a revamp of that sub-system might be in order?  Are Emby plugins digital signed at all by Emby LLC? Is there a way to prevent un-signed plugins from being loaded without a special developer password to enable that ability on the server? This doesn't have to be an Emby LLC managed password, but a password everyone sets up on there own server that is only used if they develop plug-ins and are testing.

This is after 3 MovieDb updates last week, the last on the day of the red banner announcement:

MovieDb Updated to 1.6.6
5/25/2023 11:35:36 PM

MovieDb Updated to 1.6.5
5/24/2023 11:35:36 PM

MovieDb Updated to 1.6.4
5/23/2023 12:27:22 PM

Wondering now, exactly why were so many update days apart, and what exactly are these "Compatibility Updates"? Is there somewhere I can read a changelog to see exactly what's going on? I'm pretty hesitant to reboot and finish installing this newest MovieDb update because I genuinely don't understand what it's doing. I don't typically blindly do something I don't understand, guess I just trusted Emby. But, I really need to understand and educate myself on plug-ins before I do another reboot to "Please restart the server to finish applying updates." Down another rabbit hole for me...

Edited May 30, 2023 by rdhardi
remove double quote

[justinrh 175]

Softworkz, I (the hacked user) needs confidence in what I am reading regarding how to clean up my system according to what you (Emby) knows about the malware and get past the Emby-imposed prevention of my server running.  I suppose 'cleanup' was not a good choice of words; maybe 'restore my Emby server' would be a little better.

And I apologize for not being more clear.

Edited May 30, 2023 by justinrh

[rdhardi 19]

Ok, let me try this again lol. Tried to edit and remove double quote, guess I waited too long. Anyway, here's my original reply:

I was not affected by the hack. But the fact that a plug-in was used made me realize that I don't really understand how plug-ins work or how they're installed. Just now, I checked my dashboard, and I see this:

This is after THREE MovieDb updates last week, the last on the day of the red banner announcement. I thought it was odd, so many in a week, but I rebooted to finish installs. Wondering now, exactly
why there were so many in a week and what exactly are these "Compatibility Updates" doing? Is there somewhere I can read a changelog to see exactly what's going on? I'm pretty hesitant to reboot and finish installing this newest MovieDb update because I genuinely don't understand what it's doing. I don't typically blindly do something I don't understand, guess I just trusted Emby. Now, I really need to understand and educate myself on plug-ins before I do another reboot to "finish installing". Down another rabbit hole for me...

MovieDb Updated to 1.6.6
5/25/2023 11:35:36 PM

MovieDb Updated to 1.6.5
5/24/2023 11:35:36 PM

MovieDb Updated to 1.6.4
5/23/2023 12:27:22 PM

 

[softworkz 3341]

4 minutes ago, justinrh said:

Again, I think you are missing the context, softworkz.  I (the hacked user) needs confidence in what I am reading regarding how to clean up my system according to what you (Emby) knows about the malware and get past the Emby-imposed prevention of my server running.  I suppose 'cleanup' was not a good choice of words; maybe 'restore my Emby server' would be a little better.

The problem here is that the malware is just opening doors wide for the hackers to do what they want but the code doesn't include any specific actions (except saving off login credentials).

We just don't have the slightest idea whether and what the hackers might have done or not. They could have done X on one server, Y on another server and nothing on once another server.
That's why we can't say anything specific. Also we did not scan servers for possible impacts - because digging around in users' data is absolutely none of our business.
For those reasons, it is inevitable that users take action and find out what might have been done and take measures to become secure again - eventually, a full re-install is the most secure option you have.

[softworkz 3341]

8 minutes ago, rdhardi said:

Ok, let me try this again lol. Tried to edit and remove double quote, guess I waited too long. Anyway, here's my original reply:

I was not affected by the hack. But the fact that a plug-in was used made me realize that I don't really understand how plug-ins work or how they're installed. Just now, I checked my dashboard, and I see this:

This is after THREE MovieDb updates last week, the last on the day of the red banner announcement. I thought it was odd, so many in a week, but I rebooted to finish installs. Wondering now, exactly
why there were so many in a week and what exactly are these "Compatibility Updates" doing? Is there somewhere I can read a changelog to see exactly what's going on? I'm pretty hesitant to reboot and finish installing this newest MovieDb update because I genuinely don't understand what it's doing. I don't typically blindly do something I don't understand, guess I just trusted Emby. Now, I really need to understand and educate myself on plug-ins before I do another reboot to "finish installing". Down another rabbit hole for me...

MovieDb Updated to 1.6.6
5/25/2023 11:35:36 PM

MovieDb Updated to 1.6.5
5/24/2023 11:35:36 PM

MovieDb Updated to 1.6.4
5/23/2023 12:27:22 PM

Some of the security tightening actions for not-yet-updated servers have been removed.

@Lukemust explain that -  I can't. 

[darkassassin07 432]

I've gotta say I too would like more detail about those particular updates. I can understand not revealing what you were doing at the time to prevent the attacker catching wind and acting against your detection; however now that the cats out of the bag so to speak, what exactly did those updates change/update.

 

'compatibility update' is rather unhelpful info. 

 

Still awaiting the Emby teams detailed report about this incident though.

[softworkz 3341]

I was referring to the updates made afterwards.

Besides that I have answered all questions and "revealed" everything. 

[softworkz 3341]

I can only recommend skimming through all the answers I have given in this conversation already. 
Please understand that I can't repeat things everyday.

When you have a very specific questions though, you're welcome to ask.

[softworkz 3341]

Just go here https://emby.media/community/index.php?/profile/20815-softworkz/content/

Find Thursday noon and from there on go backwards (in the list - which is forward in time).
This should get you a pretty good picture without distraction.

[darkassassin07 432]

I've been following all the threads surrounding this issue for the last 5 days or so, I have read and understand your explanations.

 

The bit that has me a little confused is the number of updates vs the number of actions you've taken. I'm counting 5 'MovieDB' updates in my alerts (1.6.4 through 1.6.7 , 1.6.5 installed twice 28hr apart), each labeled 'compatibility update', as well as a server update.

 

In which updates are you probing servers for information (getting servers to reach out to you, same difference)? How many times did that actually happen? Which of those was actually something to do with themoviedb? What actually changed to makes things 'compatible'? What's with the duplicate version number imstall? The server update was obviously the main security patch; what's going on with the two moviedb updates shortly after? (may 26th and 29th)

When it comes to updates in general, 'compatibility update' isn't very descriptive to begin with, but this incident has also made that specific term/description untrustworthy to me. When I see an update with that description now, I pause and wonder what may actually be in it, both because of the lack of detail as well as the association. I can't help but question them now.

TBH your comment above my last one makes me wonder more. If you're pushing updates to remove 'security tightening actions' why don't the update descriptions just say that? Eager for more info about that from @Luke

This was a rare incident and I've never seen you have to take such... Unprecedented? Actions. I don't feel you abuse this power or take it lightly; but that shadow of a doubt is going to last.