FULL DISCLOSURE: Data Collection in the Process of BotNet Takedown

[sross44 233]

I appreciate the quick fix (if it really wasn't a 3 year old exploit that you all have known about). However, a simple email to all addresses that you did have would have been the right course of action. A simple, we are aware and are working to fix this and if you have questions here's a link to the forum post where our staff can best answer your questions would have been great. 

It does make people antsy and I guess that's where everyone is going with this. You can't please everyone and you all made a decision that you thought was in everyone's best interests. Right or wrong you stand by it, learn from it and move on. More communication with your customer base is never a bad thing. You have a great product... broadcast that when everything is going great and broacast it as well when things are broken. People value transparency more than anything. 

[Ikario 38]

We agree to disagree, there's tons of people that could've simply updated their system and be done with it.  The fact that you are flooded with requests for help right now is because of the specific approach you took to fixing this (breaking people's installs that they cannot fix on their own) not because of the issue per se. You did not inform all affected people, only the ones that noticed their system was broken. That is not the same and it's not what I would consider a directed approach.

 

And don't get me started on the fact you very poorly chose to inform this at the beginning with a post saying how you "took down a botnet in 60 seconds ". That is in terribly poor taste. Not even an apology.

[ember1205 23]

I see a lot of deflecting, a lot of explanation, and a lot of justification. What I don't see is any sort of acknowledgement that anything is being done to improve the process. Combine that with the "marketing spin" being put on this and leaves a VERY bad taste in the mouths of users.

As has been mentioned, Emby did not "take down a botnet", it took down a specific number of servers around the world that were participating in a botnet. It is also massively misleading to claim that it was taken down "in 60 seconds" - while that may have been how long it took to post the plugin update and begin the process of pushing it out, it took TWO WEEKS from the initial hack, and more than three years since the vulnerability was exposed, before anything was "done." Further, how many servers around the world are still running affected code and have been compromised? To think that there are a whopping 1200 servers GLOBALLY that were impacted by this is not at all reasonable.

I came to Emby because the front-end functionality was significantly more reliable than that of Plex. Due to Emby being insanely slow to implement very basic, repeatedly asked for features on the back end, I have continued to run my entire Plex setup solely for the purpose of the DVR functions. It looks like I will be needing to entirely rethink my entire setup once again because this overall response it not at all well-seasoned for a commercial entity.

[MBSki 1016]

What is the 3 year exploit that everyone is talking about?

[Guest]

Just now, MBSki said:

What is the 3 year exploit that everyone is talking about?

This

[ember1205 23]

26 minutes ago, ebr said:

No what I'm saying is informing and directly protecting the right people was our top priority. 

The problem is that you informed no one. Posting a banner on the web site of the forums is not notification any more than my stapling my resume to a utility pole would count as notifying every local business that I am looking for work.

[Dreakon13 132]

The fact this was reported years ago and wasn't resolved before it became a big problem, they deserve some egg on their face for that.

The rest of this is really just a lot of hair splitting, salivating and hand wringing over the potential for a public apology/shaming from people who haven't had their favorite feature request implemented yet.  Notifying the effected servers through their logs and putting them in a position where their server is safe but they have to seek out more information was a fine response to a situation with no good answer.

Edited May 26, 2023 by Dreakon13

[Painkiller8818 203]

Maybe this will help getting more security now like the for years open request for 2FA/MFA instead of getting all time the same answer "good idea for the future, thanks"

[ebr 14913]

18 minutes ago, ember1205 said:

The problem is that you informed no one.

But we did.  Affected systems were informed via their server logs and the fact that their server would not run.  A link to the same information on this site banner was in every one of their logs but we realized many people would not think to look at their logs so also put it out here - where we knew people would come when their server couldn't run.

In the end, we did what we thought was the best thing we could possibly do in the situation with the resources we have.  No doubt mistakes were made and we are now trying to do our best to deal with them.

[Painkiller8818 203]

2 minutes ago, ebr said:

In the end, we did what we thought was the best thing we could possibly do in the situation with the resources we have.  No doubt mistakes were made and we are now trying to do our best to deal with them.

Thanks for trying to reach out everyone, i know this is a shitty situation and you are trying to do your best to inform everyone.

Just as an idea for people not visiting the forum.

Do you have any way to send out an update in the server overview like this one:



Maybe with the red warning icon, this is something everyone should see, maybe by updating another plugin and trigger such an event in addition to the update event with a warning message about the problem.

 

but this is just an idea.

Thanks for your work

[Ikario 38]

That's the point, the "oh well, what's done is done" attitude is no good because you can STILL inform via e-mail to everyone that might be affected but have no idea  what's going on. There's still stuff you can do to improve the situation, you are just refusing to do it.  Now that the main wave has passed would be a good time to let people know if, as you said, wanted to help people directly affected but not be "innundated".  It's not that you can't but you just won't for reasons that don't really make much sense to me or others.

[Painkiller8818 203]

1 minute ago, Ikario said:

That's the point, the "oh well, what's done is done" attitude is no good because you can STILL inform via e-mail to everyone that might be affected but have no idea  what's going on. There's still stuff you can do to improve the situation, you are just refusing to do it.  Now that the main wave has passed would be a good time to let people know if, as you said, wanted to help people directly affected but not be "innundated".  It's not that you can't but you just won't for reasons that don't really make much sense to me or others.

People not using emby connect or an Emby Premiere Key, does not need to enter a mailaddress, so how should they inform the users with a free server without mail addy?

[Ikario 38]

As I said before, you obviously cannot inform everyone, but you  CAN inform paid customers, everyone subbed to the forum, emby connect users, etc. That would be vastly more than what they have done.  Only thing people are asking is to make a reasonable effort to inform as many people potentially affected as possible. 

[ember1205 23]

13 minutes ago, ebr said:

But we did.  Affected systems were informed via their server logs and the fact that their server would not run.  A link to the same information on this site banner was in every one of their logs but we realized many people would not think to look at their logs so also put it out here - where we knew people would come when their server couldn't run.

In the end, we did what we thought was the best thing we could possibly do in the situation with the resources we have.  No doubt mistakes were made and we are now trying to do our best to deal with them.

Shutting my server down was not "notification." I have had numerous shutdowns of Emby having to do solely with the fact that updating the server software would prevent it from starting because the install always overwrote the startup script and changed the "runas" user. Every. Single. Time.

I don't look at the server logs. Ever. I have no cause to. When the server doesn't start, I immediately go looking for the startup script issue because that is what the problem has been every single time in the past. This time around took me a LOT more effort because... yep... you guessed it... I wasn't actively notified of anything.

The fact that you refuse to understand that shutting down the server is not, nor ever will it be, a form of "notification" is frustrating. Please just stop justifying your actions and accept the fact that, plainly, a lot of people were and are pretty pissed off about how this was handled.

2 minutes ago, Painkiller8818 said:

People not using emby connect or an Emby Premiere Key, does not need to enter a mailaddress, so how should they inform the users with a free server without mail addy?

Figure it out. Put the downloads behind a registration page, require a login to get to them, put a banner across the admin page or the various apps telling them to go sign up for a mailing list, modify the server software in a way that allows it to still start up but shows a notification message that all services are down... Personally, I don't care HOW it gets done, but it needs to get done. I was never offered any mechanism other than purchasing the Premiere license (which DOES require my email address) and joining this forum - neither of which actually brought ANY information my way until it was far too late.

 

It isn't the consumer's job to figure this sort of stuff out - it's the vendor's.

[Gilgamesh_48 943]

Whatever is finally decided it would be well for Emby to keep in mind that there are people, I believe many people, that have no need of any form of increased security and "increased" security implies that there will be increased complications. There should be a way to turn off and/or avoid any increased complications for those of us that do not want need the "enhancements."

Under no circumstances do I believe that I should have to perform extra actions because others "need" the enhancements.

I only have one user, me.
I only access one server, mine.
I do not use or turn on remote access at all.

I do not want to have to log in every time I use a client.
I do not want to have to login to access my local server and make changes.
In fact the only time I should have to log in is a first start up, if I need to do a refresh of credentials.

For me everything is local and I do not need any more so called security. The only thing more security would do for me is make my life using Emby harder.

I strongly hope the vocal people that need or want greater security do not overwhelm those of us that are very happy with what we have and I hope Emby keeps it simple  for those of us that like it simple.

Emby, you should be sure that you do not worsen the experience for those of us that like Emby just the way it is.

[andrewds 93]

15 minutes ago, Gilgamesh_48 said:

Whatever is finally decided it would be well for Emby to keep in mind that there are people, I believe many people, that have no need of any form of increased security and "increased" security implies that there will be increased complications. There should be a way to turn off and/or avoid any increased complications for those of us that do not want need the "enhancements."

Under no circumstances do I believe that I should have to perform extra actions because others "need" the enhancements.

I only have one user, me.
I only access one server, mine.
I do not use or turn on remote access at all.

I do not want to have to log in every time I use a client.
I do not want to have to login to access my local server and make changes.
In fact the only time I should have to log in is a first start up, if I need to do a refresh of credentials.

For me everything is local and I do not need any more so called security. The only thing more security would do for me is make my life using Emby harder.

I strongly hope the vocal people that need or want greater security do not overwhelm those of us that are very happy with what we have and I hope Emby keeps it simple  for those of us that like it simple.

Emby, you should be sure that you do not worsen the experience for those of us that like Emby just the way it is.

I agree. I think responding by implementing new features that further misrepresent Emby Server as a security product would be misguided. Sticky threads, KBs, and other documentation can be made available to help guide people how to properly secure Emby Server with purpose-designed solutions. However if I or @Gilgamesh_48want to run a passwordless server directly exposed to the Internet we should be able to do that, even if that means that there's a chance that our servers could be infiltrated and weaponized against others, potentially even our own government or neighbors. People who will set up Emby Server in an insecure manner will do it with everything else as well so weaponization of the hardware is arguably inevitable. At that point if it really is a problem the police or even the military could get involved if needed, that's why my taxes pay them after all.

I hope the Emby team focuses on continuing to do what they do well: deliver a feature rich and reliable media management solution.

[Gilgamesh_48 943]

6 minutes ago, andrewds said:

However if I or @Gilgamesh_48want to run a passwordless server directly exposed to the Internet we should be able to do that, even if that means that there's a chance that our servers could be infiltrated and weaponized against others, potentially even our own government or neighbors

FWIW: My server is NOT exposed to the internet at all. I have blocked Emby's internet access, except what is required for proper operation, and I disable "Remote connections" totally.

I just do not want or need and security or passwords beyond what I have.
In other words what I have to do to currently run Emby is all that I should ever have to do.

[andrewds 93]

1 minute ago, Gilgamesh_48 said:

FWIW: My server is NOT exposed to the internet at all. I have blocked Emby's internet access, except what is required for proper operation, and I disable "Remote connections" totally.

I just do not want or need and security or passwords beyond what I have.
In other words what I have to do to currently run Emby is all that I should ever have to do.

Fwiw this exploit worked even if you had "allow remote connections to this emby server" unchecked. But, that's not really the point. Internet access control should be managed by something other than, or at a minimum in addition to, Emby Server.

[pwhodges 1531]

3 hours ago, Ikario said:

a reasonable attempt to inform registered customers is the lowest bar possible and you have not cleared it.

They prioritised finding a way to deal with affected customers rather than telling (some) people who were not affected that they were not affected by something they didn't know about.

Apple and Microsoft also have issues which affect only a few people, but then get patched - are you going to complain to them about not being emailed about every one of those issues as well?

Once they have got through the rush of people needing help to sort their systems out, then they will write up the details in the article they have already got a place-holder for in their blog area.

Paul

[Racerprose 23]

I am in 100% agreement and frustration that this flaw was known for years and ignored. That is astonishing and hard to believe. It makes me feel like the developers do not take security seriously and makes me wonder what else are they maybe ignoring? Flaws can happen and things can get by. Look at what happened to Plex. That affected millions. You can never be 100% all the time, but this one hits totally different.

I also agree some more needs to be done within Emby to notify everyone. Not just those who they think are affected. I just happened to be checking reddit and I saw a notice about this.

The only thing that saved me I believe was that I had set passwords for local accounts. To me this is just common sense. I don't know why people haven't done this. Just because you are local doesn't mean you should be less safe. I would suggest to the developers @Lukeetc that they make it required/forced people must have passwords for local accounts. Do not leave it as optional.

For now I am keeping Emby as the work to migrate to something else at this time would be too much work for me. I hope those affected are able to get back up and running quickly.

Edited May 26, 2023 by Racerprose

[Luke 37068]

12 minutes ago, Racerprose said:

 

The only thing that saved me I believe was that I had set passwords for local accounts. To me this is just common sense. I don't know why people haven't done this. Just because you are local doesn't mean you should be less safe. I would suggest to the developers @Lukeetc that they make it required/forced people must have passwords for local accounts. Do not leave it as optional.

 

Hi, agreed and this will be happening in the upcoming 4.8 server release.

[Ikario 38]

50 minutes ago, pwhodges said:

They prioritised finding a way to deal with affected customers rather than telling (some) people who were not affected that they were not affected by something they didn't know about.

Apple and Microsoft also have issues which affect only a few people, but then get patched - are you going to complain to them about not being emailed about every one of those issues as well?

Once they have got through the rush of people needing help to sort their systems out, then they will write up the details in the article they have already got a place-holder for in their blog area.

Paul

First of all, the analogy is terrible, because if Apple and Microsoft had a blunder that big (a known exploit well known for over 3 years not patched creating a botnet and granting admin privileges), I would definitely receive more than one e-mail from multiple newsletters and it would be all over the news, so yeah, I would find out.  Also, those companies make SURE everyone finds out whenever they find a huge vulnerability like this one. Your point makes no sense.

What you don't seem to get is that there is for sure a ton of affected or still vulnerable systems that have no idea they have to patch their system or that it was affected and dead in the first place which should find out.  I don't usually respond to nonsensical white knighting but given that you quoted me, I feel like this was warranted.

This is not just people shouting because we are mad. I'm not mad or crazy or angry, but there's basic stuff that needs to be done to handle this better and it's not being done because apparently security through obscurity is still a thing somewhere.

 

 

[TMCsw 122]

2 hours ago, Luke said:

Hi, agreed and this will be happening in the upcoming 4.8 server release.

That’s good! (and must also apply to remote logins )

But what would be `gooder` is to let the Admin optionally set ‘user Must change-password on first login’ and a minimum strength ( i.e: pin, 4-char, 8-char-mixed_case, 10-char-mixed-special_char … (or for people who really insist ‘none’ and issue a !!WARRING!!) )

[ember1205 23]

2 hours ago, pwhodges said:

They prioritised finding a way to deal with affected customers rather than telling (some) people who were not affected that they were not affected by something they didn't know about.

Apple and Microsoft also have issues which affect only a few people, but then get patched - are you going to complain to them about not being emailed about every one of those issues as well?

Once they have got through the rush of people needing help to sort their systems out, then they will write up the details in the article they have already got a place-holder for in their blog area.

Paul

And how is it that they "dealt" with affected customers? By turning their servers off? I am going to state it again - turning my server off and writing some vague entry to a log file that I -never- look at does not constitute "notification!"

 

Let me put this a different way... What the hell right does the Emby team have to shut down my server? I understand that their "hearts were in the right place", but that is 100% -MY- property and they have absolutely no right to do ANYTHING to render it inoperable. EVER. I paid for a lifetime Premiere license. So long as I stay within what I am licensed for and licensed to do, NO ONE has a right to touch my property in any way. Do I -want- to be participating in a botnet? No. But, instead of writing code and pushing it to my server to crash it, they should have actually NOTIFIED ME of the issue and let ME decide how and when to deal with it.

 

There was absolutely no though put into recognizing the most important people to them - their PAYING customers. They told absolutely NO ONE what was going on and then slapped a banner on a web site that isn't even USED by most of their users and isn't frequented by most of the ones that DO have accounts here. I'm sick and tired of the "spin" being put on everything and the Emby team not standing up and admitting that they screwed up.

[Racerprose 23]

7 minutes ago, ember1205 said:

And how is it that they "dealt" with affected customers? By turning their servers off? I am going to state it again - turning my server off and writing some vague entry to a log file that I -never- look at does not constitute "notification!"

 

Let me put this a different way... What the hell right does the Emby team have to shut down my server? I understand that their "hearts were in the right place", but that is 100% -MY- property and they have absolutely no right to do ANYTHING to render it inoperable. EVER. I paid for a lifetime Premiere license. So long as I stay within what I am licensed for and licensed to do, NO ONE has a right to touch my property in any way. Do I -want- to be participating in a botnet? No. But, instead of writing code and pushing it to my server to crash it, they should have actually NOTIFIED ME of the issue and let ME decide how and when to deal with it.

 

There was absolutely no though put into recognizing the most important people to them - their PAYING customers. They told absolutely NO ONE what was going on and then slapped a banner on a web site that isn't even USED by most of their users and isn't frequented by most of the ones that DO have accounts here. I'm sick and tired of the "spin" being put on everything and the Emby team not standing up and admitting that they screwed up.

It seems like you are more upset with the inconvenience of your server being turned off than your system being compromised.

I agree only in that I do not think they went far enough to getting everyone notified about what was happening. I think they acted appropriately in turning off the systems affected. This happens all the time in other compromised situations where you may be temporarily logged out of your account whereby you will need to reset credentials.

At the end of the day, you do not own Emby. They do. If you don't like how they do something, of course, you are free to do whatever you want in terms of using their software or not.

Many mistakes were made here, but turning off the affected systems wasn't one of them, IMHO.

Edited May 27, 2023 by Racerprose