FULL DISCLOSURE: Data Collection in the Process of BotNet Takedown

[rbjtech 4304]

2 hours ago, ftballpack said:

If anyone could provide some IOCs hashes it would be greatly appreciated!!!!

See above

[andrewds 98]

10 minutes ago, rbjtech said:

https://www.virustotal.com/gui/file/0ae4f9ca1906d7fa6e7b4d43c32dee52d281d615de451a67e2407c0e6a32dc2b?nocache=1

https://www.virustotal.com/gui/file/037b9b3e43ab4876b4c3b7b08303833cc10c93ec11be38e4ef394a3f840eeb2c?nocache=1

 

Excellent, thanks so much for sharing this. Hopefully not from your setup?

[darkassassin07 432]

3 hours ago, rbjtech said:

I don't believe this has ever worked - nor to my knowledge has it every been reported.   As I run multiple vlan's - 'local' devices don't pass their IP's if going direct (via a f/w rule), nor do they if using my internal proxy either (with the correct X-Forwarded set).  They only report the emby server gateway address.  But as long as WAN IP's show/report (which they do) then it's not a huge issue imo as it's all inside my own security zones and device details ARE logged, just not local IP's.

Maybe something to put on the list - I agree.

It's always worked perfectly for me.

My server is behind an nginx reverse proxy with some pretty similar config to the pinned one in this forum category.

 

LAN and WAN clients have always reported their correct ip through the proxy via x-forwarded-for.

 

Since the most recent update, LAN clients now report as the ip of the nginx instance, but WAN clients still report their proxied ip as before.

[pwhodges 1534]

7 minutes ago, darkassassin07 said:

Since the most recent update, LAN clients now report as the ip of the nginx instance, but WAN clients still report their proxied ip as before.

That's the point, surely?  Now, if Emby sees a local address being passed on as the originating address by the reverse proxy, it doesn't trust it - because setting that header like that is exactly what this exploit was doing.

Paul

[darkassassin07 432]

7 minutes ago, pwhodges said:

That's the point, surely?  Now, if Emby sees a local address being passed on as the originating address by the reverse proxy, it doesn't trust it - because setting that header like that is exactly what this exploit was doing.

Paul

Yes. I'm just confirming how things are currently functioning vs how they were prior to this weeks snafu. The comment I replied to seemed to be saying it's always reported a lan ip as the ip of the gateway/proxy, but that's not been my experience previously.

 

 

 

I would like to see this header still being used to report what the client ip is, just not used as part of the authentication requirements (to decide if the client is lan/wan). Perhaps as a seprate line? Or a tooltip? I'd prefer to still see what ip a client has signed in from/reported via that header at a glance of the dashboard regardless of whether it's lan/wan. But it's a pretty minor thing as long as WAN ips still show up. If I really need to I can correlate proxy logs with emby logs 

[pünktchen 1259]

6 hours ago, pektoral said:

Hello,

i know this not the right thread to ask this, but its also security related.
My question is, do we get also a fix for the image leaking problem of emby?
This is a long time known issue and i dont want to share my private images publicly.

Image leaking without authentication:
http://xxx:8096/emby/Items/xxxx/Images/Primary?

King Regards,
pektoral

You nailed it. Hopefully the Emby devs have learned something from the current debacle.

[Gibberish 1]

What's interesting is seeing all the use cases at play here.

For one thing, excluding the docker servers, I'm surprised that those that have a reverse proxy set up, aren't also running a DNS server on their network. I would have thought that if you were technically inclined to do one, you'd also do the other. But that could also be because I had to setup my remote settings to mimic the proxy https with a cert that drops in from a powershell script. That's really just another example of how even among the most technical amongst us, there's no unified setup.

Reading back through the posts, there's one thing I didn't dwell on, the transmitting of username and passwords. Why is access to (I'm assuming unencrypted) passwords available to plugins at all? Do the plugins have access to raw request data?

Actually, now that I'm typing this out, I'm remembering that there are authorization plugins (LDAP) that would need this data to work. 

This has given me a lot to think about in regards to my own code and if it's possible for plugins in my system to do the same thing.

Edited May 28, 2023 by Gibberish

[pwhodges 1534]

Emby stores the passwords as a hash.  The malware was capturing newly typed Emby logins before hashing.

Paul

[darkassassin07 432]

1 hour ago, Gibberish said:

I'm surprised that those that have a reverse proxy set up, aren't also running a DNS server on their network. 

 

A) why do you think we arent? I haven't really seen discussion either direction. I can't speak for other's, but I do run my own dns in the form of pihole. My LAN and VPN devices are blocked from reaching public dns servers and must go through my local pihole instance.

 

 

B ) what difference does that make in this scenario? Unles that malicious domain (the one user credentials were being uploaded to in this attack) was known and previously blocked, or you're running a whitelist only setup, I don't see where that would help.

Edited May 28, 2023 by darkassassin07

[ebr 14947]

10 hours ago, jl5437 said:

Ok. So, a users login credentials are stored not encrypted? 

No, that is not correct.  Any passwords we store are encrypted.  We said you should assume all credentials are compromised because we have no way of knowing which were or weren't.  The plug-in hooked into the authentication process through the event functionality that was requested by users.  So they only had access to credentials for users that actually logged into your system during the time the plug-in was active.

[Gibberish 1]

24 minutes ago, darkassassin07 said:

 

A) why do you think we arent? I haven't really seen discussion either direction. I can't speak for other's, but I do run my own dns in the form of pihole. My LAN and VPN devices are blocked from reaching public dns servers and must go through my local pihole instance.

 

 

B ) what difference does that make in this scenario? Unles that malicious domain (the one user credentials were being uploaded to in this attack) was known and previously blocked, or you're running a whitelist only setup, I don't see where that would help.

It's more of an observation of why some users are seeing the IP of the proxy, since the DNS isn't directing them directly to the media server, they're coming the long way around and back in through the reverse proxy. And calm down, like I said, there's a lot of use case here.

[thornbill 6]

11 hours ago, softworkz said:

But when you install software from a 3rd party, there's always a certain amount of trust required. Because it doesn't make a difference whether it gets auto-installed or you do it manually: In both cases, you can't  "see" what's inside the update you are installing.

And this is one of the main reasons open source is so important. Zero trust is required if you can audit, verify, and build the changes yourself.

[Guest]

10 hours ago, KMBanana said:

Just want to mention that this type of thing is possible with literally any application that receives updates.  

I just want to mention that usually (100% of the time), updates on my server are not done without me knowing about it. Emby is the only closed-source software running on my server, as the jellyfin apps are not up to par yet.
I wont say I review every update I do on my system, but that one was really sneaky. I know there is a full explanation coming soon (tm), but I bet this won't address who in the emby team was (probably still is) able to update whatever they want on my system without me knowing about it. It may be a whole new fun time when one of the emby devs system is hacked and another update is pushed to us.

Yeah, I know I'm getting more and more angry about this, but I'm getting sick of reading all of those official posts dodging the questions, probably for legal reasons.
(Before I get a "WHAT DIDN'T YOU UNDERSTAND WE EXPLAINED IT ALL IN THE ADVISORY YOU DUMB USER", I would like to know which plugin was updated to gather data, and how was it activated without me having the "restart automatically after plugins update" option checked. Was it one of those "compatibility updates" I accepted?).
I know you did what you could do, and yeah, congrats on mitigating a huge vulnerability in your software, a vulnerability you ignored for 3 years. Here's a medal.

Edited May 28, 2023 by MrSaumon

[darkassassin07 432]

43 minutes ago, Gibberish said:

It's more of an observation of why some users are seeing the IP of the proxy, since the DNS isn't directing them directly to the media server, they're coming the long way around and back in through the reverse proxy. And calm down, like I said, there's a lot of use case here.

Ah, I see. 

 

I route all traffic to my services through the (local) reverse proxy because I access it via subdomains along with a variety of other services all though the same port. It has to go through the proxy to route to the correct service, or I have to use the ip+port that service is directly hosted on (8096 specifically for emby instead of just 443 for everything).

 

It's much easier to remember/use subdomains and letting the browser default to port 443 vs remembering everything's ports. One of the biggest reasons to use a RP imo, besides security.

 

"calm down" ......? I'm not upset, I'm just curious about your thinking is all. So I asked questions and explained some of my setup as an example.

[MBSki 1025]

@ebrWere any systems running under the nssm Windows service affected? I'm wondering how the plugin could've even gotten installed since there's no way to auto restart when running under the service.

[pwhodges 1534]

9 minutes ago, MBSki said:

@ebrWere any systems running under the nssm Windows service affected? I'm wondering how the plugin could've even gotten installed since there's no way to auto restart when running under the service.

Emby have said they didn't collect any information which would enable them to answer this.

32 minutes ago, MrSaumon said:

Was it one of those "compatibility updates" I accepted?

They have said it was.

Please try to keep your responses to the lamentable failure to plug the vulnerability and to their response to an infection separate.  Of course the response shouldn't have been necessary - but you'll see when they publish the details (the full report is being prepared) that what they did was justified and how they did it was appropriate and well executed.

Paul

[Guest]

1 minute ago, pwhodges said:

Emby have said they didn't collect any information which would enable them to answer this.

They have said it was.

Please try to keep your responses to the lamentable failure to plug the vulnerability and to their response to an infection separate.  Of course the response shouldn't have been necessary - but you'll see when they publish the details (the full report is being prepared) that what they did was justified and how they did it was appropriate and well executed.

Paul

Yeah, you're right, I should shut up and wait a few weeks for the full report, I'm too entitled.
I will keep thinking, though, that it was not, and will never be, appropriate to update systems with a fake reason and shutdown services without the admin's knowledge, but I guess that's just me. I was not impacted by this, by the way.
I won't post again, my bad.

[Q-Droid 657]

Keep in mind that it's entirely possible and even likely that the bad actors have Emby community accounts, Emby servers and even paid Emby Premiere keys. Channels used to alert the good guys would have also alerted the bad ones.

 

 

[Gibberish 1]

30 minutes ago, darkassassin07 said:

Ah, I see. 

 

I route all traffic to my services through the (local) reverse proxy because I access it via subdomains along with a variety of other services all though the same port. It has to go through the proxy to route to the correct service, or I have to use the ip+port that service is directly hosted on (8096 specifically for emby instead of just 443 for everything).

 

It's much easier to remember/use subdomains and letting the browser default to port 443 vs remembering everything's ports. One of the biggest reasons to use a RP imo, besides security.

 

"calm down" ......? I'm not upset, I'm just curious about your thinking is all. So I asked questions and explained some of my setup as an example.

I do the same for the majority* of my sub domains as well, but there's two reasons the internal DNS points directly to the server (using the same https external address and 443);

• it seemed like a waste of resources to have the reverse proxy constantly working for the LAN Emby clients.
• If something happens to the reverse proxy server, the family still has access to the Emby server at home without changing the server address.

*some apps I have proxied again through my home site (ex. the -arr apps) so that I can use a common authentication handler as much as possible.

[pwhodges 1534]

25 minutes ago, MrSaumon said:

I will keep thinking, though, that it was not, and will never be, appropriate to update systems with a fake reason and shutdown services without the admin's knowledge,

16 minutes ago, Q-Droid said:

Keep in mind that it's entirely possible and even likely that the bad actors have Emby community accounts, Emby servers and even paid Emby Premiere keys. Channels used to alert the good guys would have also alerted the bad ones.

They believed that it was necessary to do it that way, and those of us who have seen the draft of their report agree with them.  It had to be done secretly (your "fake reason") so that the hacker(s) didn't get any warning which could have given them time to adapt their hack to make the measures against it ineffective.

@MrSaumonYou don't need to shut up; I'm just trying to maintain a sense of perspective, is all.

Paul

[ftballpack 4]

12 hours ago, andrewds said:

A compromised plugin was not involved in the initial attack. It was the proxy header vulnerability that you're aware of plus admin accounts without passwords plus potentially running the Emby Server process in an administrative security context.

Sorry, @andrewdsI thought the following meant that the plugin itself was compromised also. I could be wrong. That's the problem with the forums currently, they are a crazy mix of accurate and confusing information as to what really happened.

 ```It appears that the infection is done through ScripterX

• Probably going through some package updating mechanism
• The server for ScripterX package updates is showing a certificate error
  (https://packagecatalog.emby-scripterx.com)
• The package catalog wiki link is going to an empty page
  (https://wiki.emby-scripterx.com/doku.php?id=packages)
• The whole wiki appears empty
  So it's likely that they hacked that wiki site to serve compromised update packages```

Edited May 28, 2023 by ftballpack
minor typo

[darkassassin07 432]

ScripterX was just an available tool the attacker used. It in it of itself wasn't compromised/modified, it just perhaps unwisely had too much power to manipulate the underlying system (download+execute files/payloads to the server).

 

Embyhelper.dll was the actual 'compromised' (built as a virus, by/for the attacker) plugin being called out as a virus in that thread.

 

Yeah, info is a bit hard to follow atm, especially the speculation mixed in with sprinkles of fact. Looking forward to the update/writeup from the emby team about this fiasco and the actions surrounding it.

[One2Go 98]

Just adding my observations to what happened to me.

On May 11th I found a few entries in my dashboard log which I didn't like. I check my dashboard daily. Here is a screenshot of the entries from May 11th to May 13th.

 

I checked the forum for the topic Scripter -X and found this:

New Plugin - Custom Scripting | Emby ScripterX

Concluded that this perhaps was something legitimate as it was installing and uninstalling and such a plugin existed in the catalog. Started to keep an eye on it and after it stopped on May 13th didn't think anything more about. Since then I have learned a few additional things.

The hackers used actions to create entries in the logs to deceive users including the failed in login attempts. To my surprise the server didn't start on the 25th and came here for the solutions and their implementation worked and solved the problem for my setup on a QNAP NAS, using a different port then 8096.

I can understand the rage and anger of messing with your hardware when not being given permission. However I resolved this unusual solution with the famous phrase “Logic clearly dictates that the needs of the many outweigh the needs of the few.” or the one".

Therefore patiently waiting of a full report so that I can see if anything further needs to be done or is suggested on what to do after updating to version 4.7.12

Edited May 28, 2023 by One2Go

[softworkz 3341]

14 hours ago, andrewds said:

14 hours ago, ftballpack said:

Not asking how it's possible a keylogger could have been installed, I am asking what do know for a fact that this point?

I'm sure @softworkzand the other people providing the analysis will include details like this if they are available.

No Keylogger was installed.

[justinrh 175]

6 hours ago, ebr said:

Any passwords we store are encrypted.  We said you should assume all credentials are compromised because we have no way of knowing which were or weren't.  The plug-in hooked into the authentication process through the event functionality that was requested by users.  So they only had access to credentials for users that actually logged into your system during the time the plug-in was active.

Encrypted or hashed?

Is it correct to say that the passwords are sent in clear text from the client to the server?  (assuming on a non-TLS connection)