FULL DISCLOSURE: Data Collection in the Process of BotNet Takedown

[KMBanana 84]

2 hours ago, jl5437 said:

So... you guys have the ability to remotely disable your software eh?

Just want to mention that this type of thing is possible with literally any application that receives updates.  

[softworkz 3341]

2 minutes ago, andrewds said:

So this wasn't some behind the scenes thing only the Emby team could have accomplished, it's something that could have arguably been done by any plugin maintainer. That does give me pause, and is arguably abusive, but I suppose it's less overtly nefarious than other options.

It all goes through our catalog and there's only a small number of plugin developers who are allowed to update (only their) plugins.
And there's another aspect to consider: plugins from external developers often have a small reach only - that's very different to our stock plugins.

[jl5437 1]

Read through this thread...ok..wow. The exploit been known about for 3yrs now. And only when a user under active attack reports it, does it get attention...sigh. (links to those threads, are posted a few times already)

Whelp,  that changes things for me.

Still want to see their full disclosure of the details of this event, how they rationalize taking 3yrs to close a security issue.  I am sure  LTT will at least mention this drama in WAN show, maybe even make a dedicated video on it. Curious to see his opinion on this. 

But, i will not be re-installing Emby on my newly upgraded media server build and will be looking at other solutions for the time being. Sure, nothing is 100% exploit free,  but, Emby has lost my trust at this point. However, for full transparency on my part, my needs for Emby were already diminishing already.

Edited May 28, 2023 by jl5437

[Guest EmbyEbookReader]

I wish our government would start to drop ACTUAL BOMBS on hackers and do C.1.4. Black Ops to make these people just disappear,  hackers, and criminals 

[andrewds 98]

4 minutes ago, softworkz said:

It all goes through our catalog and there's only a small number of plugin developers who are allowed to update (only their) plugins.
And there's another aspect to consider: plugins from external developers often have a small reach only - that's very different to our stock plugins.

This is an important detail, at least to me. I'd like more detail about this in the official explanation if possible.

[Guest EmbyEbookReader]

1 minute ago, jl5437 said:

Read through this thread...ok..wow. The exploit been known about for 3yrs now. And only when a user under active attack reports it, does it get attention...sigh. (links to those threads, are posted a few times already)

Whelp,  that changes things for me.

Still want to see their full disclosure of the details of this event, how they rationalize taking 3yrs to close a security issue.  I am sure LTT will at least mention this drama in WAN show, maybe even make a dedicated video on it. Curious to see his opinion on this. 

But, i will not be re-installing Emby on my newly upgraded media server build and will be looking at other solutions for the time being. Sure, nothing is 100% exploit free,  but, Emby has lost my trust at this point. However, for full transparency on my part, my needs for Emby were already diminishing already.

That’s why I run my Emby always disconnected from the Internet.  It’s like a public bathroom out there.  All kinds of nasty stuff you don’t want to come into contact with.   Even with nothing proprietary or sensitive, just ordinary media!  Of course if you have home movies and personal data you will need to take more precautions like firewall rules blocking foreigners.  I bet you that the botnet was controlled from some foreign IP address in China or Russia or Nigeria, that’s where most of these criminals originate !  

[softworkz 3341]

6 minutes ago, jl5437 said:

how they rationalize taking 3yrs to close a security issue

I wasn't involved in this, so I cannot even try to give an explanation, but we all know that we failed badly in this regard.

[ftballpack 4]

A few quick questions.

-Do we have hashes for IOCs that would help users to verify if their systems have been indeed compromised?

-I saw one forum member stated his Amazon and eBay credentials were compromised. Were these credentials compromised because user passwords are in plain text in Emby or did the the malware install a keylogger on the user's system and if the malware included a keylogger, do we have any IOCs that a user is/was infected with said keylogger?

-I saw that @andrewdssuggested an IP allow list. Would it be possible for Emby to incorporate a national allowlist of IPs or a specific blocklist of IPs for countries that attacks often originate from? I have personally used Maxmind's GeoLite2 database and ModSecurity on Apache2 to restrict connections by country code. I know The Tor Project distributes a country IP list along with their binary, would it be possible to use one of the free IP location databases to restrict connections by country within Emby itself? I feel this could be used to reduce the damage from a similar type attack from ever happening again.

Sorry for the lack of brevity. I write like garbage when tired.

Edited May 28, 2023 by ftballpack
typo, tired...

[andrewds 98]

25 minutes ago, EmbyEbookReader said:

Do I need to install Emby BETA Server to get the fix, or should I wait until an official release is released?   Thank you. 

Both the latest beta and Emby Server 4.7.12.0 contain the fix.

[andrewds 98]

17 minutes ago, ftballpack said:

 

-I saw that @andrewdssuggested an IP allow list. Would it be possible for Emby to incorporate a national allowlist of IPs or a specific blocklist of IPs for countries that attacks often originate from? I have personally used Maxmind's GeoLite2 database and ModSecurity on Apache2 to restrict connections by country code. I know The Tor Project distributes a country IP list along with their binary, would it be possible to use one of the free IP location databases to restrict connections by country within Emby itself? I feel this could be used to reduce the damage from a similar type attack from ever happening again.

The issue in this exploit is specifically that Emby Server's ability to distinguish the IP of the originating connection was not working as expected. A remote attacker was able to convince Emby Server that the connection was originating within what it considers a local network and therefore all rules applied to remote connections were bypassed. Users hidden remote were exposed, users not allowed to connect remotely could be used, users that require a password remotely but no password locally would have the remote password requirement bypassed. Even if the server were configured with "Allow remote connections to this Emby Server." unticked the instance would still be accessible remotely if it were reachable from the internet.

Emby Server should not be responsible for this kind of filtering. It should also never be exposed directly to the internet. I suspect they will remove all of that functionality or otherwise greatly overhaul it.

[andrewds 98]

23 minutes ago, ftballpack said:

-I saw one forum member stated his Amazon and eBay credentials were compromised. Were these credentials compromised because user passwords are in plain text in Emby or did the the malware install a keylogger on the user's system and if the malware included a keylogger, do we have any IOCs that a user is/was infected with said keylogger?

The exploit could be leveraged to compromise the underlying operating system. That platform could then be used to penetrate and attack more deeply into the infrastructure. Should the device also happen to be used for other computing such as online banking, online shopping, etc. then anything accessed while compromised would also be at risk. Keyloggers, screen captures, etc are possibilities. Anything a user sitting in front of the device operating in the same security context could do.

[jl5437 1]

38 minutes ago, softworkz said:

I wasn't involved in this, so I cannot even try to give an explanation, but we all know that we failed badly in this regard.

The fact that this event was easily preventable, in so many ways, 3yrs ago. Who knows how long this attack been going on and how many compromised system there are now. 

[ftballpack 4]

9 minutes ago, andrewds said:

The issue in this exploit is specifically that Emby Server's ability to distinguish the IP of the originating connection was not working as expected. A remote attacker was able to convince Emby Server that the connection was originating within what it considers a local network and therefore all rules applied to remote connections were bypassed.

Not sure why you are repeating something that's already in the announcement.

27 minutes ago, andrewds said:

Emby Server should not be responsible for this kind of filtering. It should also never be exposed directly to the internet. I suspect they will remove all of that functionality or otherwise greatly overhaul it.

Plex allows local connections without auth without issue. If not for the known three-year-old http forgery issue combined with the compromised plugin, none of this would have happened. The two vulnerabilities were used in conjunction to compromise systems. Saying Emby should not be connected to the internet because of what happened is near equivalent to saying Windows machines should never be connected to any network because of the Conficker virus.

As far as "filtering" is concerned, I setup country blocks using Modsecurity on Apache2 in one afternoon. I wrote a PHP country allowlist for my mail server's web auth in a few hours and pass the flagged IPs to fail2ban. If a non-programmer can do this in a few hours, it's not that hard to add to Emby. Honestly, the only real question is dealing with licensing/redistribution of the free IP databases that various organizations provide, like what Tor does.

30 minutes ago, andrewds said:

The exploit could be leveraged to compromise the underlying operating system.

Not asking how it's possible a keylogger could have been installed, I am asking what do know for a fact that this point? Do we know if the malware installed a keylogger and if so do we have the SHA-2 hashes for the keylogger used? I am looking for IOCs, not explanations of how malware could get onto compromised systems. If the attacker is operating at a root level on a compromised system, it's pretty self explanatory how a system could end up with a keylogger on it.

 

 

If anyone could provide some IOCs hashes it would be greatly appreciated!!!!

[pektoral 17]

Hello,

i know this not the right thread to ask this, but its also security related.
My question is, do we get also a fix for the image leaking problem of emby?
This is a long time known issue and i dont want to share my private images publicly.

Image leaking without authentication:
http://xxx:8096/emby/Items/xxxx/Images/Primary?

King Regards,
pektoral

[CHBMB 2]

6 hours ago, softworkz said:

That makes sense in general, but we cannot provide a configuration switch that "turns on" a vulnerability as big and severe as this one.

I'm not asking for a vulnerability switch, but I don't see the issue with providing a "once logged on a device once, then do not reauthenticate if not logged out"  Exceptions could be made for admin accounts, and require them to authenticate each time, I suppose, but then single users will be upset that they don't want two accounts instead of a single user/admin account.

The Roku in our lounge has been configured, is on my LAN and not going anywhere, once I'm logged in with my account and my kids are logged in with their account, I don't see the need to reauthenticate when switching between them, this is possible in Plex as far as I can remember.

 

[andrewds 98]

  

1 hour ago, ftballpack said:

Not sure why you are repeating something that's already in the announcement.

I explained because it's apparent you haven't fully familiarized yourself with the problem. Despite your obnoxious response I am genuinely attempting to provide you useful feedback. You know, like a community.

1 hour ago, ftballpack said:

Plex allows local connections without auth without issue.

So does Emby. Did someone say it doesn't? The assertion is that it's bad security practice for administrator accounts.

1 hour ago, ftballpack said:

with the compromised plugin,

A compromised plugin was not involved in the initial attack. It was the proxy header vulnerability that you're aware of plus admin accounts without passwords plus potentially running the Emby Server process in an administrative security context.

1 hour ago, ftballpack said:

Saying Emby should not be connected to the internet

I didn't say it shouldn't be connected to the internet. I said it shouldn't be exposed directly to the internet. It sounds like you have a lot of excellent solutions in place. Nice job.

1 hour ago, ftballpack said:

As far as "filtering" is concerned, I setup country blocks using Modsecurity on Apache2 in one afternoon. I wrote a PHP country allowlist for my mail server's web auth in a few hours and pass the flagged IPs to fail2ban. If a non-programmer can do this in a few hours, it's not that hard to add to Emby.

That's very impressive, maybe they will be interested in the work you did and add it to Emby.

1 hour ago, ftballpack said:

Not asking how it's possible a keylogger could have been installed, I am asking what do know for a fact that this point?

I'm sure you're aware that they have said a more detailed explanation is coming.

1 hour ago, ftballpack said:

Not asking how it's possible a keylogger could have been installed, I am asking what do know for a fact that this point?

I'm sure @softworkzand the other people providing the analysis will include details like this if they are available.

Edited May 28, 2023 by andrewds
additional clarity/detail

[CHBMB 2]

6 hours ago, softworkz said:

I don't know much about Docker, so I'm not sure how it comes that requests from local IP addresses don't seem to get through directly to the docker container?
Is there some reverse proxy configured?

If yes, then the local network request need to get directly into the container, not through a RP.

Yes, traefik is a popular reverse proxy/load balancer.   

Sorrry, I probably should have been clearer.  Prior to the update my client ip addressed were correctly reported in my  Emby admin panel, whether they were on LAN or WAN, now following the update any client locally (on LAN) is getting the IP address of my Traefik container, any WAN client is still getting the correct IP addresses, the only thing to change in this time is the recent Emby update.

Within my network settings in Emby I have LAN networks set as:

10.x.0.0/24, 100.x.200.0/24 

which are my LAN network CIDR and my custom docker network that both Emby and Traefik are on.  So Emby should realise that whether the request is coming from

a. 10.x.0.0/24 (LAN) - ie if my client IP addresses are being correctly reported or b.

b. 10.x.200.0/24 (Docker Network) - ie with the current version of Emby which reports any client from LAN as having the IP address from the Traefik reverse proxy

Secure connection mode is "Handled by reverse proxy"

that both should be treated as LAN clients and not be asked for a password to change users, but this is not the case, as a result I'm getting a lot of requests from the little people I live with of "Daddy, can you put the password in" which just isn't a sustainable situation especially when I'm at work!

Happy to provide more details if required.

 

Like I say, I appreciate the need to secure your software and wholeheartedly support that, but that shouldn't be at the expense of functionality as fundamental as not requiring a password to switch between users on a device that is already authorised with the user accounts.

 

Imagine if Netflix or Amazon Prime needed you to enter a password each time you switched between users.

 

So in summary I've got two issues:

1. An issue where client IPs are no longer being correctly reported if they come from LAN

2. Once a user is authorised on a device, if I wish to switch users on this device, I have to reauthorise each time.

[CHBMB 2]

Sorry, tried to edit the above post to make some changes and add a bit more information but apparently 6 mins is too long to leave it.

  

6 hours ago, softworkz said:

I don't know much about Docker, so I'm not sure how it comes that requests from local IP addresses don't seem to get through directly to the docker container?
Is there some reverse proxy configured?

If yes, then the local network request need to get directly into the container, not through a RP.

I've played around with requiring PIN on LAN as a halfway fix for now, but I don't think Emby is picking up something the reverse proxy is definitely passing through, as even though my client is being recorded as coming from 10.x.200.y (and I have that CIDR specified as a LAN network as mentioned above) I can't use a PIN either. 

So something quirky is going on with Emby's current "Network Settings" and they don't seem to be doing what they say they do.

[darkassassin07 432]

It's almost like there was some sort of rushed update pushed out that focused more on results than polish....

 

Reverse proxy connections can no longer report a local ip as its proxied client, the server will use the proxies ip instead. As per the most recent changes mitigating a pretty massive security vulnerability.

 

This behaviour and labeling will likely get improved, but we're not there yet, obviously.

 

Patience.

[pwhodges 1534]

7 hours ago, Gilgamesh_48 said:

The more I hear and the more it gets rationalized the more I dislike where this is going. It has become clear that those of, admittedly in the minority, that do not want or need security are going to be forced to use it and it does not matter if we want to use it or not.

Users want/needs are only being met if those wants/needs fits with Emby's agenda.

I am done with this thread, the bullies have won. That should please some people.

So Emby is including code to detect an intrusion in your system, and you object to it?  Really?

Can I then assume that you also refuse to run anti-virus software on your system, which you "know is safe"?  After all, if it prevents you running a virus then it's also bullying you...

Paul

[CHBMB 2]

3 minutes ago, darkassassin07 said:

It's almost like there was some sort of rushed update pushed out that focused more on results than polish....

 

Reverse proxy connections can no longer report a local ip as its proxied client, the server will use the proxies ip instead. As per the most recent changes mitigating a pretty massive security vulnerability.

 

This behaviour and labeling will likely get improved, but we're not there yet, obviously.

 

Patience.

I'm ok regards your point of being patient, but I was more responding as a "bug report" unless there's a better place to do it?

Like I say, I understand the need to secure the software, but the end result needs to be usable, and unless I convey the issue I'm facing then I can't expect it to be fixed.

That's the funny thing though, clients coming in from WAN, do correctly show their IP.  It's only clients on LAN that are showing with the proxy IP and nothing I can do in settings seems to convince Emby Server these are LAN clients.

The ingress of traffic is the same, although on LAN I have a redirect from my domain name to docker host IP (Port 443)

[pwhodges 1534]

1 hour ago, jl5437 said:

Who knows how long this attack been going on and how many compromised system there are now. 

They know the date the domain that the infection sent data to was created - it was a few weeks ago.

Last count I saw of compromised systems was just over 1500 - remember, they now report in...

Paul

[rbjtech 4304]

47 minutes ago, darkassassin07 said:

It's almost like there was some sort of rushed update pushed out that focused more on results than polish....

 

Reverse proxy connections can no longer report a local ip as its proxied client, the server will use the proxies ip instead. As per the most recent changes mitigating a pretty massive security vulnerability.

 

This behaviour and labeling will likely get improved, but we're not there yet, obviously.

 

Patience.

I don't believe this has ever worked - nor to my knowledge has it every been reported.   As I run multiple vlan's - 'local' devices don't pass their IP's if going direct (via a f/w rule), nor do they if using my internal proxy either (with the correct X-Forwarded set).  They only report the emby server gateway address.  But as long as WAN IP's show/report (which they do) then it's not a huge issue imo as it's all inside my own security zones and device details ARE logged, just not local IP's.

Maybe something to put on the list - I agree.

Edited May 28, 2023 by rbjtech

[andrewds 98]

29 minutes ago, pwhodges said:

They know the date the domain that the infection sent data to was created - it was a few weeks ago.

Last count I saw of compromised systems was just over 1500 - remember, they now report in...

Paul

Thanks for the additional info Paul. To @ftballpack's question do you know if there are any MD5 hashes available? We know that the Emby team did not have direct access to analyze the systems of impacted users but perhaps someone they've worked with has made the malware payloads available to them. If so they could provide a hash of the helper.dll file or anything else useful.

[rbjtech 4304]

17 minutes ago, andrewds said:

Thanks for the additional info Paul. To @ftballpack's question do you know if there are any MD5 hashes available? We know that the Emby team did not have direct access to analyze the systems of impacted users but perhaps someone they've worked with has made the malware payloads available to them. If so they could provide a hash of the helper.dll file or anything else useful.

https://www.virustotal.com/gui/file/0ae4f9ca1906d7fa6e7b4d43c32dee52d281d615de451a67e2407c0e6a32dc2b?nocache=1

https://www.virustotal.com/gui/file/037b9b3e43ab4876b4c3b7b08303833cc10c93ec11be38e4ef394a3f840eeb2c?nocache=1