FULL DISCLOSURE: Data Collection in the Process of BotNet Takedown

[guynamedbilly 13]

12 minutes ago, darkassassin07 said:

I've been following all the threads surrounding this issue for the last 5 days or so, I have read and understand your explanations.

 

The bit that has me a little confused is the number of updates vs the number of actions you've taken. I'm counting 5 'MovieDB' updates in my alerts (1.6.4 through 1.6.7 , 1.6.5 installed twice 28hr apart), each labeled 'compatibility update', as well as a server update.

 

In which updates are you probing servers for information (getting servers to reach out to you, same difference)? How many times did that actually happen? Which of those was actually something to do with themoviedb? What actually changed to makes things 'compatible'? What's with the duplicate version number imstall? The server update was obviously the main security patch; what's going on with the two moviedb updates shortly after? (may 26th and 29th)

When it comes to updates in general, 'compatibility update' isn't very descriptive to begin with, but this incident has also made that specific term/description untrustworthy to me. When I see an update with that description now, I pause and wonder what may actually be in it, both because of the lack of detail as well as the association. I can't help but question them now.

TBH your comment above my last one makes me wonder more. If you're pushing updates to remove 'security tightening actions' why don't the update descriptions just say that? Eager for more info about that from @Luke

This was a rare incident and I've never seen you have to take such... Unprecedented? Actions. I don't feel you abuse this power or take it lightly; but that shadow of a doubt is going to last.

Your comments are worth consideration.  I think it's more of a "don't send in the insurance adjusters while the building is still smoldering" kind of situation at the moment.

[softworkz 3341]

7 minutes ago, darkassassin07 said:

n which updates are you probing servers for information (getting servers to reach out to you, same difference)? How many times did that actually happen? Which of those was actually something to do with themoviedb? What actually changed to makes things 'compatible'? What's with the duplicate version number imstall? The server update was obviously the main security patch; what's going on with the two moviedb updates shortly after? (may 26th and 29th)

Here are the publish dates for the versions:

164: 2023-05-22 19:47:02
165: 2023-05-24 04:42:23
166: 2023-05-25 22:41:43
167: 2023-05-29 00:38:23

Phase 1 was 164
Phase 2 was 165

The other versions came past this incident.
 

[Gibberish 1]

12 hours ago, darkassassin07 said:

I've just never really been concerned about resource consumption, I have plenty of overhead space.

 

The second point I actually see the other way: if the main server machine is offline for whatever reason, the proxy serves a 'service unavailable, please try again later' page vs just timing out the connection. (same for the rest of my services as well) let's me know just the emby server is offline vs a much larger issue.

 

 

We've definitely all got some interesting differences in our setups.

I have redundant heartbeat monitors spread out among my various servers that'll alert me if any of them go down. As for overhead resources, DOS was a larger part of my childhood than any flavor of windows, so to this day I code or build for efficacy. My Emby file server and things my family use in the day to day are built as close as I can get it to my family's technical ability (none), so even things like the 225TB pool is pooled and mirrored using DrivePool (something goes wrong with a drive, just swap in a new one), with backups for important stuff going off site but still accessible easily. Apps like Sonarr have made this easier. I also moved from miniPCs as clients to Fire Sticks\Cubes, with instructions that if something happens to me, use the server till it dies and then sign up for streaming services. Even for home automation, nothing in installed that's not tied to a switch somewhere and can't exist in some form without the main server (Home Assistant) or zwave boxes (zwavejs on rPis).

Edited May 30, 2023 by Gibberish

[softworkz 3341]

17 minutes ago, darkassassin07 said:

This was a rare incident and I've never seen you have to take such... Unprecedented?

Yes - for both Emby and myself. I've never been much into security nor hacking . The used procedures were "invented" during the process. There were no such such plans somewhere on the shelf or intended 

44 minutes ago, darkassassin07 said:

Actions. I don't feel you abuse this power or take it lightly; but that shadow of a doubt is going to last.

There always needs to exist a level of trust. towards the developers of a software - you can't follow and validate all changes.

[jl5437 1]

So, where is the full report doc or post they promised was coming that details all of this?

[Gilgamesh_48 945]

3 hours ago, jl5437 said:

So, where is the full report doc or post they promised was coming that details all of this?

Right here:

And here:

And here:

Do you want more??????

[softworkz 3341]

The incident has forced us to put a lot of things on hold, including user support. We're still catching up on those things.

The report I promised is coming, though.

Yet, I have provided many details already in individual posts, so when you have an urgent desire, you might do as suggested here:

On 5/30/2023 at 5:02 AM, softworkz said:

Just go here https://emby.media/community/index.php?/profile/20815-softworkz/content/

Find Thursday noon and from there on go backwards (in the list - which is forward in time).
This should get you a pretty good picture without distraction.

There's also another subject coming up regarding those so-called "isolated" or "local-only" setups of Emby Server where users are assuming they would be safe, even when using empty admin passwords.

I'm just not clear yet, about how to explain the vulnerability without giving hints to potential attackers for exploiting it.

[guynamedbilly 13]

So there's currently no known risk that would make it a bad idea to deploy a new server right now, as long as everything is password protected?

[softworkz 3341]

2 minutes ago, guynamedbilly said:

So there's currently no known risk that would make it a bad idea to deploy a new server right now, as long as everything is password protected?

• Install a 4.7.12 or later
• Assign passwords to all accounts
• Don't enable "Do not require a password in the local network"

That's the safest way for new (and existing) installations.

[rbjtech 4304]

6 hours ago, softworkz said:

I'm just not clear yet, about how to explain the vulnerability without giving hints to potential attackers for exploiting it.

imo - It shouldn't be disclosed - until it is patched. 

5 hours ago, softworkz said:

• Don't enable "Do not require a password in the local network"

To clarify - As I have non-Admin multi-users (all with complex passwords for remote use) sharing local devices, it would be totally impractical to enter a password each time on the local network - thus I have enabled 'Do not require a password in the local network' for these users.   The Admin account is set to require a password and is set to no remote access.  All behind a RP.

Is this setup still considered acceptable - or are there still issues where a non-admin user can elevate priviledges for example ?

[softworkz 3341]

11 minutes ago, rbjtech said:

6 hours ago, softworkz said:

I'm just not clear yet, about how to explain the vulnerability without giving hints to potential attackers for exploiting it.

imo - It shouldn't be disclosed - until it is patched. 

That goes without saying. What I meant is that even a rough description of the nature of this might give away too much.
There is no specific patch for this one, but the introduction of a hard requirement for having admin passwords plus rate-limiting of authentication attempts will close this.

19 minutes ago, rbjtech said:

To clarify - As I have non-Admin multi-users (all with complex passwords for remote use) sharing local devices, it would be totally impractical to enter a password each time on the local network - thus I have enabled 'Do not require a password in the local network' for these users.   The Admin account is set to require a password and is set to no remote access.  All behind a RP.

Is this setup still considered acceptable - or are there still issues where a non-admin user can elevate priviledges for example ?

A full and direct elevation exploit is not known. But in .13 we have fixed a specific path of actions through which elevation would have been possible, when certain pre-conditions are fulfilled.

[softworkz 3341]

On 6/4/2023 at 10:01 AM, rbjtech said:

it would be totally impractical to enter a password each time on the local network - thus I have enabled 'Do not require a password in the local network' for these users

While 4.7.12 fixes the proxy header vulnerability, the other vulnerability would also benefit from the special "local network" treatment and there might be other - yet unknown - ways to leverage this. That's why the "local network" distinction will be removed, alongside the "Do not require password in local network" option.

Also, empty passwords for regular users will at least be strongly discouraged. Compensation with regards to user convenience, will be provided by new client features, for storing credentials for multiple users. That's the rough direction, same as laid out in previous posts already. Of course it will take some time to deliver all this and it won't come all at once. 

Meanwhile - for improved safety, you might find the following considerations and recommendations useful:

• Update to the latest stable or latest beta
• I would start moving away from using 'Do not require a password in the local network'
• Make sure all users (admin and non-admin) have passwords assigned
• As a temporary alternative, you could use the "easy password" (PIN) option - even though it is bound to the "local network" distinction as well, 
• Generally, having any password, even the most trivial one, even when it would be just very few letters, is increasing safety by a magnitude 
  Why?
  • Because an attacker doesn't know about your password length. It could be one or ten chars long and it rarely makes sense to start brute-forcing which could take a massive amount of time (while you are actually seeking for getting access easily via passwordless accoutns).
  • This will soon become even much more unattractive with the introduction of rate limiting authentication attempts.
• If you want it simple, you can use the same password for all (non-admin) accounts. This doesn't provide significant benefit to a hacker, not even if he knew that you're doing so
• Use the new Notifications feature and configure it in a way that you get notifications for "User Authentication Failed"
  Have them sent to your phone or via e-mail or whichever way you prefer - but make sure you'll notice these notifications - it doesn't help when these would land in some folder which your are just watching occasionally
• Don't use the default ports 8096 and 8920!
  This is how they can find your server most easily. Better:
  • Use ports 80 and 443: Will "hide" your server within the masses of other web servers
  • Use arbitrary ports at the upper end of the allowed range (1-65535)
    Your server may still be found in various ways, but doing so, moves you out of the group of the most-easy-to-find servers
• Risk Assessment
  When setting up your server, take a moment and think about the worst case: some hacker gains control over your server
  What would be the worst thing that can happen? How much would it hurt you?
  If it's just a server with media content and there's no private data of any kind accessible to the hacker, then it might be an annoyance but not the end of the world. It's much different when you have a multi-purpose server which is also used to story private data, documents, passwords, etc.
  So, isolation of concerns is another reasonable measure you can take. It doesn't reduce the risk of an incident to happen, but it reduces the impact in case it would happen
   

Edited June 5, 2023 by softworkz

[justinrh 175]

On 5/29/2023 at 8:52 PM, softworkz said:

We just don't have the slightest idea whether and what the hackers might have done or not. They could have done X on one server, Y on another server and nothing on once another server.

I still must have not clearly communicated my thought.  The point is simply that Emby did provide instructions for cleanup (for what Emby knows about the problem), but those instructions are a mess.  How can a user have confidence in executing the instructions provided by Emby if they instructions are a mess?

[softworkz 3341]

34 minutes ago, justinrh said:

I still must have not clearly communicated my thought.  The point is simply that Emby did provide instructions for cleanup (for what Emby knows about the problem), but those instructions are a mess.  How can a user have confidence in executing the instructions provided by Emby if they instructions are a mess?

It was based on everything we knew at that point in time and has been adjusted several times to clarify details based on user feedback.

It is not quite clear to me what your message is...

Should we have waited a few days to write a perfect article, leaving users waiting and wondering?
Or do you have any specific question about some detail?

[rbjtech 4304]

5 hours ago, softworkz said:

• Don't use the default ports 8096 and 8196!
   

btw - not that I use them, but default ports are 8096 and 8920 .. not sure where 8196 has come from ..

[rbjtech 4304]

5 hours ago, softworkz said:

While 4.7.12 fixes the proxy header vulnerability, the other vulnerability would also benefit from the special "local network" treatment and there might be other - yet unknown - ways to leverage this. That's why the "local network" distinction will be removed, alongside the "Do not require password in local network" option.

Also, empty passwords for regular users will at least be strongly discouraged. Compensation with regards to user convenience, will be provided by new client features, for storing credentials for multiple users. That's the rough direction, same as laid out in previous posts already. Of course it will take some time to deliver all this and it won't come all at once. 

Meanwhile - for improved safety, you might find the following considerations and recommendations useful:

• Update to the latest stable or latest beta
• I would start moving away from using 'Do not require a password in the local network'
• Make sure all users (admin and non-admin) have passwords assigned
• As a temporary alternative, you could use the "easy password" (PIN) option - even though it is bound to the "local network" distinction as well, 
• Generally, having any password, even the most trivial one, even when it would be just very few letters, is increasing safety by a magnitude 
  Why?
  • Because an attacker doesn't know about your password length. It could be one or ten chars long and it rarely makes sense to start brute-forcing which could take a massive amount of time (while you are actually seeking for getting access easily via passwordless accoutns).
  • This will soon become even much more unattractive with the introduction of rate limiting authentication attempts.
• If you want it simple, you can use the same password for all (non-admin) accounts. This doesn't provide significant benefit to a hacker, not even if he knew that you're doing so
• Use the new Notifications feature and configure it in a way that you get notifications for "User Authentication Failed"
  Have them sent to your phone or via e-mail or whichever way you prefer - but make sure you'll notice these notifications - it doesn't help when these would land in some folder which your are just watching occasionally
• Don't use the default ports 8096 and 8196!
  This is how they can find your server most easily. Better:
  • Use ports 80 and 443: Will "hide" your server within the masses of other web servers
  • Use arbitrary ports at the upper end of the allowed range (1-65535)
    Your server may still be found in various ways, but doing so, moves you out of the group of the most-easy-to-find servers
     

Definately worth putting this as a sticky or a wiki article imo - some great general best practice items there.

[darkassassin07 432]

5 hours ago, softworkz said:

 

• Use the new Notifications feature and configure it in a way that you get notifications for "User Authentication Failed"
  Have them sent to your phone or via e-mail or whichever way you prefer - but make sure you'll notice these notifications - it doesn't help when these would land in some folder which your are just watching occasionally

Is this beta only? The current stable build doesn't have any notification options related to users except playback started/stopped (not sure when you'd ever want that ...). All the others are updates, server restarts, or new media added. 

 

The feature is quite underdeveloped imo.

[rbjtech 4304]

5 hours ago, softworkz said:

Compensation with regards to user convenience, will be provided by new client features, for storing credentials for multiple users. That's the rough direction, same as laid out in previous posts already. Of course it will take some time to deliver all this and it won't come all at once. 

This is very welcome news and will make great strides in making things more secure - and multi-user remote users will be delivered as a bonus.

Thanks for the updates.  

[rbjtech 4304]

2 minutes ago, darkassassin07 said:

Is this beta only? The current stable build doesn't have any notification options related to users except playback started/stopped (not sure when you'd ever want that ...). All the others are updates, server restarts, or new media added. 

 

The feature is quite underdeveloped imo.

4.8.0.37 Beta does ?

Mine is sent to send a Pushover alert on Auth Failed - seem to work fine.  (High Priority Alert would have been nice, but small steps and all that..)

But agree it still needs abit of work - I would like all Auth requests sent to a file for example and then use that as the log for Fail2Ban.   The issue with using the normal Emby logs is they are huge, even with Debug off ...

 

[softworkz 3341]

13 minutes ago, rbjtech said:

6 hours ago, softworkz said:

• Don't use the default ports 8096 and 8196!

btw - not that I use them, but default ports are 8096 and 8920 .. not sure where 8196 has come from ..

I could never remember that second number, at some point I had just written 8196 out of laziness, and it got stuck somehow. 
Interestingly, I'm not the only one: https://emby.media/community/index.php?/search/&q=8196&quick=1

No idea whether I can claim the "invention", though - maybe it's just a natural assumption that it might be plus 100...  

[darkassassin07 432]

10 minutes ago, rbjtech said:

4.8.0.37 Beta does ?

Mine is sent to send a Pushover alert on Auth Failed - seem to work fine.  (High Priority Alert would have been nice, but small steps and all that..)

But agree it still needs abit of work - I would like all Auth requests sent to a file for example and then use that as the log for Fail2Ban.   The issue with using the normal Emby logs is they are huge, even with Debug off ...

 

This is all we've got on the stable branch:

[softworkz 3341]

18 minutes ago, darkassassin07 said:

Is this beta only?

Yes.

18 minutes ago, darkassassin07 said:

The feature is quite underdeveloped imo.

Feedback is welcome!
(in the beta forum please)

[KMBanana 84]

12 hours ago, softworkz said:

• Don't use the default ports 8096 and 8920!
  This is how they can find your server most easily. Better:
  • Use ports 80 and 443: Will "hide" your server within the masses of other web servers
  • Use arbitrary ports at the upper end of the allowed range (1-65535)
    Your server may still be found in various ways, but doing so, moves you out of the group of the most-easy-to-find servers

Changing the default port for a directly exposed to the internet port is better than nothing, but only just. 

If your server is responding to requests from the internet at literally on any IP:Port scheme at all by presenting an Emby login screen, it's not hidden within the masses of other web servers.  Bad guys are scanning every IPv4:Port combination on the internet constantly. 

If you want to really hide your Emby server you can setup a VPN server, remote users would need to connect to that, and once connected they can access your Emby server through a secure VPN tunnel. 

If you want to marginally hide your Emby server from bulk internet scans while still making it available on the public internet, you can setup a reverse proxy. Someone hitting your IP:Port would hit your reverse proxy, but the reverse proxy won't forward that request to your Emby server unless they are accessing it via the domain name.  I would recommend not using something common like emby.domain.tld, as domains can be indexed by bots as well.  

Both a reverse proxy and VPN servers are going to be more secure in general than exposing Emby directly to the internet, they simply have a lot more dev/security focus on the dangers associated with being exposed directly to the internet.  In any case, VPN, reverse proxy, or Emby directly to the internet you need to keep the application and settings up to date and secure.

Finally a small extra thing you can do is geoblocking countries you don't have users in.  A lot of the probes used to find and index servers and services come from China, Russia, and a few others.  You can block these countries IPs in a few different ways.  This won't fully stop scans, they still use compromised machines in other countries as well as VPN services themselves.  

Even with geoblocking and a reverse proxy I still would not consider an Emby Server "hidden", be careful out there folks!

[softworkz 3341]

6 minutes ago, KMBanana said:

Someone hitting your IP:Port would hit your reverse proxy, but the reverse proxy won't forward that request to your Emby server unless they are accessing it via the domain name.

Ah, great, thanks for the reminder!

Host header checking is something I wanted to put on our list a few days ago and then it slipped out of attention..

[rbjtech 4304]

7 hours ago, KMBanana said:

Changing the default port for a directly exposed to the internet port is better than nothing, but only just. 

If your server is responding to requests from the internet at literally on any IP:Port scheme at all by presenting an Emby login screen, it's not hidden within the masses of other web servers.  Bad guys are scanning every IPv4:Port combination on the internet constantly. 

If you want to really hide your Emby server you can setup a VPN server, remote users would need to connect to that, and once connected they can access your Emby server through a secure VPN tunnel. 

If you want to marginally hide your Emby server from bulk internet scans while still making it available on the public internet, you can setup a reverse proxy. Someone hitting your IP:Port would hit your reverse proxy, but the reverse proxy won't forward that request to your Emby server unless they are accessing it via the domain name.  I would recommend not using something common like emby.domain.tld, as domains can be indexed by bots as well.  

Both a reverse proxy and VPN servers are going to be more secure in general than exposing Emby directly to the internet, they simply have a lot more dev/security focus on the dangers associated with being exposed directly to the internet.  In any case, VPN, reverse proxy, or Emby directly to the internet you need to keep the application and settings up to date and secure.

Finally a small extra thing you can do is geoblocking countries you don't have users in.  A lot of the probes used to find and index servers and services come from China, Russia, and a few others.  You can block these countries IPs in a few different ways.  This won't fully stop scans, they still use compromised machines in other countries as well as VPN services themselves.  

Even with geoblocking and a reverse proxy I still would not consider an Emby Server "hidden", be careful out there folks!

Security through obscurity (STO) as the only way to protect your system is a bad idea.   If you have a web server on the public internet, then consder it available for exploit - it's that simple.

The security of any system needs to be built on increasing the number of layers of security through the OSI layer - ensuring each one is of a similiar security level vs budget vs risk.   For example, there is zero point in implementing MFA if you allow http traffic for example.

There are many other 'layers' of additional protection you could implement (not an exhaustive list..) -

- IPS/IDS (network packet layer filtering - to detect and drop all traffic with known exploits so they won't even get to your firewall)

- WAF (Web Application Security) - a firewall/IPS/IDS for Application layer protocols (http etc) - sometimes incorporated into the IPS/IDS and/or Reverse Proxy.

- Blacklists - know blacklisted public IP's and VPN source IP's (for example) are dropped.

- Network Segmentation - put your devices into security 'zones' - protected by firewalls or ACL's - that way a 'breech' is kept controlled to only devices in that zone.

-  Monitoring and Management are a key point, frequently over-looked - there is little point have lots of 'layers' of protection if you have no idea if they are working or a layer has been breeched.