FULL DISCLOSURE: Data Collection in the Process of BotNet Takedown

[softworkz 3341]

2 minutes ago, MrSaumon said:

would it not be possible to have shown the same red banners that are on top of the forum right now on our servers?

This is one of the ideas that we are currently collecting for evaluation.

 

10 minutes ago, MrSaumon said:

and then shutdown emby servers worldwide (yeah, I still am not very comfortable with this...)

3 minutes ago, MrSaumon said:

is discovered, a "news" section in the control panel or something like that.

That wouldn't have worked in this case. The explanation is written, the text is long and still needs a bit of work. Please allow a little more time, but I really want everybody to understand why we've done it this way and what the alternatives would have been (and why your suggestion wouldn't have worked in this specific case.

Thanks for your understanding.

[guynamedbilly 13]

I'm curious how they found which systems to target.  Was the Emby Connect server compromised?  And should we be concerned about any clients who might be using a connect account?

[andrewds 98]

7 minutes ago, guynamedbilly said:

I'm curious how they found which systems to target.  Was the Emby Connect server compromised?  And should we be concerned about any clients who might be using a connect account?

There are people/bot networks out there basically constantly probing ip addresses to see what is listening. If someone opens a port on their consumer router and port forwards external connections to Emby then when someone scans a port they will see that it is open. That invites attack.

[softworkz 3341]

9 minutes ago, guynamedbilly said:

I'm curious how they found which systems to target.  Was the Emby Connect server compromised?  And should we be concerned about any clients who might be using a connect account?

A port scan (for a single port) over the full IPv4 address range takes about a day nowadays. But AFAIR, that figure applies when you have a provider-scale infrastructure available.
So - yes, most likely port scans done by another BotNet or even by those servers which they already had.

We didn't include any identifiable information in the reports so we have no IP addresses, host names and ports and I can't even tell whether all the hacked servers had 8096 or 8196 as ports or whether they had found others.

Generally, I would recommend to never use 8096 or 8196 publicly. Either use ports without a known specific service type or use 80 and 443. Using the latter makes it much more difficult to find Emby servers - because:

• When you find an 8096 or 8196 port, then there's a high probability that it's an Emby server.
  • That means that you can find Emby servers with those ports alone through the portscan
• But when you publish via 80/443, a portscan doesn't help much, because like every 2nd or 3rd device has a webserver included
  • Now, you need to establish a TCP connection, make a request to the web server and analyze what you get in return
  • This procedure is more expensive by magnitudes compared to the port scan
  • So it's much less likely that your server will quickly be found

[pwhodges 1538]

Emby Connect is entirely separate from local server, and has not been involved in this incident in any way.

The only systems that need immediate attention are those which have been shut down because they are compromised; but all others should be updated to the current version which doesn't have the vulnerability as soon as possible.

As good practice, all users should review their approach to password security, in particular avoiding using the option to log in as an administrator without using a password, even locally.  Emby are also expected to roll out significant changes to password requirements in a while, once the necessary design work has been done.

Paul

[guynamedbilly 13]

4 minutes ago, softworkz said:

A port scan (for a single port) over the full IPv4 address range takes about a day nowadays. But AFAIR, that figure applies when you have a provider-scale infrastructure available.
So - yes, most likely port scans done by another BotNet or even by those servers which they already had.

We didn't include any identifiable information in the reports so we have no IP addresses, host names and ports and I can't even tell whether all the hacked servers had 8096 or 8196 as ports or whether they had found others.

Generally, I would recommend to never use 8096 or 8196 publicly. Either use ports without a known specific service type or use 80 and 443. Using the latter makes it much more difficult to find Emby servers - because:

• When you find an 8096 or 8196 port, then there's a high probability that it's an Emby server.
  • That means that you can find Emby servers with those ports alone through the portscan
• But when you publish via 80/443, a portscan doesn't help much, because like every 2nd or 3rd device has a webserver included
  • Now, you need to establish a TCP connection, make a request to the web server and analyze what you get in return
  • This procedure is more expensive by magnitudes compared to the port scan
  • So it's much less likely that your server will quickly be found

Good info.  I'll change up my ports.

[pir8radio 1292]

Im sure this has been mentioned, I didn't read all 6 pages of comments..  but maybe an option to enable/disable data collection,  defaulted to ON would be fine.    having the ability to turn it off..   I understand that defeats the purpose a bit if then hacked, one could turn it off. but you would have the info up until that point. 

[andrewds 98]

2 minutes ago, guynamedbilly said:

Good info.  I'll change up my ports.

Beyond just changing ports I recommend evaluating whether you really need this service exposed to the entire Internet at all times.

I have a few friends and family that access the service from time to time. I use an ACL on my internet facing firewall to control who can even communicate with Emby Server on the Internet facing port. This means out of the billions of public IP addresses in the v4 space, only a handful will even be allowed past the firewall to see that there is anything there. To everyone else the port is not just closed but will not respond at all.

It does mean that from time to time they have to contact me with a new IP address so I can update the configuration but that's a small price to pay in my opinion.

[justinrh 175]

How was Emby made aware of this botnet?

Does Emby know if every compromised (or poorly configured machine) has been shutdown per their 'take down' action?

Since there is nothing stopping a user from setting the sign in mode back to 'do not require a pwd', the botnet could be resurrected, correct?

[moviefan 184]

7 minutes ago, andrewds said:

Beyond just changing ports I recommend evaluating whether you really need this service exposed to the entire Internet at all times.

I have a few friends and family that access the service from time to time. I use an ACL on my internet facing firewall to control who can even communicate with Emby Server on the Internet facing port. This means out of the billions of public IP addresses in the v4 space, only a handful will even be allowed past the firewall to see that there is anything there. To everyone else the port is not just closed but will not respond at all.

It does mean that from time to time they have to contact me with a new IP address so I can update the configuration but that's a small price to pay in my opinion.

And if you do have it exposed to the internet, restricting access by country of origin is another mechanism you can use to further reduce risk.

[moviefan 184]

2 minutes ago, justinrh said:

How was Emby made aware of this botnet?

Does Emby know if every compromised (or poorly configured machine) has been shutdown per their 'take down' action?

Since there is nothing stopping a user from setting the sign in mode back to 'do not require a pwd', the botnet could be resurrected, correct?

As far as I can tell Emby was made aware of this botnet through user reported issues on this forum.

The latest version of Emby server removed the vulnerability that allowed remote users to masquerade as local users to the system by changing the HTTP headers; mitigating the specific vulnerability that was used to exploit the servers in this case.

[Gilgamesh_48 945]

6 minutes ago, pwhodges said:

As good practice, all users should review their approach to password security, in particular avoiding using the option to log in as an administrator without using a password, even locally.  Emby are also expected to roll out significant changes to password requirements in a while, once the necessary design work has been done.

So it is planed to require admin users on systems that are NEVER accessed remotely to have and use a password? That, if it is so, is NOT what I hoped for. But I can, with some trepidation, get by. I do not "want" to have to use a password local access but, if that is truly the best solution you can come up with, it would be acceptable, with protest.

Requiring a password for local users that do not use any form of remote access is both foolish and irresponsible and very user unfriendly.

I am sure I can find a way to automate the login process, even the Windows crappy password management would, I think, work. It is just an action that is unwanted by me and totally unneeded for me.

But, like many other companies, Emby knows what is good for me and will do nothing to me that i don't need, in their not at all humble opinion. and allow me to do nothing that might by a far fetched imagined method allow me to be harmed.

Emby is still the best media manager out there but by intruding on the way I manage my personal computers and by forcing me to use passwords on my strictly local server they have shown that they can get into a "cover your butts" mode and disregard the users that do not need or want the intrusion.

Once again I understand that the server software is owned by Emby and they are within their rights to make it do or require anything the want. But by not allowing strictly local usage without password they are showing that they have a good bit of the bully in them.

And remember that bullies, at least those that are not extremely rich, rarely get the respect they crave.

[pwhodges 1538]

4 minutes ago, justinrh said:

How was Emby made aware of this botnet?

A savvy user noticed odd behaviour on his system, and cooperated with Emby to let them analyse what was happening on their system.  After that analysis, they realised they had to find a way to get information from other compromised servers, and a way to deal with them.

Paul

[softworkz 3341]

6 minutes ago, pir8radio said:

Im sure this has been mentioned, I didn't read all 6 pages of comments..  but maybe an option to enable/disable data collection,  defaulted to ON would be fine.    having the ability to turn it off..   I understand that defeats the purpose a bit if then hacked, one could turn it off. but you would have the info up until that point. 

There is no regular data collection happening!

As explained in the OP

On 5/26/2023 at 2:08 PM, softworkz said:

Conditions

The local detection that we ran, performed a categorization into the following thee cases:

1. Systems that got hacked
2. Systems that are in danger of getting hacked
3. Systems that are not vulnerable

We have decided to report back only in cases 1 and 2. In case 3, no action was taken and no report was sent back to us.

To recap: 

In case 1, your system gets shutdown, you need to carefully investigate before you restart Emby server.

In case 2, your user security configuration gets tightened

(sometimes 1 and 2 are applied is applicable)

After 1 or/and 2

• You restart your system
• Is it still hacked? => NO
• Does user security need to be tightened? => NO

That means, you are now in category 3 and in category 3, no data is reported at all.

 

So the reporting is just a one-time thing and we don't hear anything from your server anymore unless you loosen user security or get infected again.

[andrewds 98]

7 minutes ago, justinrh said:

How was Emby made aware of this botnet?

Does Emby know if every compromised (or poorly configured machine) has been shutdown per their 'take down' action?

Since there is nothing stopping a user from setting the sign in mode back to 'do not require a pwd', the botnet could be resurrected, correct?

The exploit was disclosed on open forum here.

This user was actively experiencing attack and seeking help on the forum.

[pwhodges 1538]

1 minute ago, Gilgamesh_48 said:

Requiring a password for local users that do not use any form of remote access is both foolish and irresponsible and very user unfriendly.

I'm sure that Emby don't want to impose unreasonable requirements on users; but they want to have more secure ways of providing convenience, safer defaults, and warnings if users select options which are considered bad practice.

Paul

[justinrh 175]

3 minutes ago, softworkz said:

we don't hear anything from your server anymore unless you loosen user security or get infected again

Hold on pa-pa.  How do you know if I have loosened security or are infected (again) if reporting was a one-time action? 

[pir8radio 1292]

5 minutes ago, softworkz said:

There is no regular data collection happening!

As explained in the OP

To recap: 

In case 1, your system gets shutdown, you need to carefully investigate before you restart Emby server.

In case 2, your user security configuration gets tightened

(sometimes 1 and 2 are applied is applicable)

After 1 or/and 2

• You restart your system
• Is it still hacked? => NO
• Does user security need to be tightened? => NO

That means, you are now in category 3 and in category 3, no data is reported at all.

 

So the reporting is just a one-time thing and we don't hear anything from your server anymore unless you loosen user security or get infected again.

understood, but I'm actually suggesting regular data collection if we opt in..  could help speed development.

[softworkz 3341]

9 minutes ago, justinrh said:

Does Emby know if every compromised (or poorly configured machine) has been shutdown per their 'take down' action?

No. We have even no idea about how many Emby servers are out there.

We're just sure that we got a very large part covered, but we also keep getting reports (sparse) from servers that have probably auto-restart disabled or that had been switched off for a whole.

9 minutes ago, justinrh said:

Since there is nothing stopping a user from setting the sign in mode back to 'do not require a pwd', the botnet could be resurrected, correct?

No. At least not with the binaries floating around.

The DNS domain which they used for sinking data has either been taken down (by our abuse reports) or the hackers have abandoned it themselves and are just cleaning up.
It is currently in a "parking state'": http://spxaebjhxtmddsri.xyz/

[justinrh 175]

7 minutes ago, Gilgamesh_48 said:

So it is planed to require admin users on systems that are NEVER accessed remotely to have and use a password?

Could you not just create a non-admin user acct for your normal usage of Emby?  Then, when admin actions are required you could switch to the admin acct.

[Gilgamesh_48 945]

Just now, pwhodges said:

I'm sure that Emby don't want to impose unreasonable requirements on users; but they want to have more secure ways of providing convenience, safer defaults, and warnings if users select options which are considered bad practice.

Paul

All that say is that Emby is entitled to control their server that is running about as completely isolated as possible with no remote access at all. That is the act, as I said before, of a bully and I dislike being bullied.

Emby is still the best media server but "solving" this issue by forcing unaffected systems into Emby's pattern of use dropps them down a bit.

It is the kind of action I would expect from Plex or Apple or Microsoft. It is not a good action at all.

[softworkz 3341]

2 minutes ago, pir8radio said:

understood, but I'm actually suggesting regular data collection if we opt in..  could help speed development.

Each time that I thought it would be good to have certain data, it was so specific, that nobody would have thought about setting up collection for that earlier.

There has been an opt-in (or -out) collection years ago and it wasn't well-received, so it was dropped and the subject closed..

[Gilgamesh_48 945]

1 minute ago, justinrh said:

Could you not just create a non-admin user acct for your normal usage of Emby?  Then, when admin actions are required you could switch to the admin acct.

I do not want to do that as I am my only user and therefore I do not need or want another. If I am using my computer I might want to watch content or I might want to perform admin tasks and I do not wish to switch users just for that and even then I would, under current plans I think, still have to enter a password for the admin user. So that really would solve nothing for me. In fact it would increase complications.

[andrewds 98]

2 minutes ago, softworkz said:

Each time that I thought it would be good to have certain data, it was so specific, that nobody would have thought about setting up collection for that earlier.

There has been an opt-in (or -out) collection years ago and it wasn't well-received, so it was dropped and the subject closed..

Well I know it certainly doesn't apply to anyone here but I'm sure there is a non-trivial number of people concerned about the felonious amount of illegally obtained content that might be exposed via data collection.

[moviefan 184]

4 minutes ago, Gilgamesh_48 said:

All that say is that Emby is entitled to control their server that is running about as completely isolated as possible with no remote access at all. That is the act, as I said before, of a bully and I dislike being bullied.

Emby is still the best media server but "solving" this issue by forcing unaffected systems into Emby's pattern of use dropps them down a bit.

It is the kind of action I would expect from Plex or Apple or Microsoft. It is not a good action at all.

Requiring a password to perform administrative activities is simply good security.  Having the option to not use a password for the admin account was a bad idea to begin with.

I can't imagine you are regularly administrating your server via your streaming devices.  I assume you are doing this with a browser.  And in the case of a browser, you simply have to check the remember me checkbox after typing in a password one time, and the cookie logs you in for future browser sessions.  So, you have to type a password one time.  This really doesn't seem very onerous to me.