Server got hacked through API key over the Internet?

[unisoft 283]

Surely all API calls have to be authenticated and have session timeout?

[rbjtech 4287]

1 hour ago, unisoft said:

Surely all API calls have to be authenticated and have session timeout?

The API/Token IS the Authentication ...it is uniquely generated post client Auth - but I'm not sure what is going on behind the scenes - re authorization - maybe some sort of JWT is being used..    It may be the api/token is identification only.. 

https://dev.emby.media/doc/restapi/API-Key-Authentication.html

Edited April 22 by rbjtech

[rbjtech 4287]

36 minutes ago, unisoft said:

...  and have session timeout?

not that I'm aware of.   I've never been forced to logout of a client due to timeout of a token.

But agree - a token should not be allowed to last forever,  but the issue emby currently has, is it's such a pigs ear to log in again with any half decent password, that they probably don't want to do this at the moment.   Once there is a modern way to login - then yes, revoke the token peroidically would be best practice.

Edited April 22 by rbjtech

[kuldan5853 1]

12 hours ago, softworkz said:

@kuldan5853 Are you affiliated with a 4-letter company starting with 'O' from a city starting with 'N'?

If the answer is yes, then there might be a bigger problem..

Nope, not even close. 

[kuldan5853 1]

They're still trying to access my server with the old api key... very weird. 

I'm also seeing failed login attempts for random usernames (so far: tv, tt, david, admin)

Even weirder - can anyone explain why they try to (I assume scripted) stream videos? What is in it for them? 

(This is not emby log but nginx log):
 

2024/04/22 10:56:17 [error] 13988#6984: *6791 CreateFile() "C:\nginx/html/videos/720337/original.mkv" failed (3: Das System kann den angegebenen Pfad nicht finden), client: 24.134.136.165, server: MY_URL, request: "GET /videos/720337/original.mkv?DeviceId=86aebf2b605cc0da35437b51df9135dc&MediaSourceId=c9dac238402cb60e26050a7cbe374646&PlaySessionId=48aea457ab76ba8cba7ebb2948a7570b&api_key=18ff6e1756264241a73b659936a04f91 HTTP/1.1", host: "MY_URL"
2024/04/22 10:56:17 [error] 13988#6984: *6792 CreateFile() "C:\nginx/html/videos/1237905/original.mkv" failed (3: Das System kann den angegebenen Pfad nicht finden), client: 24.134.136.165, server: MY_URL, request: "GET /videos/1237905/original.mkv?DeviceId=86aebf2b605cc0da35437b51df9135dc&MediaSourceId=8a9892826452f3875a6955a13816d01e&PlaySessionId=ad42f46e72b14728aed7288eeea0903c&api_key=18ff6e1756264241a73b659936a04f91 HTTP/1.1", host: "MY_URL"
2024/04/22 10:56:18 [error] 13988#6984: *6793 CreateFile() "C:\nginx/html/videos/519609/original.mkv" failed (3: Das System kann den angegebenen Pfad nicht finden), client: 24.134.136.165, server: MY_URL, request: "GET /videos/519609/original.mkv?DeviceId=86aebf2b605cc0da35437b51df9135dc&MediaSourceId=8f40d70d32df4d70569ed2c13377a33e&PlaySessionId=892de435263c47d24f46a14a4ff0629f&api_key=18ff6e1756264241a73b659936a04f91 HTTP/1.1", host: "MY_URL"
2024/04/22 10:56:18 [error] 13988#6984: *6794 CreateFile() "C:\nginx/html/videos/255900/original.mkv" failed (3: Das System kann den angegebenen Pfad nicht finden), client: 24.134.136.165, server: MY_URL, request: "GET /videos/255900/original.mkv?DeviceId=86aebf2b605cc0da35437b51df9135dc&MediaSourceId=742db48e917d1681e3db8ec9ddd154dc&PlaySessionId=7c37b7d72e51f0b854638996eb2e6d13&api_key=18ff6e1756264241a73b659936a04f91 HTTP/1.1", host: "MY_URL"

[softworkz 3339]

14 hours ago, kuldan5853 said:

On 4/22/2024 at 5:44 AM, softworkz said:

@kuldan5853 Are you affiliated with a 4-letter company starting with 'O' from a city starting with 'N'?

If the answer is yes, then there might be a bigger problem..

Nope, not even close. 

Do you know somebody from Nürnberg, Germany?

The first attempt came from a company in that city. Once the attacker realized that he has some success, he quickly changed to continue attacks from a machine in the Azure cloud. 
I still have the suspicion that the initial API key might have been stolen somehow with access to your local network. Another option might be some UPnP/DLNA exploit.

[kuldan5853 1]

4 hours ago, softworkz said:

Do you know somebody from Nürnberg, Germany?

The first attempt came from a company in that city. Once the attacker realized that he has some success, he quickly changed to continue attacks from a machine in the Azure cloud. 
I still have the suspicion that the initial API key might have been stolen somehow with access to your local network. Another option might be some UPnP/DLNA exploit.

I live in Germany, but I have no specific connection to Nürnberg myself (even though those geolocation tools within Germany tend to be wildly inaccurate anyway...). 

As for the key, there are no signs of breach on my local net - also, if there was, I'd guess the attack would have been a bit more directed (why use two wrong API keys first?). 

However, you are right that my local network security is lacking in the sense that I (as of now, I plan to change that) have no separated guest network for my visitors - so a breached phone or something like that being on my network in the past is a possibility. (Why use it to attack emby of all things then is a mystery to me though). 

 

[softworkz 3339]

47 minutes ago, kuldan5853 said:

(even though those geolocation tools within Germany tend to be wildly inaccurate anyway...). 

It's not geo-location. There's a reverse DNS registered for the source IP: firewall.rz.opal-consulting.com
(didn't want to post it right away, in case you got any relation to them  )

47 minutes ago, kuldan5853 said:

As for the key, there are no signs of breach on my local net - also, if there was, I'd guess the attack would have been a bit more directed (why use two wrong API keys first?). 

I'm not closely familiar with the DLNA implementation of Emby, but I know that you can also use a token for the api_key and maybe those were tokens which had been automatically issued to other local DLNA clients which the attacker has gotten hold of.

I'm just guessing in the wild, though, but the question that I'm wondering about is not why he tried two non-working keys first - it's rather: "Why did the third one actually work?"

This is what has lead me to believe from the first moment on that it must have been a targeted attack in some form. Maybe a neighbor or a visitor who got access to your LAN. Maybe also somebody in the same subnet of the dynamic IP address assigned by your provider to the router and he managed to get some DLNA communication working through your router (any ports open? UPnP enabled?).

47 minutes ago, kuldan5853 said:

Why use it to attack emby of all things then is a mystery to me though

That's weird indeed and another reason which brought me in the direction that it might be somebody you know and who might get back to you later in some way like "haha, I've hacked your Emby Server and watched your videos!"

And once another detail is that it  appears that he (probably) started his actions from his own workplace (that company). This doesn't look like a professional hacker. (If he would have hacked that company, there would have been no need to switch over to Azure for continuing.)

The other possibility is that it's an arbitrary hacker who found your server accidentally and is working out a general exploit. Did he really watch videos for a long time like from start to end and then the next one?

Edited April 23 by softworkz

[rbjtech 4287]

So after confirming with @softworkz(thanks) - a clarification of API/Token usage below -

• Manually created API keys (using the UI) have full admin priviledges.
• Session/Login based Client created API keys/tokens have the permissions of the user that created them.

So if you ever needed any more reason to not use full admin accounts for general usage - here is one more reason ..

I'd like to see the manual API keys have granular priviledges (as it's clearly possible) - but for now, as I've changed them all recently as part of routine account hygiene, I'm not that concerned. 

However, for others that run clients using admin accounts, or have manually created api keys, and have posted logs on forums without them being obfuscated, I'd suggest some account hygiene on the manual api keys and forced client re-logins to regenerate new tokens.

[softworkz 3339]

Please also note: A password change invalidates all existing tokens for an account

[roaku 795]

2 hours ago, softworkz said:

I'm just guessing in the wild, though, but the question that I'm wondering about is not why he tried two non-working keys first - it's rather: "Why did the third one actually work?"

This is what has lead me to believe from the first moment on that it must have been a targeted attack in some form. Maybe a neighbor or a visitor who got access to your LAN. Maybe also somebody in the same subnet of the dynamic IP address assigned by your provider to the router and he managed to get some DLNA communication working through your router (any ports open? UPnP enabled?).

That was my initial suspicion. The question for me is whether or not api key(s) are included in the urls generated by Emby's DLNA/UPnP implementation.

 

 

[rdhardi 19]

34 minutes ago, roaku said:

The question for me is whether or not api key(s) are included in the urls generated by Emby's DLNA/UPnP implementation.

Ok, this answers my question that I posted earlier about Emby generating api keys. When I started reading this thread, I checked my api keys in the dashboard and I was surprised to see one there, that I never created, so I deleted it. Now that I think about it, it could have said DLNA. So, when are those created...when a user actually accesses DLNA? As far as I know, in my use-case, we don't use DLNA. I've disabled it, but is it safe to uninstall the plugin if we don't use it?

[roaku 795]

7 minutes ago, rdhardi said:

Ok, this answers my question that I posted earlier about Emby generating api keys. When I started reading this thread, I checked my api keys in the dashboard and I was surprised to see one there, that I never created, so I deleted it. Now that I think about it, it could have said DLNA. So, when are those created...when a user actually accesses DLNA? As far as I know, in my use-case, we don't use DLNA. I've disabled it, but is it safe to uninstall the plugin if we don't use it?

Yes.

[rdhardi 19]

Great, thank you!

[kuldan5853 1]

8 hours ago, softworkz said:

Did he really watch videos for a long time like from start to end and then the next one?

So, I kept the logs from the day before I noticed the intrusion.

First, there was an attempt to watch live TV for 15 seconds - which probably failed because my tuner is currently not set up correctly.

Then he watched a movie for 71 Minutes. 

Then he switched over to a TV Show episode - first attempt only 2 seconds, then the same episode for 27 minutes, which I assume is the full episode. 
Then another Episode of another show, for 15 Minutes - again, I assume the length of the episode. 
Then, a comedy special for 11 seconds, turned off, switched back to a TV show, for 21 minutes, so also a full episode. 

(This was all spread out over a full day, not close together). 

This would point to a human indeed, but I know nobody that works at the company you quoted above - it's also not a company my employer deals with. 

To me, this feels less like a targeted attack and more like a way in to actually watch stuff freely. 
 

[vaise 307]

There was a DLNA API key there that the system has created.  I dont use DLNA and its deactivated, so I removed this key.

I have added keys manually for autoscan and watchstate - neither of which are forward facing to anything remote or users.

I also FW limit access to my server from only the country I have users.

[rbjtech 4287]

7 hours ago, vaise said:

I have added keys manually for autoscan and watchstate - neither of which are forward facing to anything remote or users.

If those keys have ever been unintentionally published and scraped (such as in previously posted unobfuscated logs for other issues) - then your system may be vulnerable.  I would change them peroidically - a manual api key has the equivalent of a full admin user with no password, so keep those keys secret. 

Edited April 25 by rbjtech