Server got hacked through API key over the Internet?

[kuldan5853 1]

Hi Forum / Team.

I have a curious one... I am running my Emby on a Windows Server on the Internet (yeah, I will look into NGINX now...), at the time (yesterday) it was at Version 4.8.3.0. 

When checking the server today, I saw a user streaming media that I have not created - and checking the logs, I found that it was created at 3am last night. 

Now, what is curious is that this user was created via an API Key - which I have actually never written down anywhere but inside of Emby, so I have no clue how it got leaked to the outside? 

What's even weirder is that the account (with full permissions) did not even try to take over the server, take away my admin or anything - it just started to stream a few videos over the day. 

I saw some generic probing of the port during the day for common sites like wordpress admin etc., then when it figured out it is emby a few login attempts which were denied, then some attempts with invalid API keys, and suddenly it was using the one API key that was specified in my EMBY Server for DLNA use, and created a user called "tv" with it. 

The user then logged in a minute later and..started watching videos. It's really weird. 

Any idea how they could have gotten the API Key?

(I have since turned off remote access to the server for now and deleted the API key)

Here's an exerpt of the logfile:

2024-04-20 02:54:03.315 Info Server: http/1.1 POST https://‌‍‍MY_PUBLIC_IP‌:443/wordpress/wp-login.php. UserAgent: Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/62.0.3202.9 Safari/537.36
2024-04-20 02:54:03.315 Info Server: http/1.1 Response 404 to ‌‍‍104.211.2.187‌. Time: 0ms. POST https://‌‍‍MY_PUBLIC_IP‌:443/wordpress/wp-login.php
2024-04-20 02:55:29.305 Info Server: http/2 POST https://‌‍‍SERVER_URL.com‌/emby/Users/authenticatebyname?X-Emby-Client=Emby Web&X-Emby-Device-Name=Google Chrome Windows&X-Emby-Device-Id=6f5ca88a-c36c-4da7-b949-5e1beb7d5aa7&X-Emby-Client-Version=4.8.3.0&X-Emby-Language=de. Accept=application/json, Host=‌‍‍SERVER_URL.com‌, User-Agent=Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/123.0.0.0 Safari/537.36, :method=POST, Accept-Encoding=gzip, deflate, br, zstd, Accept-Language=de-DE,de;q=0.9,en-DE;q=0.8,en-US;q=0.7,en;q=0.6, Content-Type=application/x-www-form-urlencoded; charset=UTF-8, Origin=‌‍‍https://SERVER_URL.com‌, Referer=‌‍‍https://SERVER_URL.com/web/index.html‌, Content-Length=11, sec-ch-ua="Google Chrome";v="123", "Not:A-Brand";v="8", "Chromium";v="123", DNT=1, sec-ch-ua-mobile=?0, sec-ch-ua-platform="Windows", sec-fetch-site=same-origin, sec-fetch-mode=cors, sec-fetch-dest=empty
2024-04-20 02:55:29.305 Error DefaultAuthenticationProvider: Invalid username or password. No user named tv exists
2024-04-20 02:55:29.308 Error UserManager: Error authenticating with provider Default
    *** Error Report ***
    Version: 4.8.3.0
    Command line: C:\Users\USERNAME\AppData\Roaming\Emby-Server\system\EmbyServer.dll C:\Users\USERNAME\AppData\Roaming\Emby-Server\system\EmbyServer.dll -noautorunwebapp
    Operating system: Microsoft Windows 10.0.22621
    Framework: .NET 6.0.27
    OS/Process: x64/x64
    Runtime: C:/Users/USERNAME/AppData/Roaming/Emby-Server/system/System.Private.CoreLib.dll
    Processor count: 4
    Data path: C:\Users\USERNAME\AppData\Roaming\Emby-Server
    Application path: C:\Users\USERNAME\AppData\Roaming\Emby-Server\system
    System.Exception: System.Exception: Invalid username or password.
       at Emby.Server.Implementations.Library.DefaultAuthenticationProvider.Authenticate(String username, String password, User resolvedUser)
       at Emby.Server.Implementations.Library.UserManager.AuthenticateWithProvider(IAuthenticationProvider provider, String username, String password, User resolvedUser, CancellationToken cancellationToken)
    Source: Emby.Server.Implementations
    TargetSite: System.Threading.Tasks.Task`1[MediaBrowser.Controller.Authentication.ProviderAuthenticationResult] Authenticate(System.String, System.String, MediaBrowser.Controller.Entities.User)
    
2024-04-20 02:55:29.308 Warn Server: AUTH-ERROR: 176.9.66.26 - Invalid username or password entered.
2024-04-20 02:55:29.308 Error Server: Invalid username or password entered.
2024-04-20 02:55:29.309 Info Server: http/2 Response 401 to ‌‍‍176.9.66.26‌. Time: 4ms. POST https://‌‍‍SERVER_URL.com‌/emby/Users/authenticatebyname?X-Emby-Client=Emby Web&X-Emby-Device-Name=Google Chrome Windows&X-Emby-Device-Id=6f5ca88a-c36c-4da7-b949-5e1beb7d5aa7&X-Emby-Client-Version=4.8.3.0&X-Emby-Language=de
2024-04-20 02:55:43.832 Info Server: http/2 Response 404 to ‌‍‍176.9.66.26‌. Time: 0ms. GET https://‌‍‍SERVER_URL.com‌/Systemn/Info/Public
2024-04-20 02:56:41.338 Info Server: http/1.1 POST https://‌‍‍SERVER_URL.com‌/Users/New?api_key=‌f2bafedd4afc4029950b6365f7d8216c‌. UserAgent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/123.0.0.0 Safari/537.36 X-Middleton/1
2024-04-20 02:56:41.342 Warn Server: AUTH-ERROR: 176.9.66.26 - Access token is invalid or expired.
2024-04-20 02:56:41.342 Error Server: Access token is invalid or expired.
2024-04-20 02:56:41.343 Info Server: http/1.1 Response 401 to ‌‍‍176.9.66.26‌. Time: 4ms. POST https://‌‍‍SERVER_URL.com‌/Users/New?api_key=‌f2bafedd4afc4029950b6365f7d8216c‌
2024-04-20 02:57:52.663 Info Server: http/1.1 POST https://‌‍‍SERVER_URL.com‌/Users/New?api_key=‌18ff6e1756264241a73b659936a04f91‌. UserAgent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/123.0.0.0 Safari/537.36 X-Middleton/1
2024-04-20 02:57:52.720 Info Server: http/1.1 Response 200 to ‌‍‍2001:9e8:34c3:4e00:f023:70:d05e:e328‌. Time: 56ms. POST https://‌‍‍SERVER_URL.com‌/Users/New?api_key=‌18ff6e1756264241a73b659936a04f91‌
2024-04-20 02:58:09.721 Info Server: http/2 POST https://‌‍‍SERVER_URL.com‌/emby/Users/authenticatebyname?X-Emby-Client=Emby Web&X-Emby-Device-Name=Google Chrome Windows&X-Emby-Device-Id=6f5ca88a-c36c-4da7-b949-5e1beb7d5aa7&X-Emby-Client-Version=4.8.3.0&X-Emby-Language=de. Accept=application/json, Host=‌‍‍SERVER_URL.com‌, User-Agent=Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/123.0.0.0 Safari/537.36, :method=POST, Accept-Encoding=gzip, deflate, br, zstd, Accept-Language=de-DE,de;q=0.9,en-DE;q=0.8,en-US;q=0.7,en;q=0.6, Content-Type=application/x-www-form-urlencoded; charset=UTF-8, Origin=‌‍‍https://SERVER_URL.com‌, Referer=‌‍‍https://SERVER_URL.com/web/index.html‌, Content-Length=11, sec-ch-ua="Google Chrome";v="123", "Not:A-Brand";v="8", "Chromium";v="123", DNT=1, sec-ch-ua-mobile=?0, sec-ch-ua-platform="Windows", sec-fetch-site=same-origin, sec-fetch-mode=cors, sec-fetch-dest=empty
2024-04-20 02:58:09.724 Info UserManager: Authentication request for tv has succeeded.
2024-04-20 02:58:09.724 Info SessionManager: Creating new access token for user 17 tv
 

[Q-Droid 653]

They attempted with two different API keys. Have you used your key for anything anywhere?

 

[Happy2Play 8296]

Do you have static api keys on API Keys page?

[kuldan5853 1]

2 minutes ago, Q-Droid said:

They attempted with two different API keys. Have you used your key for anything anywhere?

 

That's the weird part - the other key(s) I don't know. 

The one they got in with was created on my server back in 2022 - I named it "Emby DLNA", so I assume that was me playing around with DLNA devices in my home (and then subsequently forgetting the key exists) 

So no, besides setting it up to test some DLNA stuff I have never used that key for anything. 

[kuldan5853 1]

I'm going to restore my Emby Server from Backup from 2 days ago (I checked the logs of the past days and the probing etc. only started around midnight on the 20th) and will set up nginx...

Based on my reading, it should be possible to set up a subpath block in nginx only allowing access to /web and /videos, which in theory should block API based vulnerabilities/attack vectors? 

I will of course also delete the api key from Emby after the restore and just for good measure I will also create a local only admin account and remove admin rights from my remote-enabled account. 

Anything else I should do to make my setup more secure? 

[kuldan5853 1]

Small update - I have seen several login attempts for the account they created yesterday, as well (after that didn't work) attempts to exploit the same API key to create the user again - this time stopped by an nginx deny rule. 

Fascinating stuff.. 

2024/04/21 19:04:37 [error] 14540#6884: *197 access forbidden by rule, client: 64.227.21.251, server: URL.COM, request: "POST /Users/d682d01351154ba085f6727559764213/Policy?api_key=18ff6e1756264241a73b659936a04f91 HTTP/1.1", host: "URL.COM"

2024/04/21 19:05:01 [error] 14540#6884: *199 access forbidden by rule, client: 64.227.21.251, server: URL.COM, request: "POST /Users?api_key=18ff6e1756264241a73b659936a04f91 HTTP/1.1", host: "URL.COM"
 

[Luke 37109]

Quote

Anything else I should do to make my setup more secure? 

Hi, mainly passwords and SSL but it looks like you've got that now.

[rbjtech 4287]

So API keys obviously don't need passwords - that's the entire point - but of course allowing remote access via an API is a big no-no in the first place.   

@Lukeimo - API keys need a bit of work - a) are api-keys not obfuscated in the logs ?  b) we should have the ability to disable an individual key - currently you can only delete it.   c) remote access via manually created keys (usually created manually for local api access to 3rd party apps only) should not have remote access granted.   You should not have to rely on nginx to block the request.    Maybe you need to do the same as you did for users/passwords and have a 'remote access' column to grant permissions ?

20 hours ago, kuldan5853 said:

Any idea how they could have gotten the API Key?

So were the previous API keys published in logs ?   

2024-04-20 02:57:52.663 Info Server: http/1.1 POST https://‌‍‍SERVER_URL.com‌/Users/New?api_key=‌18ff6e1756264241a73b659936a04f91‌. UserAgent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/123.0.0.0 Safari/537.36 X-Middleton/1

The log above is showing the api_key in plain text ...      If the server URL was known - then a remote request to setup a new user is a simple thing to do ..

 

[kuldan5853 1]

1 minute ago, rbjtech said:

So API keys obviously don't need passwords - that's the entire point - but of course allowing remote access via an API is a big no-no in the first place.   

@Lukeimo - API keys need a bit of work - a) are api-keys not obfuscated in the logs ?  b) we should have the ability to disable an individual key - currently you can only delete it.   c) remote access via manually created keys (usually created manually for local api access to 3rd party apps only) should not have remote access granted.   You should not have to rely on nginx to block the request.    Maybe you need to do the same as you did for users/passwords and have a 'remote access' column to grant permissions ?

So were the previous API keys published in logs ?   

2024-04-20 02:57:52.663 Info Server: http/1.1 POST https://‌‍‍SERVER_URL.com‌/Users/New?api_key=‌18ff6e1756264241a73b659936a04f91‌. UserAgent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/123.0.0.0 Safari/537.36 X-Middleton/1

The log above is showing the api_key in plain text ...      If the server URL was known - then a remote request to setup a new user is a simple thing to do ..

 

Yeah, but the API key itself was only used by me within my network. The question is how did the attacker acquire it from the internet in the first place to use it in the creation of the new user account.. also, they seemingly tried two other API keys first that I don't recognize, so that's pretty weird too. 

But to the rest of your comment, yes I'm kind of shocked that the API was by default exposed to the greater internet / is mixed with paths that emby needs to function normally.. 

[roaku 795]

The DLNA/Upnp element has me curious.

[Luke 37109]

If you download the log file using the web interface with anonymization enabled then you shouldn't see any api keys in there. If you do then please let us know and we'll correct it. Thanks.

[rbjtech 4287]

21 minutes ago, Luke said:

If you download the log file using the web interface with anonymization enabled then you shouldn't see any api keys in there. If you do then please let us know and we'll correct it. Thanks.

What about the other items - should user created api remote access be allowed by default ?

@softworkz- Any thoughts here ?

[darkassassin07 429]

I just went to check this myself, and have found the one and only API key I've created, in plaintext within my anonymized log file.

The key is used for ombi, and is seen in a line beginning with:

Info Server: http/1.1 POST http://[local ip manually redacted]:8096/emby/users/authenticatebyname

(I logged into ombi using an emby user, then triggered a sync between ombi+emby to ensure some activity before grabbing this log)

 

The file is definitely supposed to be anonymized as most lines are something like:

Info Server: http/1.1 Response 401 to host3. Time: 0ms. POST http://emby_remote_ip/Sync/data

 

 

So potentially every log file on this forum, anonymized or not, contains keys into people's servers... (at least those that have api keys in use) Those should probably be removed as well.

Edited April 21 by darkassassin07

[pünktchen 1259]

4 minutes ago, rbjtech said:

What about the other items - should user created api remote access be allowed by default ?

@softworkz- Any thoughts here ?

It's not only about remote access or not. The whole api key thing needs a granular control similar to user accounts - server management, item deletion, library access, remote access. That it needs work was even acknowledged by the devs multiple times over several years. But as is usual in the Emby world, that's all we get - at most an acknowledgment of the problem, but not a solution.

[Luke 37109]

13 minutes ago, darkassassin07 said:

I just went to check this myself, and have found the one and only API key I've created, in plaintext within my anonymized log file.

The key is used for ombi, and is seen in a line beginning with:

Info Server: http/1.1 POST http://[local ip manually redacted]:8096/emby/users/authenticatebyname

(I logged into ombi using an emby user, then triggered a sync between ombi+emby to ensure some activity before grabbing this log)

 

The file is definitely supposed to be anonymized as most lines are something like:

Info Server: http/1.1 Response 401 to host3. Time: 0ms. POST http://emby_remote_ip/Sync/data

 

 

So potentially every log file on this forum, anonymized or not, contains keys into people's servers... (at least those that have api keys in use) Those should probably be removed as well.

Can you please provide the full log line, with data redacted of course. Thanks.

[rbjtech 4287]

9 minutes ago, pünktchen said:

It's not only about remote access or not. The whole api key thing needs a granular control similar to user accounts - server management, item deletion, library access, remote access. That it needs work was even acknowledged by the devs multiple times over several years. But as is usual in the Emby world, that's all we get - at most an acknowledgment of the problem, but not a solution.

1 hour ago, rbjtech said:

So API keys obviously don't need passwords - that's the entire point - but of course allowing remote access via an API is a big no-no in the first place.   

@Lukeimo - API keys need a bit of work - a) are api-keys not obfuscated in the logs ?  b) we should have the ability to disable an individual key - currently you can only delete it.   c) remote access via manually created keys (usually created manually for local api access to 3rd party apps only) should not have remote access granted.   You should not have to rely on nginx to block the request.    Maybe you need to do the same as you did for users/passwords and have a 'remote access' column to grant permissions ?

 

 

Yep - I allude to that above

Agree 100% - and tbh why I copied in softworkz as no doubt you are aware that he was leading the previous security incident investigation.

If api keys are being scraped from the forums - then I expect to see many more posts on this issue.

Not allowing any manually created API keys remote access seems to be a sensible first step ..  If system api keys are out there, then emby have a problem ...

Edited April 21 by rbjtech

[darkassassin07 429]

2024-04-21 14:37:18.839 Info Server: http/1.1 POST http://[local ip redacted]:8096/emby/users/authenticatebyname. Accept=application/json, Host=host5, User-Agent=Ombi/4.43.5 (https://ombi.io/), Content-Type=application/json, traceparent=00-114ce0e854ef0b69f95a5d87fbac2722-d25a3dfb2323e38d-00, Content-Length=51, X-Emby-Authorization=MediaBrowser Client="Ombi", Device="Ombi", DeviceId="v3", Version="v3", X-MediaBrowser-Token=[api key redacted], Device=Ombi, X-Forwarded-For=host1

 

[Luke 37109]

10 minutes ago, darkassassin07 said:

2024-04-21 14:37:18.839 Info Server: http/1.1 POST http://[local ip redacted]:8096/emby/users/authenticatebyname. Accept=application/json, Host=host5, User-Agent=Ombi/4.43.5 (https://ombi.io/), Content-Type=application/json, traceparent=00-114ce0e854ef0b69f95a5d87fbac2722-d25a3dfb2323e38d-00, Content-Length=51, X-Emby-Authorization=MediaBrowser Client="Ombi", Device="Ombi", DeviceId="v3", Version="v3", X-MediaBrowser-Token=[api key redacted], Device=Ombi, X-Forwarded-For=host1

 

Thanks.

[C.S. 62]

2 hours ago, rbjtech said:

If system api keys are out there, then emby have a problem ...

Questions for anyone:

- As someone who has never created an api key, if I've uploaded a log here which contains a reference to one, then that would definitely be my system api. Correct?

- Assuming the answer is yes, how do I get a new api key, short of rebuilding from scratch?

Edit:

Ok I just checked and I see there are different api keys all over the place in the logs, so I don't know enough to even ask questions. Carry on, everyone, but sooner or later somebody needs to let us laymen know how to deal with this.

 

Edited April 22 by C.S.

[kuldan5853 1]

I just checked for both forum accounts I have - I have never posted a log with an API key in it on the forum before posting this, so I'm still baffled how they got the API key in the first place to get in to my environment.. 

[rdhardi 19]

In the Dashboard, under Api Keys , I had an API key that I didn't create. Was that created by the system? I deleted it, but wondering where it came from? Will it come back after reboots?

Also, as C.S. said, there are API keys all over my log. What are these?!

1 hour ago, C.S. said:

Ok I just checked and I see there are different api keys all over the place in the logs, so I don't enough to even ask questions. Carry on, everyone, but sooner or later somebody needs to let us laymen know how to deal with this.

 

 

Edited April 22 by rdhardi

[softworkz 3340]

@kuldan5853 Are you affiliated with a 4-letter company starting with 'O' from a city starting with 'N'?

If the answer is yes, then there might be a bigger problem..

[rbjtech 4287]

10 hours ago, C.S. said:

Questions for anyone:

- As someone who has never created an api key, if I've uploaded a log here which contains a reference to one, then that would definitely be my system api. Correct?

- Assuming the answer is yes, how do I get a new api key, short of rebuilding from scratch?

Edit:

Ok I just checked and I see there are different api keys all over the place in the logs, so I don't know enough to even ask questions. Carry on, everyone, but sooner or later somebody needs to let us laymen know how to deal with this.

 

API Keys/Tokens are uniquely generated every time a client logs on.   Unless you 'logout' that token for that client is retained.    This in itself is not a problem because a) it should be transmitted over https and b) it's retained in the system logs behind authentication.    If you create a manual API key for talking to 3rd party apps then that token does not change.    Unfortunately, the manually generated tokens have full 'admin' privileges - and thus IF made available, can do anything remotely that you could normally do via the UI (and more).    So - if you have any manually created API Keys that you have not created yourself, then this is bad - and they should be reviewed/removed asap and user/passwords changed asap to force client logouts (and new tokens).   Hygiene of admin accounts should always include API keys - they are effectively 'backdoors' into systems and are actually more of a concern than 'users' as they don't appear on dashboards etc.    The part I'm unclear on myself (need to test this) is if a client api key/token has these same full privileges - if yes, then it has no less privileges than a manually created api key...           Thus anybody that has posted any API key in logs on the forum that have not been 'anonymised' (default setting) has a potential problem IF that key/token is still valid.

[PaulE123 18]

2 hours ago, rbjtech said:

API Keys/Tokens are uniquely generated every time a client logs on.   Unless you 'logout' that token for that client is retained.    This in itself is not a problem because a) it should be transmitted over https and b) it's retained in the system logs behind authentication.    If you create a manual API key for talking to 3rd party apps then that token does not change.    Unfortunately, the manually generated tokens have full 'admin' privileges - and thus IF made available, can do anything remotely that you could normally do via the UI (and more).    So - if you have any manually created API Keys that you have not created yourself, then this is bad - and they should be reviewed/removed asap and user/passwords changed asap to force client logouts (and new tokens).   Hygiene of admin accounts should always include API keys - they are effectively 'backdoors' into systems and are actually more of a concern than 'users' as they don't appear on dashboards etc.    The part I'm unclear on myself (need to test this) is if a client api key/token has these same full privileges - if yes, then it has no less privileges than a manually created api key...           Thus anybody that has posted any API key in logs on the forum that have not been 'anonymised' (default setting) has a potential problem IF that key/token is still valid.

Many thanks for the explanation rbjtech, it makes sense. What I don't understand is how @kuldan5853fell victim to this without exposing his API key in a log/forum post....

[rbjtech 4287]

34 minutes ago, PaulE123 said:

Many thanks for the explanation rbjtech, it makes sense. What I don't understand is how @kuldan5853fell victim to this without exposing his API key in a log/forum post....

Unfortunately, that cannot comprehensively be answered - there are too many variables.   The danger here is the fact (I *think*) that any API/Token can be used for other things for which it was not intended.  I won't go into details on here, but needless to say, a little more info from the Dev's would be welcome.

Edited April 22 by rbjtech