Failed logins from 127.0.0.1?

[zefritz 0]

I noticed there were a bunch of failed logins to my emby server early this morning, from 127.0.0.1?  What could cause this? 

My emby is running on a minimal debian installation, no desktop/gui, no browsers, no links/elinks/lynx, etc..  The only thing running on it is emby. My first thought was that someone exploited the emby to get local access, because only the http/https ports for Emby are open.  My other thought is that maybe it's a webcrawler of some kind that stumbled onto it? 

Whatever it was, it tried to login to every user, so it was able to enumerate the users despite the "hide this user from login screens when connecting remotely" box being checked for all my users (except one, [[USER2]] in the output below, that's a user with no password that anyone can connect too, assuming anyone is interested in pictures and videos of my cat)

Can someone help me make sense of this?  I see this in the logs, does this look like some kind of web crawler?  I'm assuming "X-Forwarded-For=127.0.0.1" is what lets the request appear to be coming locally, and somehow it was able to use this to view the list of users?

 

 

The logs are full of this kind of thing:

2023-04-19 02:08:41.355 Info Server: http/1.1 POST http://host1/emby/Users/authenticatebyname?X-Emby-Client=Emby%20Web&X-Emby-Device-Name=Chrome%20Windows&X-Emby-Device-Id=8256f619-fcd8-4620-a447-faa13ffc6b5a&X-Emby-Client-Version=4.7.2.0&X-Emby-Language=en-us. Accept=*/*, Host=[[MYIP]], User-Agent=Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/538.21 (KHTML, like Gecko) Chrome/113.0.0.0 Safari/538.21 Edg/113.0.1211.42, Accept-Encoding=gzip, deflate, Content-Type=application/x-www-form-urlencoded, Content-Length=18, X-Forwarded-For=127.0.0.1
2023-04-19 02:08:42.074 Error UserManager: Error authenticating with provider Default
*** Error Report ***
Version: 4.7.11.0
Command line: /opt/emby-server/system/EmbyServer.dll -programdata /var/lib/emby -ffdetect /opt/emby-server/bin/ffdetect -ffmpeg /opt/emby-server/bin/ffmpeg -ffprobe /opt/emby-server/bin/ffprobe -restartexitcode 3 -updatepackage emby-server-deb_{version}_amd64.deb
Operating system: Linux version 4.19.0-23-amd64 (debian-kernel@lists.debian.org) (gcc version 8.3.0 (Debian 8.3.0-6)) #1 SMP Debian 4.19.269-1 (2022-12-20)
Framework: .NET 6.0.8
OS/Process: x64/x64
Runtime: opt/emby-server/system/System.Private.CoreLib.dll
Processor count: 8
Data path: /var/lib/emby
Application path: /opt/emby-server/system
System.Exception: System.Exception: Invalid username or password..
at Emby.Server.Implementations.Library.DefaultAuthenticationProvider.Authenticate(String username, String password, User resolvedUser)
at Emby.Server.Implementations.Library.UserManager.AuthenticateWithProvider(IAuthenticationProvider provider, String username, String password, User resolvedUser, CancellationToken cancellationToken)
Source: Emby.Server.Implementations
TargetSite: System.Threading.Tasks.Task`1[MediaBrowser.Controller.Authentication.ProviderAuthenticationResult] Authenticate(System.String, System.String, MediaBrowser.Controller.Entities.User)
2023-04-19 02:08:42.074 Info HttpClient: POST https://connect.emby.media/service/user/authenticate
2023-04-19 02:08:42.601 Info UserManager: Authentication request for [[USER1]] has been denied.
2023-04-19 02:08:42.670 Warn Server: AUTH-ERROR: 127.0.0.1 - Invalid username or password entered.
2023-04-19 02:08:42.670 Error Server: Invalid username or password entered.
2023-04-19 02:08:42.670 Info Server: http/1.1 Response 401 to 127.0.0.1. Time: 1315ms. http://host1/emby/Users/authenticatebyname?X-Emby-Client=Emby%20Web&X-Emby-Device-Name=Chrome%20Windows&X-Emby-Device-Id=8256f619-fcd8-4620-a447-faa13ffc6b5a&X-Emby-Client-Version=4.7.2.0&X-Emby-Language=en-us
2023-04-19 02:08:43.257 Info Server: http/1.1 POST http://host1/emby/Users/authenticatebyname?X-Emby-Client=Emby%20Web&X-Emby-Device-Name=Chrome%20Windows&X-Emby-Device-Id=8256f619-fcd8-4620-a447-faa13ffc6b5a&X-Emby-Client-Version=4.7.2.0&X-Emby-Language=en-us. Accept=*/*, Host=[[MYIP]], User-Agent=Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/538.21 (KHTML, like Gecko) Chrome/113.0.0.0 Safari/538.21 Edg/113.0.1211.42, Accept-Encoding=gzip, deflate, Content-Type=application/x-www-form-urlencoded, Content-Length=19, X-Forwarded-For=127.0.0.1
2023-04-19 02:08:43.277 Info UserManager: Authentication request for [[USER2]] has succeeded.
2023-04-19 02:08:43.277 Info SessionManager: Creating new access token for user 11
2023-04-19 02:08:43.365 Info Server: http/1.1 Response 200 to 127.0.0.1. Time: 109ms. http://host1/emby/Users/authenticatebyname?X-Emby-Client=Emby%20Web&X-Emby-Device-Name=Chrome%20Windows&X-Emby-Device-Id=8256f619-fcd8-4620-a447-faa13ffc6b5a&X-Emby-Client-Version=4.7.2.0&X-Emby-Language=en-us
2023-04-19 02:08:43.912 Info Server: http/1.1 POST http://host1/emby/Users/authenticatebyname?X-Emby-Client=Emby%20Web&X-Emby-Device-Name=Chrome%20Windows&X-Emby-Device-Id=8256f619-fcd8-4620-a447-faa13ffc6b5a&X-Emby-Client-Version=4.7.2.0&X-Emby-Language=en-us. Accept=*/*, Host=[[MYIP]], User-Agent=Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/538.21 (KHTML, like Gecko) Chrome/113.0.0.0 Safari/538.21 Edg/113.0.1211.42, Accept-Encoding=gzip, deflate, Content-Type=application/x-www-form-urlencoded, Content-Length=20, X-Forwarded-For=127.0.0.1
2023-04-19 02:08:43.913 Error UserManager: Error authenticating with provider Default
*** Error Report ***
Version: 4.7.11.0
Command line: /opt/emby-server/system/EmbyServer.dll -programdata /var/lib/emby -ffdetect /opt/emby-server/bin/ffdetect -ffmpeg /opt/emby-server/bin/ffmpeg -ffprobe /opt/emby-server/bin/ffprobe -restartexitcode 3 -updatepackage emby-server-deb_{version}_amd64.deb
Operating system: Linux version 4.19.0-23-amd64 (debian-kernel@lists.debian.org) (gcc version 8.3.0 (Debian 8.3.0-6)) #1 SMP Debian 4.19.269-1 (2022-12-20)
Framework: .NET 6.0.8
OS/Process: x64/x64
Runtime: opt/emby-server/system/System.Private.CoreLib.dll
Processor count: 8

 

Edited April 20, 2023 by zefritz
formatting

[Abobader 3464]

Hello zefritz,

** This is an auto reply **

Please wait for someone from staff support or our members to reply to you.

It's recommended to provide more info, as it explain in this thread:

Thank you.

Emby Team

[ebr 16176]

Hi.  Did you install a plugin or other component somewhere that tries to interface with Emby?

[Q-Droid 989]

Put up a reverse proxy. I think your diagnosis is correct and there's a known exploit to impersonate local connections. You might want to disable remote access until your RP is up and running. It took me around 30 minutes to spin up a Caddy instance when I learned about this exploit. Tweaking took more time but was already protected and the tweaks were for logging and extra headers. 

I don't know if this hole was fixed yet. Might have been for the current beta. 

 

Edited April 20, 2023 by Q-Droid

[zefritz 0]

16 minutes ago, ebr said:

Hi.  Did you install a plugin or other component somewhere that tries to interface with Emby?

No plugins that I'm aware of.  I only use Emby Theater on my PC and phone, and while I have accounts for my family and a few friends none of them actually ever use it, only my Mom and brother at home and they use Emby Theater too, and they wouldn't know how to do something like that. On the server side I don't believe I've installed any additional plugins from the default set. 

 

[zefritz 0]

12 minutes ago, Q-Droid said:

Put up a reverse proxy. I think your diagnosis is correct and there's a known exploit to impersonate local connections. You might want to disable remote access until your RP is up and running. It took me around 30 minutes to spin up a Caddy instance when I learned about this exploit. Tweaking took more time but was already protected and the tweaks were for logging and a extra headers. 

I don't know if this hole was fixed yet. Might have been for the current beta. 

 

Thanks Q-Droid, I'll look into the reverse proxy further this evening. How would this help though?  I just skimmed through this link and I'm not sure I understand how it will aid my situation. Better logging capability via Caddy?  I have encryption working on my Emby server and use my own domain name to connect.

[Q-Droid 989]

Read this thread and the other one referenced a few posts in.  

tl;dr - the reverse proxy will set the correct value for that header and prevent the spoofing. 

https://emby.media/community/index.php?/topic/111836-where-do-i-report-a-security-vulnerability/

[rbjtech 5284]

2 hours ago, Q-Droid said:

Put up a reverse proxy. I think your diagnosis is correct and there's a known exploit to impersonate local connections. You might want to disable remote access until your RP is up and running. It took me around 30 minutes to spin up a Caddy instance when I learned about this exploit. Tweaking took more time but was already protected and the tweaks were for logging and extra headers. 

I don't know if this hole was fixed yet. Might have been for the current beta. 

 

It was raised over 7 months ago - so one would hope a security vulnerability of this significance made it's way into the main release but looking at the OP's post - I'm not so sure ...

Possible of more significance is how the unauthenticated web query got the emby user list ?

[Q-Droid 989]

It's the normal LAN side landing page. The spoof makes it look like you're local and most have users viewable internally. 

It wouldn't take much to script finding open emby servers, spoof and scrape the page then run through logins. Unauthenticated admin accounts can make it worse. 

 

[rbjtech 5284]

4 minutes ago, Q-Droid said:

It's the normal LAN side landing page. The spoof makes it look like you're local and most have users viewable internally. 

It wouldn't take much to script finding open emby servers, spoof and scrape the page then run through logins. Unauthenticated admin accounts can make it worse. 

 

Of course - thanks - so emby is just being spoofed into thinking it local. 

Oh dear - when are Emby gonna take Remote Security seriously ...  @softworkzimplied this was fixed on the original discovery thread ?

Looking at my nginx log,  it's actually interesting to see what the scanners are doing .. I've got some really weird web requests hitting my emby url - so I know they are not just general scans ...

[Q-Droid 989]

I hope I'm wrong and that it was fixed already. If not in stable at least the current beta release. 

[softworkz 5066]

@zefritz- The interesting question is: How did you make your Emby server accessible from outside? How did you publish it to the internet?

Your log is showing that the request is actually coming from 127.0.0.1 (at the network level), so the question is how does that come? It would not be like that when you would have published the server through a router (which is the typical case).

When you don't have a reverse proxy and are publishing your server through a router, then the X-FORWARDED-FOR header doesn't have any effect.

The only case where Emby is taken X-FORWARDED-FOR and X-REALIP headers into account is when the request at the network level is coming from the local machine.
In that case, Emby makes the assumption that you are running a reverse proxy (with proper configuration) and only then it is using those headers.

Maybe that (automatic internal) assumption is a bit too keen. The question is though: Why does the request come from 127.0.0.1 even though you don't have a reverse proxy in place?

Edited April 20, 2023 by softworkz

[softworkz 5066]

15 minutes ago, softworkz said:

Maybe that (automatic internal) assumption is a bit too keen.

Seems to be too much of "automagic". There are other scenarios possible where requests are coming in through local loopback (virtualization, software routers, local VPN software) - so that's actually not a suitable condition to assume the presence of a reverse proxy.

Probably we should have an option like "I am publishing Emby through a Reverse Proxy", Description "Enabling this will make Emby trust HTTP headers indicating remote and local forwarded endpoints" - or something similar.

Edited April 20, 2023 by softworkz

[rbjtech 5284]

31 minutes ago, softworkz said:

Seems to be too much of "automagic". There are other scenarios possible where requests are coming in through local loopback (virtualization, software routers, local VPN software) - so that's actually not a suitable condition to assume the presence of a reverse proxy.

Probably we should have an option like "I am publishing Emby through a Reverse Proxy", Description "Enabling this will make Emby trust HTTP headers indicating remote and local forwarded endpoints" - or something similar.

Does this not exist already ?

In Admin | Server | Network

[Q-Droid 989]

1 hour ago, softworkz said:

@zefritz- The interesting question is: How did you make your Emby server accessible from outside? How did you publish it to the internet?

Your log is showing that the request is actually coming from 127.0.0.1 (at the network level), so the question is how does that come? It would not be like that when you would have published the server through a router (which is the typical case).

When you don't have a reverse proxy and are publishing your server through a router, then the X-FORWARDED-FOR header doesn't have any effect.

The only case where Emby is taken X-FORWARDED-FOR and X-REALIP headers into account is when the request at the network level is coming from the local machine.
In that case, Emby makes the assumption that you are running a reverse proxy (with proper configuration) and only then it is using those headers.

Maybe that (automatic internal) assumption is a bit too keen. The question is though: Why does the request come from 127.0.0.1 even though you don't have a reverse proxy in place?

A typical non-technical user who wants to allow remote access to their server would check the box in Network Settings and forward the ports through their router. And that's it, they're done - as unfortunate as this may be.  Some don't even manage their router and rely on UPnP Port Mapper plug-in to do this for them, also unfortunate.

This is all it takes. The current stable release (4.7.11) is respecting the "x-forwarded-for" header and acting accordingly. So by spoofing the header I can gain access to a server as a local LAN user and all of the perks that offers - depending on what you consider "perks".

 

[softworkz 5066]

39 minutes ago, rbjtech said:

Does this not exist already ?

In Admin | Server | Network

This is about SSL, not about headers, but maybe it could be combined with that.

[softworkz 5066]

20 minutes ago, Q-Droid said:

A typical non-technical user who wants to allow remote access to their server would check the box in Network Settings and forward the ports through their router. And that's it, they're done - as unfortunate as this may be.  Some don't even manage their router and rely on UPnP Port Mapper plug-in to do this for them, also unfortunate.

This is all it takes. The current stable release (4.7.11) is respecting the "x-forwarded-for" header and acting accordingly. So by spoofing the header I can gain access to a server as a local LAN user and all of the perks that offers - depending on what you consider "perks".

Are you sure about this? From looking at the code, "x-forwarded-for" is only used when the connection originates from the server itself (locally) but not when it comes from an external endpoint through a router.

Edited April 20, 2023 by softworkz

[Q-Droid 989]

14 minutes ago, softworkz said:

Are you sure about this? From looking at the code, "x-forwarded-for" is only used when the connection originates from the server itself (locally) but not when it comes from an external endpoint through a router.

I'm pretty sure and have tested it. When I learned about this from the forums then tested I immediately spun up a reverse proxy. This is a bad one in my opinion. 

[rbjtech 5284]

48 minutes ago, softworkz said:

This is about SSL, not about headers, but maybe it could be combined with that.

Yep - aware of that  - I'm suggesting it gets combined or dual purposed with this 'setting'.. 

[softworkz 5066]

54 minutes ago, Q-Droid said:

I'm pretty sure and have tested it. When I learned about this from the forums then tested I immediately spun up a reverse proxy. This is a bad one in my opinion. 

Do you have a chance to test this with the beta server?

[Q-Droid 989]

1 hour ago, softworkz said:

Do you have a chance to test this with the beta server?

I can reproduce it on 4.8.0.21 beta which appears to be the latest available on Docker.

The versioning was my bad. I pulled 4.8.0.29 beta and can still reproduce.

 

Edited April 20, 2023 by Q-Droid

[softworkz 5066]

14 minutes ago, Q-Droid said:

I can reproduce it on 4.8.0.21 beta which appears to be the latest available on Docker.

The versioning was my bad. I pulled 4.8.0.29 beta and can still reproduce.

Could you please show a few log lines like above (http request and response)?

[Q-Droid 989]

Attached.

 

embyserver-test-scrub.txt

[Q-Droid 989]

Adding one with debug enabled.

 

embyserver-test2-debug.txt

[softworkz 5066]

Thanks. I can see that the request is coming from 127.0.0.1 - not from outside through a router?