the http server is inaccessible.

[igeoorge 26]

good night friends,

Since I use the emby, I have a problem with service drops.
On days with a lot of use the http server goes down frequently

The embyserver service is still active on the VPS, however, the http server is inaccessible.

I'm leaving the logs attached, please help me with this problem.

I've been trying to find out what's been going on for over 1 year.

https://www.mediafire.com/file/56jmmf2cc4mexif/embyserver.txt/file

[Carlo 4561]

Hi, Can you still ping the server?

[igeoorge 26]

Hey friend,

Sorry for the delay in responding.

When it falls, it doesn't ping.

I also use uptimerobot.com website for monitoring and the website also reports the fall.

Please help me resolve this.

The problem is because emby doesn't support many simultaneous users?

[Carlo 4561]

I would check your network/NIC drivers.

If it's not a driver issue I would try adding a different NIC card to use for networking as it sounds like the NIC has stopped working.

[igeoorge 26]

Hey friend,

Thanks for the feedback.

I'll check this question and get back to you soon with news

[igeoorge 26]

Hey friend,

I asked VPS support to check for possible problems with the drive and the network card.

After analysis by VPS support, no network card problems were found, but VPS attacks were identified.

The VPS firewall has been enabled and we are monitoring to see if the crashes will continue.

So far, thanks for the help. When I have news, I'll be back in contact

[Luke 42078]

Let us know if you find anything. Thanks.

[igeoorge 26]

Hi Luke,

The network card of the VPS was changed and a full reset of the machine was also done.

I'm currently waiting for traffic to increase so I can analyze it.

I leave today's logs attached.

embyserver.txt

[Carlo 4561]

That's good.  Let's see if this makes a difference now.

[igeoorge 26]

Hi friends,

Unfortunately changing the network card and restarting the VPS didn't solve the problem.

I'm leaving the logs from last night, 25/08.

In the night period is the period that the server usually goes down.

When the emby stops responding, VPS access remains normal.

Please help me resolve.

https://minohost.com.br/embyserver-63765532800.txt

[Luke 42078]

How many users connect to this server?

[igeoorge 26]

Hi Luke,

Today I have around 500 friends who use it recurrently.

All with direct reproduction, we do not transcode.

Do you have something in mind that might help me?

[igeoorge 26]

Gentlemen,

I still have the problem.

Do you think the problem might be the number of users?

I'm leaving the log attached.

Please help me.

 

https://www.mediafire.com/file/6grlqowrlz5nnth/embyserver-63766042836.txt/file

[Carlo 4561]

I don't see any issues in the log.

What it looks like to me is the NIC shutting down and traffic just stops.

[igeoorge 26]

Friends, I'm bringing new logs attached.

Please review these logs, maybe we can find the problem.

It only occurs when I have a lot of access

 

https://www.mediafire.com/file/lkwkbef8n32a5ir/embyserver-63766655931.txt/file

[Carlo 4561]

At the very end of this log file is a couple errors similar to:

2021-09-07 23:58:50.225 Error Server: Error processing request
    *** Error Report ***
    Version: 4.6.4.0
    Command line: /opt/emby-server/system/EmbyServer.dll -programdata /var/lib/emby-server -ffdetect /opt/emby-server/bin/ffdetect -ffmpeg /opt/emby-server/bin/ffmpeg -ffprobe /opt/emby-server/bin/ffprobe -restartexitcode 3 -updatepackage emby-server-rpm_{version}_x86_64.rpm
    Operating system: Linux version 3.10.0-1160.21.1.el7.x86_64 (mockbuild@kbuilder.bsys.centos.org) (gcc version 4.8.5 20150623 (Red Hat 4.8.5-44) (GCC) ) #1 SMP Tue Mar 1
    Framework: .NET Core 3.1.13
    OS/Process: x64/x64
    Runtime: opt/emby-server/system/System.Private.CoreLib.dll
    Processor count: 12
    Data path: /var/lib/emby-server
    Application path: /opt/emby-server/system
    Microsoft.AspNetCore.Server.Kestrel.Core.BadHttpRequestException: Microsoft.AspNetCore.Server.Kestrel.Core.BadHttpRequestException: Unexpected end of request content.
       at Microsoft.AspNetCore.Server.Kestrel.Core.BadHttpRequestException.Throw(RequestRejectionReason reason)
       at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Http.Http1ContentLengthMessageBody.ReadAsyncInternal(CancellationToken cancellationToken)
       at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Http.HttpRequestStream.CopyToAsyncInternal(Stream destination, CancellationToken cancellationToken)
       at ServiceStack.StreamExtensions.CopyToNewMemoryStreamAsync(Stream stream) in C:\BuildAgent\work\912418dcce86a188\src\ServiceStack.Text\StreamExtensions.cs:line 517
       at ServiceStack.Memory.NetCoreMemory.DeserializeAsync(Stream stream, Type type, DeserializeStringSpanDelegate deserializer) in C:\BuildAgent\work\912418dcce86a188\src\ServiceStack.Text\NetCoreMemory.cs:line 169
       at Emby.Server.Implementations.Services.ServiceHandler.CreateRequest(HttpListenerHost host, IRequest httpReq, RestPath restPath)
       at Emby.Server.Implementations.Services.ServiceHandler.ProcessRequestAsync(HttpListenerHost appHost, IRequest httpReq, IResponse httpRes, RestPath restPath, String responseContentType, CancellationToken cancellationToken)
       at Emby.Server.Implementations.HttpServer.HttpListenerHost.RequestHandler(IRequest httpReq, ReadOnlyMemory`1 urlString, ReadOnlyMemory`1 localPath, CancellationToken cancellationToken)
    Source: Microsoft.AspNetCore.Server.Kestrel.Core
    TargetSite: Void Throw(Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Http.RequestRejectionReason)

@Luke Have you seen this before?

[igeoorge 26]

thanks for reviewing the logs.

I'll wait for @Luke response too

[igeoorge 26]

Friends @cayars @Luke

I continue with the problem of falls.

I'm attaching more logs so they can be analyzed.

Please I need help

Is there anything I can do?

 

https://www.mediafire.com/file/7p7f352w0xgqfgx/embyserver.txt/file

[igeoorge 26]

    *** Error Report ***
    Version: 4.6.4.0
    Command line: /opt/emby-server/system/EmbyServer.dll -programdata /var/lib/emby-server -ffdetect /opt/emby-server/bin/ffdetect -ffmpeg /opt/emby-server/bin/ffmpeg -ffprobe /opt/emby-server/bin/ffprobe -restartexitcode 3 -updatepackage emby-server-rpm_{version}_x86_64.rpm
    Operating system: Linux version 3.10.0-1160.21.1.el7.x86_64 (mockbuild@kbuilder.bsys.centos.org) (gcc version 4.8.5 20150623 (Red Hat 4.8.5-44) (GCC) ) #1 SMP Tue Mar 1
    Framework: .NET Core 3.1.13
    OS/Process: x64/x64
    Runtime: opt/emby-server/system/System.Private.CoreLib.dll
    Processor count: 12
    Data path: /var/lib/emby-server
    Application path: /opt/emby-server/system
    System.IO.FileNotFoundException: System.IO.FileNotFoundException: Could not find file '/opt/emby-server/system/dashboard-ui/bg.jpg'.
    File name: '/opt/emby-server/system/dashboard-ui/bg.jpg'
       at Interop.ThrowExceptionForIoErrno(ErrorInfo errorInfo, String path, Boolean isDirectory, Func`2 errorRewriter)
       at Microsoft.Win32.SafeHandles.SafeFileHandle.Open(String path, OpenFlags flags, Int32 mode)
       at System.IO.FileStream..ctor(String path, FileMode mode, FileAccess access, FileShare share, Int32 bufferSize, FileOptions options)
       at Emby.Server.Implementations.IO.ManagedFileSystem.GetFileStream(String path, FileOpenMode mode, FileAccessMode access, FileShareMode share, Int32 bufferSize, FileOpenOptions fileOpenOptions)
       at Emby.Server.Implementations.IO.ManagedFileSystem.GetFileStream(String path, FileOpenMode mode, FileAccessMode access, FileShareMode share, FileOpenOptions fileOpenOptions)
       at Emby.Server.Implementations.IO.IOManager.GetResponse(HttpRequestOptions options, MediaProtocol protocol)
       at Emby.Server.Implementations.IO.IOManager.GetResponse(HttpRequestOptions options)
       at Emby.Server.Implementations.HttpServer.HttpResultFactory.GetContent(StaticFileResultOptions options, Int64 offset, Int64 length, CancellationToken cancellationToken)
       at Emby.Server.Implementations.HttpServer.FileWriter.SetContentResponseHeaders(Nullable`1 reportedTotalContentLength, CancellationToken cancellationToken)
       at Emby.Server.Implementations.HttpServer.HttpResultFactory.GetStaticResult(IRequest requestContext, StaticResultOptions options)
       at Emby.Web.Api.DashboardService.Get(GetDashboardResource request)
       at Emby.Server.Implementations.Services.ServiceController.GetTaskResult(Task task)
       at Emby.Server.Implementations.Services.ServiceHandler.ProcessRequestAsync(HttpListenerHost appHost, IRequest httpReq, IResponse httpRes, RestPath restPath, String responseContentType, CancellationToken cancellationToken)
       at Emby.Server.Implementations.HttpServer.HttpListenerHost.RequestHandler(IRequest httpReq, ReadOnlyMemory`1 urlString, ReadOnlyMemory`1 localPath, CancellationToken cancellationToken)
    Source: System.Private.CoreLib
    TargetSite: Void ThrowExceptionForIoErrno(ErrorInfo, System.String, Boolean, System.Func`2[Interop+ErrorInfo,Interop+ErrorInfo])

[igeoorge 26]

I have not seen it. It's in the logs.

Does that mean something?

[Happy2Play 9780]

5 minutes ago, igeoorge said:

    *** Error Report ***
    Version: 4.6.4.0
    Command line: /opt/emby-server/system/EmbyServer.dll -programdata /var/lib/emby-server -ffdetect /opt/emby-server/bin/ffdetect -ffmpeg /opt/emby-server/bin/ffmpeg -ffprobe /opt/emby-server/bin/ffprobe -restartexitcode 3 -updatepackage emby-server-rpm_{version}_x86_64.rpm
    Operating system: Linux version 3.10.0-1160.21.1.el7.x86_64 (mockbuild@kbuilder.bsys.centos.org) (gcc version 4.8.5 20150623 (Red Hat 4.8.5-44) (GCC) ) #1 SMP Tue Mar 1
    Framework: .NET Core 3.1.13
    OS/Process: x64/x64
    Runtime: opt/emby-server/system/System.Private.CoreLib.dll
    Processor count: 12
    Data path: /var/lib/emby-server
    Application path: /opt/emby-server/system
    System.IO.FileNotFoundException: System.IO.FileNotFoundException: Could not find file '/opt/emby-server/system/dashboard-ui/bg.jpg'.
    File name: '/opt/emby-server/system/dashboard-ui/bg.jpg'
       at Interop.ThrowExceptionForIoErrno(ErrorInfo errorInfo, String path, Boolean isDirectory, Func`2 errorRewriter)
       at Microsoft.Win32.SafeHandles.SafeFileHandle.Open(String path, OpenFlags flags, Int32 mode)
       at System.IO.FileStream..ctor(String path, FileMode mode, FileAccess access, FileShare share, Int32 bufferSize, FileOptions options)
       at Emby.Server.Implementations.IO.ManagedFileSystem.GetFileStream(String path, FileOpenMode mode, FileAccessMode access, FileShareMode share, Int32 bufferSize, FileOpenOptions fileOpenOptions)
       at Emby.Server.Implementations.IO.ManagedFileSystem.GetFileStream(String path, FileOpenMode mode, FileAccessMode access, FileShareMode share, FileOpenOptions fileOpenOptions)
       at Emby.Server.Implementations.IO.IOManager.GetResponse(HttpRequestOptions options, MediaProtocol protocol)
       at Emby.Server.Implementations.IO.IOManager.GetResponse(HttpRequestOptions options)
       at Emby.Server.Implementations.HttpServer.HttpResultFactory.GetContent(StaticFileResultOptions options, Int64 offset, Int64 length, CancellationToken cancellationToken)
       at Emby.Server.Implementations.HttpServer.FileWriter.SetContentResponseHeaders(Nullable`1 reportedTotalContentLength, CancellationToken cancellationToken)
       at Emby.Server.Implementations.HttpServer.HttpResultFactory.GetStaticResult(IRequest requestContext, StaticResultOptions options)
       at Emby.Web.Api.DashboardService.Get(GetDashboardResource request)
       at Emby.Server.Implementations.Services.ServiceController.GetTaskResult(Task task)
       at Emby.Server.Implementations.Services.ServiceHandler.ProcessRequestAsync(HttpListenerHost appHost, IRequest httpReq, IResponse httpRes, RestPath restPath, String responseContentType, CancellationToken cancellationToken)
       at Emby.Server.Implementations.HttpServer.HttpListenerHost.RequestHandler(IRequest httpReq, ReadOnlyMemory`1 urlString, ReadOnlyMemory`1 localPath, CancellationToken cancellationToken)
    Source: System.Private.CoreLib
    TargetSite: Void ThrowExceptionForIoErrno(ErrorInfo, System.String, Boolean, System.Func`2[Interop+ErrorInfo,Interop+ErrorInfo])

The bg.jpg is the background image for the blueradiance theme but have no idea why it would be looking there. 

You can see early in the log the image is gotten correctly by what looks to be a PS4 and later lots of different PS4's are looking in the wrong place.

Spoiler

Line 425315: 2021-09-15 10:31:19.685 Info Server: http/1.1 GET https://xxxx,.xxxxxxxxxx.com/web/modules/themes/blueradiance/bg.jpg. UserAgent: Mozilla/5.0 (PlayStation; PlayStation 4/9.00) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/14.0 Safari/605.1.15
    Line 425343: 2021-09-15 10:31:20.332 Info Server: http/1.1 Response 200 to xxx.xxx.xxx.xxx. Time: 648ms. https://xxxx.xxxxxxxxxx.com/web/modules/themes/blueradiance/bg.jpg

Spoiler

Line 433591: 2021-09-15 10:34:03.841 Info Server: http/1.1 GET https://xxxx.xxxxxxxxxx.com/web/bg.jpg. UserAgent: Mozilla/5.0 (PlayStation; PlayStation 4/9.00) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/14.0 Safari/605.1.15
    Line 433603:     System.IO.FileNotFoundException: System.IO.FileNotFoundException: Could not find file '/opt/emby-server/system/dashboard-ui/bg.jpg'.
    Line 433604:     File name: '/opt/emby-server/system/dashboard-ui/bg.jpg'
    Line 433622: 2021-09-15 10:34:03.843 Info Server: http/1.1 Response 404 to xxx.xxx.xxx.xxx. Time: 2ms. https://play.ropynbr.com/web/bg.jpg

Would not think that is related even though that is the last cut off entry in the log.

[igeoorge 26]

The server was 5 days without any crashes.

So it crashed tonight and when I went to check the logs, I noticed this error again.

What do you think I should do?

Should I restart the server and see if the error keeps showing up?

@Happy2Play

[Carlo 4561]

Hi, I don't see anything in the logs either which still makes it seem like it's something outside of Emby causing this.

Do you see any crash reports in the OS?

Do you have other services running on this machine?  When you can't get to Emby can you get to other software on the machine?

[igeoorge 26]

Hi friend, thanks for helping me!

When the emby crashes, the embyserver.exe process is still active but inaccessible.

I believe that whoever falls is the http server

When this happens, we can access the machine normally.

The machine only has emby and Grafana installed.

When the emby http drops, grafana continues to work normally.

When the emby http drops, machine access works normally.

When the emby http drops, the machine's CWP works normally.

Do you think this might be the amount of users? 

[Carlo 4561]

That's good info.  So it does appear the machine itself is still working ok.
It also sounds like nothings crashed but that Emby just stopped working (at the web).

I'm kind of at loss to be honest in what to suggest.
Have you checked system logs for any errors?  On Windows Event Viewer would be used to look at this info but not sure on your OS (I don't use it).
If something went wrong in a net library that could get logged in the os logs.

Do you have any kind of monitor setup on the machine to log stats?
What might be helpful is being able to see memory usage of embyserver.exe.  If we see something that shows emby starting to climb in memory usage that could be helpful.
Could be useful to see network utilization of embyserver as well if possible.
Same with CPU usage.

I don't really know how to answer the question about users but I'll just say I don't think I've seen another system with more users setup.
But number of users setup doesn't translate to number of active logged in users/devices so I'm not sure if that matters.

Maybe someone else will have some ideas as well to help diagnose this.

PS one last thing to check next time you see this.  When http stops responding take a look at the log file and see if any new lines are being written to the log. It's unlikely to see much as there won't be any communication with a client going on.  If you use RTM for library monitoring you might be able to have a "dummy" movie ready to go that you could move to a library path.  Within a couple of minutes if the core of Emby is still working it will try to process the file and that might show up in the log.  Something like this could help us isolate if the issue is strictly web/http related or not.