Two emby and one nginx server balancing

[igeoorge 26]

Hi friends,

I tried this configuration on cloudflare, but I was unsuccessful.

This way, the videos also don't reproduce

 

 

[Carlo 4330]

You may want to change it to be like this:
 

[igeoorge 26]

Seems like it's caching and reproducing.

Thank you very much friend.

I will monitor to see the result of this.

[igeoorge 26]

19 hours ago, pir8radio said:

yes i run  like this             INTERNET---------Cloudflare---------Nginx---Emby

 

Hey friend,

Talking to the cloudflare team, they informed me that cloudflare already does what Nginx does, they informed that it doesn't have the need, as it will have no effect

Cloudflare Answer:
"If you add one more layer, it's just going to create one more point of failure, and it's going to make diagnosis more difficult."

What do you think?

[Carlo 4330]

There is another issue a couple people are having with Cloudflare and video files that we have been testing.  That is something I just started testing today on my Cloudflare account so it hasn't gotten much testing.  If you don't get any complaints of people not being able to playback MP4 and MKV files you should be fine.

Yes Cloudflare is correct on that general assumption and why I don't personally run nginx. However, some people aren't running just Emby with nginx or may be using redirection, header modification and other slick features.

You yourself might be using it for load balancing.  So nginx can do a lot more than the little part they are talking about.

[igeoorge 26]

I made changes to cloudflare as you directed and so far, everything is working fine.

Gradually it is starting to cache.

I don't understand much about nginx so I went to ask more people. Including the cloudflare itself.

I'm attaching the cache usage graph

 

 

[Carlo 4330]

So far so good, sounds promising.

It would be interesting after a day or two, to see if any performance metrics changes have taken place locally for Emby.
Overall bandwidth used will still be high as most of that is video but as the cache builds on CF the amount of small requests for images should drop.
All the small amounts of work Emby does to manage cache, render images, transport them as well as IO and CPU use should decrease.

Of course, ideally end user rendering of pages of info (web or client) should be faster as a result.

[pir8radio 1292]

On 9/21/2021 at 6:28 PM, cayars said:

There is another issue a couple people are having with Cloudflare and video files that we have been testing.  That is something I just started testing today on my Cloudflare account so it hasn't gotten much testing.  If you don't get any complaints of people not being able to playback MP4 and MKV files you should be fine.

Yes Cloudflare is correct on that general assumption and why I don't personally run nginx. However, some people aren't running just Emby with nginx or may be using redirection, header modification and other slick features.

You yourself might be using it for load balancing.  So nginx can do a lot more than the little part they are talking about.

here are my end rules..    should catch mostly videos only...   and enable caching of images..   what we have found during testing is some apps use  "yourdomain.com/emby/videos" and "yourdomain.com/emby/items"  and some apps do not,  only use url "yourdomain.com/videos" and "yourdomain.com/items/"   the rules below will block all video caching on cloudflare and cache all images  regardless of client app being used.      or if /emby/ is in the url or not.

[pir8radio 1292]

On 9/21/2021 at 9:49 PM, igeoorge said:

I made changes to cloudflare as you directed and so far, everything is working fine.

Gradually it is starting to cache.

I don't understand much about nginx so I went to ask more people. Including the cloudflare itself.

I'm attaching the cache usage graph

 

 

cloudflare is running nginx..  that's how their service works.    all cloudflare is, is hosted nginx.   but they charge you to make certain changes..  like load balancing, and other things..   The advantage of having your own nginx is you have full control of all features, while still being able to use the cloudflare CDN and edge servers..    

[igeoorge 26]

Friends, at certain times of the day, I receive this message when trying to access.

 Do you know what this is about?

I'm leaving the logs attached

 

https://www.mediafire.com/file/bgba45l7jdlk8hg/embyserver.txt/file

[Carlo 4330]

Most OSes have a maximum number of files a given process can open.

 

Depending on OS try to check these with:

sysctl kern.maxfiles
sysctl kern.maxfilesperproc

You can increase the limits (at your own risk) with:

sysctl -w kern.maxfiles=20480 (or whatever number you choose)
sysctl -w kern.maxfilesperproc=18000 (or whatever number you choose)

Typically to make the change permanent, use sudo to put your settings in /etc/sysctl.con (which you may have to create), like this:

kern.maxfiles=20480
kern.maxfilesperproc=18000

Research this yourself for your particular OS, besides doing anything more than a check.

[igeoorge 26]

Thank you friend! you really help me a lot

I'll follow the tips and change the limit

[igeoorge 26]

On 10/1/2021 at 12:00 PM, pir8radio said:

here are my end rules..    should catch mostly videos only...   and enable caching of images..   what we have found during testing is some apps use  "yourdomain.com/emby/videos" and "yourdomain.com/emby/items"  and some apps do not,  only use url "yourdomain.com/videos" and "yourdomain.com/items/"   the rules below will block all video caching on cloudflare and cache all images  regardless of client app being used.      or if /emby/ is in the url or not.

 

hi friends, how are you?

I just came back to give you feedback.

After using these settings, I can greatly improve the cache of pages.

Go from 50 megs of cache per day to 17GB of cache per day.

Thanks again my friends!

[Luke 37095]

1 hour ago, igeoorge said:

 

hi friends, how are you?

I just came back to give you feedback.

After using these settings, I can greatly improve the cache of pages.

Go from 50 megs of cache per day to 17GB of cache per day.

Thanks again my friends!

Thanks for sharing.

[Carlo 4330]

That's really great news. That should also greatly reduce the number of images that Emby has to process and return to clients!

Edited October 6, 2021 by cayars

[igeoorge 26]

hi friends good afternoon

After a whole week without problems, today the server had an unexpected stop.

The embyserver.exe service is active, but the server is unreachable.

I had to restart the server for it to come back.

Analyzing the server logs after it restarted, I noticed that after 15:33h the same error was repeated many times.

Can you see anything strange in these logs?

 

https://www.mediafire.com/file/n2rn6khb2sov1mh/embyserver.txt/file

[Carlo 4330]

Can you upload the server log directly to our forum vs using 3rd party sites?

[igeoorge 26]

Hey friend,

Yes definitely

embyserver.txt

[igeoorge 26]

hi friends, it happened again now

Occurred at 5:28 pm
  and at 5:41 pm

 

unfortunately it is not possible to upload the file on the forum.

https://www.mediafire.com/file/sw1nx0qywz0ffcr/embyserver+2.txt/file

[igeoorge 26]

Friend, is that the problem?

 

 *** Error Report ***
    Version: 4.6.4.0
    Command line: /opt/emby-server/system/EmbyServer.dll -programdata /var/lib/emby-server -ffdetect /opt/emby-server/bin/ffdetect -ffmpeg /opt/emby-server/bin/ffmpeg -ffprobe /opt/emby-server/bin/ffprobe -restartexitcode 3 -updatepackage emby-server-rpm_{version}_x86_64.rpm
    Operating system: Linux version 3.10.0-1160.21.1.el7.x86_64 (mockbuild@kbuilder.bsys.centos.org) (gcc version 4.8.5 20150623 (Red Hat 4.8.5-44) (GCC) ) #1 SMP Tue Mar 1
    Framework: .NET Core 3.1.13
    OS/Process: x64/x64
    Runtime: opt/emby-server/system/System.Private.CoreLib.dll
    Processor count: 12
    Data path: /var/lib/emby-server
    Application path: /opt/emby-server/system
    System.IO.FileNotFoundException: System.IO.FileNotFoundException: Could not find file '/opt/emby-server/system/dashboard-ui/bg.jpg'.
    File name: '/opt/emby-server/system/dashboard-ui/bg.jpg'
       at Interop.ThrowExceptionForIoErrno(ErrorInfo errorInfo, String path, Boolean isDirectory, Func`2 errorRewriter)
       at Microsoft.Win32.SafeHandles.SafeFileHandle.Open(String path, OpenFlags flags, Int32 mode)
       at System.IO.FileStream..ctor(String path, FileMode mode, FileAccess access, FileShare share, Int32 bufferSize, FileOptions options)
       at Emby.Server.Implementations.IO.ManagedFileSystem.GetFileStream(String path, FileOpenMode mode, FileAccessMode access, FileShareMode share, Int32 bufferSize, FileOpenOptions fileOpenOptions)
       at Emby.Server.Implementations.IO.ManagedFileSystem.GetFileStream(String path, FileOpenMode mode, FileAccessMode access, FileShareMode share, FileOpenOptions fileOpenOptions)
       at Emby.Server.Implementations.IO.IOManager.GetResponse(HttpRequestOptions options, MediaProtocol protocol)
       at Emby.Server.Implementations.IO.IOManager.GetResponse(HttpRequestOptions options)
       at Emby.Server.Implementations.HttpServer.HttpResultFactory.GetContent(StaticFileResultOptions options, Int64 offset, Int64 length, CancellationToken cancellationToken)
       at Emby.Server.Implementations.HttpServer.FileWriter.SetContentResponseHeaders(Nullable`1 reportedTotalContentLength, CancellationToken cancellationToken)
       at Emby.Server.Implementations.HttpServer.HttpResultFactory.GetStaticResult(IRequest requestContext, StaticResultOptions options)
       at Emby.Web.Api.DashboardService.Get(GetDashboardResource request)
       at Emby.Server.Implementations.Services.ServiceController.GetTaskResult(Task task)
       at Emby.Server.Implementations.Services.ServiceHandler.ProcessRequestAsync(HttpListenerHost appHost, IRequest httpReq, IResponse httpRes, RestPath restPath, String responseContentType, CancellationToken cancellationToken)
       at Emby.Server.Implementations.HttpServer.HttpListenerHost.RequestHandler(IRequest httpReq, ReadOnlyMemory`1 urlString, ReadOnlyMemory`1 localPath, CancellationToken cancellationToken)
    Source: System.Private.CoreLib
    TargetSite: Void ThrowExceptionForIoErrno(ErrorInfo, System.String, Boolean, System.Func`2[Interop+ErrorInfo,Interop+ErrorInfo])

[igeoorge 26]

Oi amigo,

Copiei o arquivo: bg.jpg 
da pasta: dashboard-ui\modules\themes\blueradiance

e colei na pasta: dashboard-ui

 

    System.IO.FileNotFoundException: System.IO.FileNotFoundException: Could not find file '/opt/emby-server/system/dashboard-ui/bg.jpg'.
    File name: '/opt/emby-server/system/dashboard-ui/bg.jpg'

 

I made this change at 6:18 pm

It is now 6:28pm

After the change, the error no longer appeared in the logs

 

https://www.mediafire.com/file/8jbswza1lvvu8vg/embyserver+3.txt/file

 

 

[Carlo 4330]

There is nothing in the log that shows a related problem.
That missing graphic file is not a problem so don't worry about it.

It really just looks like your pushing things beyond what the hardware/OS itself can handle on peak loads.
What is probably needed is to get a specialist that can look over and tune the OS.

The Emby logs and this forum most likely are not going to be able to help you in this matter.
We've already seen things like exhausted file handles and other OS resources which would indicate the OS has likely never been tuned for the load you're putting on it.

Having 500+ simultaneous streams going is going to be very taxing on that machine/OS. 

[igeoorge 26]

Hey friend,

After your last comment, I was a little sad. Because we know that the server where the emby is is fully capable of handling a lot of traffic.

However, I would like to thank you for your help, you have a good heart.

I would like to report feedback, maybe this will help more people:

Yesterday I made a change that solved the problem of falls.

Using a plugin called Device list Cleaner, I performed a clean on the devices.

After performing the clean, I had no more interruptions, even with a lot of friends online and with 1GB/s of network traffic.

Thanks @chef for developing this plugin!

[chef 3746]

If it helped then that's awesome!  Thanks for the feedback.

[Happy2Play 8296]

OT:

Then would that suggest device limit clutter affecting server operations with this many users?  Does device list maintenance needs to be done?

With this many users I can see a issues like this being limited to only a setup like this, or taking many years for a home server to see this.  Similar to too many items in transcode-temp folder causing dashboard issues.