Two emby and one nginx server balancing

[igeoorge 26]

In the device list, there were 12,000 registered devices.

I have this server for over 3 years.

I did the cleaning and got about a thousand devices registered.

I love the emby and I really like it when I can help with something, I'm really happy when I can help here on the Forum.

[igeoorge 26]

When my friends are online, I can't open emby Dashboard, it causes big CPU usage and Emby HTTP server crash.

I hope I can help with the information.

[Happy2Play 8296]

1 minute ago, igeoorge said:

When my friends are online, I can't open emby Dashboard, it causes big CPU usage and Emby HTTP server crash.

I hope I can help with the information.

I can potentially see why with the Dashboard trying to show that many active devices and activity logging.

[igeoorge 26]

Just now, Happy2Play said:

Posso ver o porquê com o painel tentando mostrar tantos dispositivos ativos e registro de atividades.

That's right.

I don't care too much about it. I'm happy just to see the emby active and no problems.

It gives me peace.

[igeoorge 26]

Hi friends

I have a brand new server and I have few friends using it.

But even with few people, it is falling.

I'm attaching the logs, could you help me?

embyserver-63774730789.txt

[neik 837]

1 hour ago, igeoorge said:

few friends

The amount of people described by "few" is probably depending on the definition.
Your log is a 10hours-log and accumulates to ~170MB.

Got me laughing..  

[igeoorge 26]

I'm comparing with the other server.

The other has no problems and in 24h it makes 1.6gb of log.

[Luke 37095]

3 hours ago, igeoorge said:

Hi friends

I have a brand new server and I have few friends using it.

But even with few people, it is falling.

I'm attaching the logs, could you help me?

embyserver-63774730789.txt 167.55 MB · 1 download

What do you mean by falling?

[igeoorge 26]

Oi lucas,

Obrigado pela atenção.

O processo de incorporação está ativo, mas sem acesso ao servidor.

 

In order for it to be accessible again, it is necessary to force a reboot

Edited December 10, 2021 by igeoorge

[Happy2Play 8296]

@LukeDon't think we have seen this before.  Actually, older topics suggest FFMPEG issues as error starting ffmpeg follows this error.

2021-12-10 10:54:39.190 Error GlobalExceptionHandler: UnhandledException
	*** Error Report ***
	Version: 4.6.7.0
	Command line: /opt/emby-server/system/EmbyServer.dll -programdata /var/lib/emby -ffdetect /opt/emby-server/bin/ffdetect -ffmpeg /opt/emby-server/bin/ffmpeg -ffprobe /opt/emby-server/bin/ffprobe -restartexitcode 3 -updatepackage emby-server-rpm_{version}_x86_64.rpm
	Operating system: Linux version 3.10.0-1160.42.2.el7.x86_64 (mockbuild@kbuilder.bsys.centos.org) (gcc version 4.8.5 20150623 (Red Hat 4.8.5-44) (GCC) ) #1 SMP Tue Sep 7
	Framework: .NET Core 3.1.21
	OS/Process: x64/x64
	Runtime: opt/emby-server/system/System.Private.CoreLib.dll
	Processor count: 8
	Data path: /var/lib/emby
	Application path: /opt/emby-server/system
	System.InvalidOperationException: System.InvalidOperationException: Cannot process request because the process (14906) has exited.
	   at System.Diagnostics.Process.ThrowIfExited(Boolean refresh)
	   at System.Diagnostics.Process.EnsureState(State state)
	   at System.Diagnostics.Process.GetStat()
	   at System.Diagnostics.Process.get_StartTimeCore()
	   at System.Diagnostics.Process.get_StartTime()
	   at Emby.ProcessRun.CoreProcess.CoreProcessLocal.get_StartTime()
	   at Emby.ProcessRun.Extensions.ProcessCpuMonitor.TimerCallback(Object state)
	   at System.Threading.ExecutionContext.RunFromThreadPoolDispatchLoop(Thread threadPoolThread, ExecutionContext executionContext, ContextCallback callback, Object state)
	--- End of stack trace from previous location where exception was thrown ---
	   at System.Threading.TimerQueueTimer.CallCallback(Boolean isThreadPool)
	   at System.Threading.TimerQueueTimer.Fire(Boolean isThreadPool)
	   at System.Threading.ThreadPoolWorkQueue.Dispatch()
	Source: System.Diagnostics.Process
	TargetSite: Void ThrowIfExited(Boolean)
	
2021-12-10 10:54:39.211 Info Server: Stopping HttpListener...

 

Edited December 10, 2021 by Happy2Play

[Luke 37095]

5 minutes ago, Happy2Play said:

@LukeDon't think we have seen this before.  

2021-12-10 10:54:39.190 Error GlobalExceptionHandler: UnhandledException
	*** Error Report ***
	Version: 4.6.7.0
	Command line: /opt/emby-server/system/EmbyServer.dll -programdata /var/lib/emby -ffdetect /opt/emby-server/bin/ffdetect -ffmpeg /opt/emby-server/bin/ffmpeg -ffprobe /opt/emby-server/bin/ffprobe -restartexitcode 3 -updatepackage emby-server-rpm_{version}_x86_64.rpm
	Operating system: Linux version 3.10.0-1160.42.2.el7.x86_64 (mockbuild@kbuilder.bsys.centos.org) (gcc version 4.8.5 20150623 (Red Hat 4.8.5-44) (GCC) ) #1 SMP Tue Sep 7
	Framework: .NET Core 3.1.21
	OS/Process: x64/x64
	Runtime: opt/emby-server/system/System.Private.CoreLib.dll
	Processor count: 8
	Data path: /var/lib/emby
	Application path: /opt/emby-server/system
	System.InvalidOperationException: System.InvalidOperationException: Cannot process request because the process (14906) has exited.
	   at System.Diagnostics.Process.ThrowIfExited(Boolean refresh)
	   at System.Diagnostics.Process.EnsureState(State state)
	   at System.Diagnostics.Process.GetStat()
	   at System.Diagnostics.Process.get_StartTimeCore()
	   at System.Diagnostics.Process.get_StartTime()
	   at Emby.ProcessRun.CoreProcess.CoreProcessLocal.get_StartTime()
	   at Emby.ProcessRun.Extensions.ProcessCpuMonitor.TimerCallback(Object state)
	   at System.Threading.ExecutionContext.RunFromThreadPoolDispatchLoop(Thread threadPoolThread, ExecutionContext executionContext, ContextCallback callback, Object state)
	--- End of stack trace from previous location where exception was thrown ---
	   at System.Threading.TimerQueueTimer.CallCallback(Boolean isThreadPool)
	   at System.Threading.TimerQueueTimer.Fire(Boolean isThreadPool)
	   at System.Threading.ThreadPoolWorkQueue.Dispatch()
	Source: System.Diagnostics.Process
	TargetSite: Void ThrowIfExited(Boolean)
	
2021-12-10 10:54:39.211 Info Server: Stopping HttpListener...

 

No, never seen that before, but it's an easy fix to trap and catch for the next release. Thanks.

[igeoorge 26]

You guys are awesome!

My heart fills with hope when you listen to us.

Thank you very much!

[Carlo 4330]

Hi, did you clone an existing machine or do a clean install?

Have you tried shutting down Emby Server and doing another install right over the existing?  That could fix a file if it somehow got corrupted or something.

[igeoorge 26]

Hey friend,

I installed emby entirely from scratch.

So you think the problem is a corrupt file?

[Luke 37095]

10 hours ago, igeoorge said:

Hey friend,

I installed emby entirely from scratch.

So you think the problem is a corrupt file?

HI, I would say we're not sure yet given that this is the first time we've seen this.

[igeoorge 26]

Hi Luke,

Maybe this information goes to something:

A library called Recordings was created.
This library appeared after a friend decided to record a channel.

What happens is the following:

Every time I delete the Recordings library, 12 hours after the deletion the server goes down.

I stopped deleting for testing and it's been 5 days since it crashes.

I'm really happy when I can help with something, so I hope that information will do.

Thank you all for your attention

[Luke 37095]

Thanks for the update.

[igeoorge 26]

Hi friends,

the fall occurred again.

I'll attach the logs.

Hope I can help figure out the problem.

embyserver-63775905949.txt

[Happy2Play 8296]

11 minutes ago, igeoorge said:

Hi friends,

the fall occurred again.

I'll attach the logs.

Hope I can help figure out the problem.

embyserver-63775905949.txt 34.62 MB · 0 downloads

If I had to guess the error says it all.

InnerException: System.IO.IOException: No space left on device

 

[igeoorge 26]

Hi Happy2Play

Thanks for the feedback.

So the problem is lack of space?

[Luke 37095]

2 hours ago, igeoorge said:

Hi Happy2Play

Thanks for the feedback.

So the problem is lack of space?

In this case, yes. It could cause any number of problems.

[igeoorge 26]

Hi friends,

I checked the vps disk and noticed that it has more than 150GB of free space.

Anyway, I delete all the logs that the emby generated.

How strange all this.

[igeoorge 26]

Hi friends,

After analyzing it, I realized that the problem was really lack of space.

the folder: emby/transcoding-temp takes about 80GB of temporary files in a few hours.

And that made the record fill up.

The question I have today is: Can I delete all files in this folder?

These are all .ts files.

[pir8radio 1292]

On 12/31/2021 at 10:32 AM, igeoorge said:

Hi friends,

After analyzing it, I realized that the problem was really lack of space.

the folder: emby/transcoding-temp takes about 80GB of temporary files in a few hours.

And that made the record fill up.

The question I have today is: Can I delete all files in this folder?

These are all .ts files.

it should auto delete those when no longer needed...   you might want to increase the transcode drive size.   

[Happy2Play 8296]

On 12/31/2021 at 8:32 AM, igeoorge said:

The question I have today is: Can I delete all files in this folder?

Whenever a session is stopped it should clean up all the segments it created.  If that is not happening, then yes you will have rogue session data left behind that will have to be manually cleaned up or is cleaned up on a Emby restart.

But as mentioned you many need to place "transcoding-temp" on a larger drive.