Emby Server not releasing CPU process after cancelled HW Transcode

[ctaranto 14]

Command line: /usr/lib/emby-server/EmbyServer.dll -programdata /var/lib/emby -ffdetect /usr/bin/ffdetect-emby -ffmpeg /usr/bin/ffmpeg-emby -ffprobe /usr/bin/ffprobe-emby -restartexitcode 3
Operating system: Linux version 5.4.89-1-MANJARO (builduser@LEGION) (gcc version 10.2.0 (GCC)) #1 SMP PREEMPT Tue Jan 12 23:39:44 UTC 2021
Framework: .NET Core 3.1.8
OS/Process: x64/x64
Runtime: usr/share/dotnet/shared/Microsoft.NETCore.App/3.1.8/System.Private.CoreLib.dll
Processor count: 8
Data path: /var/lib/emby
Application path: /usr/lib/emby-server
32 GB RAM

Emby server is configured with GPU Hardware Transcoding (nvidia 1660 Super).  It appears to be working great - without GPU HW transcoding, all CPU threads spike to 100% and the content freezes.  With GPU HW, a single CPU thread hovers around 70% and content plays great.

When I stop a movie, the CPU thread continues to run (attached is the process list).  The "white" lines are the main processes.  The green lines are threads and can be ignored.  The main process (on top) is the CPU process that shouldn't be there, as the client is no longer connected to the server.

When transcoding by CPU, the process all stop (though transcoding isn't necessarily successful since my CPU doesn't have enough oomph - i7-4790k overclocked to 4.4Ghz).

Restarting the Emby Server makes the process go away.  Until I do another HW transcode...

Any thoughts on how I can resolve this?

[Luke 40086]

Hi there, how did you stop the movie?

[ctaranto 14]

Hi Luke,

Small correction (that I just realized).  It's not just exiting content playing (I just tried that and the CPU process went away).

If during playback I change the bitrate (gear icon on bottom right), I will see a new ffmpeg-emby process appear (so two in total).  Exiting playback cleans up one of the ffmpeg-emby processes, but one of them remains indefinitely.

What's interesting is if I change the bitrate a few more times, there are at-most 2 processes running, and when exiting (hitting the back arrow), one process always remains.

If I re-start playback when a process is already running, another process starts.  If I change the bitrate, a 3rd process appears.  Exiting will remove one of the processes and two will remain.

Let me know if there's a log I can provide to help...

Update:  This issue seems media dependent.  One movie has the issue, the other does not.

Both have:

Video:
Title 4K HEVC
Codec HEVC
Profile Main 10
Level 153
Resolution 3840x2160
Aspect Ratio 16:9
Interlaced No
Framerate 23.976
Video Range HDR
Color Primaries bt2020
Color Space bt2020nc
Color Transfer smpte2084
Bit Depth 10 bit
Pixel Format yuv420p10le
Reference Frames 1

Audio:
Title English TRUEHD 7.1 (Default)
Language English
Codec TRUEHD
Layout 7.1
Channels 8 ch
Sample Rate 48,000 Hz
Bit Depth 24 bit
Default Yes

The one with an issue of processes hanging around when switching the playback bitrate has a video bitrate of 69,873 kbps while the one that doesn't have this issue has a bitrate of 56,424 kbps.

 

Edited February 4, 2021 by ctaranto

[Luke 40086]

How are you exiting playback?

[ctaranto 14]

The answer was mixed in the message above.  "...(hitting the back arrow)".  The back arrow at the top left.  This is in the web player.  Tried both edge and chrome.

[Luke 40086]

Can you please attach the emby server log from when this happened? Thanks.

[ctaranto 14]

PM sent with files.  Thanks.

[ctaranto 14]

Any updates on the fix for this?  I am still seeing the issue.  Thanks.

[Luke 40086]

Hi, can you please attach the emby server and ffmpeg log files from when this happened? Thanks.

[ctaranto 14]

PM sent with files.

[Luke 40086]

In the logs you provided over PM, the ffmpeg process appeared to exit normally. We'll see if @softworkz has any thoughts. Thanks.

[softworkz 4569]

@ctaranto - How did you install Emby? Which package did you use?

[softworkz 4569]

@ctaranto - What happens when you don't use the browser's back button, and click on the on-screen back-arrow (left top) instead?

[ctaranto 14]

I installed (and continually update) Emby through the Manjaro package manager using the "emby-server" package.

I don't use the browser back button - I use the on-screen back arrow.  I wasn't clear when I mentioned this above.

I also don't think it has to do with the back arrow - just changing the bit rate using the gear icon will result in two ffmpeg processes when only one should exist.  Using the on-screen back arrow will stop the ffmpeg process that is being used, but doesn't stop the ffmpeg process that shouldn't be there in the first place.

And to be clear, this doesn't happen for all media content.  It happens all the time in "Greenland" but doesn't for other movies.

Thanks...

 

Edited March 30, 2021 by ctaranto

[Luke 40086]

How are you determining that? because the log you provided shows the ffmpeg process exiting normally.

[ctaranto 14]

htop on the console.

[ctaranto 14]

Playing Greenland at 4k (auto bitrate):

 

 

Changing the bitrate to 1080p / 20Mpbs during playback.  Two ffmpegs:

 

 

Exiting the playing moving using the on-screen back button.  One ffmpeg is remaining and never goes away until I restart the Emby server:

 

 

If you need more files, let me know and I can PM them to you.

 

 

Edited March 30, 2021 by ctaranto

[softworkz 4569]

9 minutes ago, Luke said:

How are you determining that? because the log you provided shows the ffmpeg process exiting normally.

The server log includes in fact one ffmpeg that hasn't ended (within the time covered by the log)

[ctaranto 14]

What I find odd is the UUID of the ffmpeg process changes when I change bit rates.

For example, if the ffmpeg uuid when playing content at 4k is 1, and I switch bitrates to 1080p, there are then two ffmpeg process with uuid's 2 and 3.  The ffmpeg uuid of 1 is no longer there.  When I exit using the on screen back button, ffmpeg process with uuid 3 goes away.  That can be seen in the screen shots of htop I posted above, as well as the 3 transcodes that are listed by softworkz.

Edited March 30, 2021 by ctaranto

[softworkz 4569]

The ffmpeg process ids are random. Each process gets a new id.

Could you please uninstall the Playback Reporting plugin, restart the server and retry?

[ctaranto 14]

Will doing this delete the data behind the plugin?

Nevermind.  Found the backup button.

Edited March 30, 2021 by ctaranto

[softworkz 4569]

Just now, ctaranto said:

Will doing this delete the data behind the plugin?

Probably not, but I can't guarantee that as I don't know the plugin.

[Happy2Play 9444]

10 minutes ago, softworkz said:

  6 minutes ago, ctaranto said:

Will doing this delete the data behind the plugin?

It has its own database, only thing affected would be stats while the plugin is not installed. 

Edited March 30, 2021 by Happy2Play

[ctaranto 14]

I removed the plugin and restarted.  Unfortunately, it did not solve the issue.

 

Edited March 30, 2021 by ctaranto

[softworkz 4569]

OK, now I to understand what's happening at least:

• When the user changes the bitrate, the client tells the server to stop the playsession
• ffmpeg process is stopped
• Then, the player requests another segment (which shouldn't happen) with the parameters for the stopped playsession
• A new ffmpeg process is created to satisfy the segment request
• The client receives the playlist for the new playsession (with the changes bandwidth params)
• Client requests a segment from that playlist
• Server starts a new ffmpeg process for the that request
• The intermediate process becomes "orphaned"

That's what's happening, I just don't know why... @Luke - any idea?