Transcoding Temp Out of Control

[BAlGaInTl 279]

Maybe someone can tell me why this is happening...

My Unraid server crashed today.  I'm running Emby in Docker.

When I tracked the problem down, it was because I ran out of space on my 1TB cache drive.  A little more digging led me to the transcoding-temp directory of Emby which is stored on the cache drive.  There were almost 800K files totalling over 500GB of space. The files go back to May of 2019.

A large file transfer caused the drive to fill up the rest of the way, and then crash the system.

Shouldn't these files be cleaned up by Emby?

Is there something in my Unraid/Docker config that could be causing the issue?

Edited October 21, 2020 by BAlGaInTl

[BAlGaInTl 279]

A bit more poking around, and I think I may have found the issue.

It seems that I had a legacy path to transcoding-temp that no longer existed.  This would have been from back before I switched over to Unraid. I had it pointed to a special mapped directory on an SSD from my old server.  That container path was not mapped in the install.

It's odd...

So would Emby write to one transcoding-temp direcory (default /config/transcoding-temp) but try to delete from a different directory and fail (e.g., /embycache/transcoding-temp)?

I just realized that May 2019 is when I purchased/installed the 1TB cache drive because my 256GB wasn't big enough.  So no transcoding files have been deleted since then.

[Luke 38559]

Quote

So would Emby write to one transcoding-temp direcory (default /config/transcoding-temp) but try to delete from a different directory and fail (e.g., /embycache/transcoding-temp)?

HI, how did you determine this?

[BAlGaInTl 279]

I first just noticed that Untraid crashed due to my cache drive running out of space.  I did a "du -shc *" a few times on the drive to track down the runaway directory to the Emby transcoding-temp directory.

I determined my error in setup by comparing the configuration to my Emby Beta docker that was not having the same issue. 

I noticed that in the problem install, I had a custom path set for my transcoding-temp directory. It was a legacy directory from back when I had a smaller SSD which I mounted as a separate container path and pointed Emby to that.  

I didn't see any errors in log, but I was guessing that Emby didn't have access to /embycache/transcoding-temp, so it was using the default /config/transcoding-temp instead to write temporary files. I'm guessing that when it tried to remove the temp files, it only looked for the directory that didn't exist and stopped.

My fix was just to remove custom cache and transcoding directories in the Emby configuration.  I'm hoping that fixes the problem, but I will monitor.                                                                                      

[Luke 38559]

Thanks for the info. We'll take a look.

[BAlGaInTl 279]

4 hours ago, Luke said:

Thanks for the info. We'll take a look.

Just a quick update.

I did some quick testing today for confirmation.  Temp files are now being removed from the transcoding-temp directory when a playback session is closed.

Oh yeah... and.... 1K posts!

[BAlGaInTl 279]

Looks like I may have spoken too soon.

I've confirmed that "most" temp files are cleaned up.  I've seen it happen when the web client, Android, Amazon Fire, Android TV, and Emby Theater have transcoded.

However.... a remote user with a Roku device connected yesterday and the temporary files were not removed. Some of the clients above have connected remotely, so I know it isn't just an issue with location.

Why does this happen? Does Emby try to clean up these orphaned files at any point?

I've attached the server and transcode logs.

embyserver-63738921600.txt ffmpeg-transcode-5787fe0a-ba7b-4982-a9a2-abf5cc3ca6f5_1.txt

[BAlGaInTl 279]

Well I haven't traced what's causing the issue yet, but I've created a script to delete transcoding-temp files that are older than a day.

 

Has anyone else seen this behavior before? 

[Happy2Play 9082]

12 hours ago, BAlGaInTl said:

a remote user with a Roku device connected yesterday and the temporary files were not removed.

Is the user properly backing out of playing media to send a proper stop?  The Roku does not have proper kill commands when say a user hits the home button instead of the back arrow to stop the media. (discussed in other topic by @speechles).  Not sure if the Roku "Are you still Watching" option would help.  But would need to know what the user is doing.  I do not see a Stopped command in the logs provided above for that transcoded item.

But ideally I would think Emby needs a transcode-temp clean up task for these abandoned files.  Sure a restart or proper stops do clean up these files though.

[BAlGaInTl 279]

2 minutes ago, Happy2Play said:

Is the user properly backing out of playing media to send a proper stop?  The Roku does not have proper kill commands when say a user hits the home button instead of the back arrow to stop the media. (discussed in other topic by @speechles).  Not sure if the Roku "Are you still Watching" option would help.  But would need to know what the user is doing.  I do not see a Stopped command in the logs provided above for that transcoded item.

But ideally I would think Emby needs a transcode-temp clean up task for these abandoned files.  Sure a restart or proper stops do clean up these files though.

Good to know about Roku.  I'm guessing that's what's actually going on.

Agreed... after a day or two... the temp files are likely not being used and can be removed.

So I added this to my Unraid server in a cron job script:

#!/bin/bash
find /mnt/cache/appdata/EmbyServer/transcoding-temp/* -mtime +1 -delete

That should delete anything in the transcoding-temp directory that is more than a day old.

Seems like something similar could easily be added to Emby's scheduled tasks to clear the temp path.

[BAlGaInTl 279]

13 hours ago, Happy2Play said:

Sure a restart or proper stops do clean up these files though.

Restarting was not clearing my orphaned files... but I think that was more to do with my custom path setting in Emby.

Now I'm wondering if all of those orphaned files for a year and a half were from Roku devices.  Too late to figure that out now because I nuked them all.

[BAlGaInTl 279]

Has anyone taken a look at this?

The problem seems to be getting worse, and I think it may be because of all the Roku devices people got for the Holiday.  It stll seems to be tied to those devices somehow.

I have thousands of transcoding-temp files that build up.  850GB in just 2 days.

I have a 1TB cache drive in Unraid, so this eats it up really quickly.

I've had to update my script above to run every hour and delete anything older than 6 hours to try and keep up.

 

[Luke 38559]

We are looking into it. Thanks for reporting.

[BAlGaInTl 279]

On 1/2/2021 at 8:00 PM, Luke said:

We are looking into it. Thanks for reporting.

Thanks for looking in to it.

For now, my cron job is keeping things cleaned up. 

When you think you have it fixed, let me know and I'll disable that and monitor.

[Capt.Insano 3]

On 23/10/2020 at 00:45, BAlGaInTl said:

Good to know about Roku.  I'm guessing that's what's actually going on.

Agreed... after a day or two... the temp files are likely not being used and can be removed.

So I added this to my Unraid server in a cron job script:

#!/bin/bash
find /mnt/cache/appdata/EmbyServer/transcoding-temp/* -mtime +1 -delete

That should delete anything in the transcoding-temp directory that is more than a day old.

Seems like something similar could easily be added to Emby's scheduled tasks to clear the temp path.

 

Thanks so much for this, I am having the same issue and never realised it was from Roku clients.

 

Is the root issue any closer to being fixed?

[Luke 38559]

On 1/12/2023 at 12:51 PM, Capt.Insano said:

 

Thanks so much for this, I am having the same issue and never realised it was from Roku clients.

 

Is the root issue any closer to being fixed?

HI, yes we are looking into it. Thanks.

[madqubit 0]

On 10/22/2020 at 6:45 PM, BAlGaInTl said:

#!/bin/bash
find /mnt/cache/appdata/EmbyServer/transcoding-temp/* -mtime +1 -delete

Popped this script in cron and all of my problems disappeared.
Thank you!!!

Please add functionality to scheduled tasks or allowing users to add tasks from the UI. I feel that would help a lot in the way of yet another the other media server applications don't have.