Jump to content

Transcoding Temp Out of Control


BAlGaInTl
 Share

Recommended Posts

BAlGaInTl

Maybe someone can tell me why this is happening...

My Unraid server crashed today.  I'm running Emby in Docker.

When I tracked the problem down, it was because I ran out of space on my 1TB cache drive.  A little more digging led me to the transcoding-temp directory of Emby which is stored on the cache drive.  There were almost 800K files totalling over 500GB of space. The files go back to May of 2019.

A large file transfer caused the drive to fill up the rest of the way, and then crash the system.

Shouldn't these files be cleaned up by Emby?

Is there something in my Unraid/Docker config that could be causing the issue?

Edited by BAlGaInTl
Link to comment
Share on other sites

BAlGaInTl

A bit more poking around, and I think I may have found the issue.

It seems that I had a legacy path to transcoding-temp that no longer existed.  This would have been from back before I switched over to Unraid. I had it pointed to a special mapped directory on an SSD from my old server.  That container path was not mapped in the install.

It's odd...

So would Emby write to one transcoding-temp direcory (default /config/transcoding-temp) but try to delete from a different directory and fail (e.g., /embycache/transcoding-temp)?

I just realized that May 2019 is when I purchased/installed the 1TB cache drive because my 256GB wasn't big enough.  So no transcoding files have been deleted since then.

Link to comment
Share on other sites

Quote

So would Emby write to one transcoding-temp direcory (default /config/transcoding-temp) but try to delete from a different directory and fail (e.g., /embycache/transcoding-temp)?

HI, how did you determine this?

Link to comment
Share on other sites

BAlGaInTl

I first just noticed that Untraid crashed due to my cache drive running out of space.  I did a "du -shc *" a few times on the drive to track down the runaway directory to the Emby transcoding-temp directory.

I determined my error in setup by comparing the configuration to my Emby Beta docker that was not having the same issue. 

I noticed that in the problem install, I had a custom path set for my transcoding-temp directory. It was a legacy directory from back when I had a smaller SSD which I mounted as a separate container path and pointed Emby to that.  

I didn't see any errors in log, but I was guessing that Emby didn't have access to /embycache/transcoding-temp, so it was using the default /config/transcoding-temp instead to write temporary files. I'm guessing that when it tried to remove the temp files, it only looked for the directory that didn't exist and stopped.

My fix was just to remove custom cache and transcoding directories in the Emby configuration.  I'm hoping that fixes the problem, but I will monitor.                                                                                      

Link to comment
Share on other sites

BAlGaInTl
4 hours ago, Luke said:

Thanks for the info. We'll take a look.

Just a quick update.

I did some quick testing today for confirmation.  Temp files are now being removed from the transcoding-temp directory when a playback session is closed.

Oh yeah... and.... 1K posts!

:)

Link to comment
Share on other sites

BAlGaInTl

Looks like I may have spoken too soon.

I've confirmed that "most" temp files are cleaned up.  I've seen it happen when the web client, Android, Amazon Fire, Android TV, and Emby Theater have transcoded.

However.... a remote user with a Roku device connected yesterday and the temporary files were not removed. Some of the clients above have connected remotely, so I know it isn't just an issue with location.

Why does this happen? Does Emby try to clean up these orphaned files at any point?

I've attached the server and transcode logs.

embyserver-63738921600.txt ffmpeg-transcode-5787fe0a-ba7b-4982-a9a2-abf5cc3ca6f5_1.txt

Link to comment
Share on other sites

BAlGaInTl

Well I haven't traced what's causing the issue yet, but I've created a script to delete transcoding-temp files that are older than a day.

 

Has anyone else seen this behavior before? 

Link to comment
Share on other sites

Happy2Play
12 hours ago, BAlGaInTl said:

a remote user with a Roku device connected yesterday and the temporary files were not removed.

Is the user properly backing out of playing media to send a proper stop?  The Roku does not have proper kill commands when say a user hits the home button instead of the back arrow to stop the media. (discussed in other topic by @speechles).  Not sure if the Roku "Are you still Watching" option would help.  But would need to know what the user is doing.  I do not see a Stopped command in the logs provided above for that transcoded item.

But ideally I would think Emby needs a transcode-temp clean up task for these abandoned files.  Sure a restart or proper stops do clean up these files though.

Link to comment
Share on other sites

BAlGaInTl
2 minutes ago, Happy2Play said:

Is the user properly backing out of playing media to send a proper stop?  The Roku does not have proper kill commands when say a user hits the home button instead of the back arrow to stop the media. (discussed in other topic by @speechles).  Not sure if the Roku "Are you still Watching" option would help.  But would need to know what the user is doing.  I do not see a Stopped command in the logs provided above for that transcoded item.

But ideally I would think Emby needs a transcode-temp clean up task for these abandoned files.  Sure a restart or proper stops do clean up these files though.

Good to know about Roku.  I'm guessing that's what's actually going on.

Agreed... after a day or two... the temp files are likely not being used and can be removed.

So I added this to my Unraid server in a cron job script:

#!/bin/bash
find /mnt/cache/appdata/EmbyServer/transcoding-temp/* -mtime +1 -delete

That should delete anything in the transcoding-temp directory that is more than a day old.

Seems like something similar could easily be added to Emby's scheduled tasks to clear the temp path.

Link to comment
Share on other sites

BAlGaInTl
13 hours ago, Happy2Play said:

Sure a restart or proper stops do clean up these files though.

Restarting was not clearing my orphaned files... but I think that was more to do with my custom path setting in Emby.

Now I'm wondering if all of those orphaned files for a year and a half were from Roku devices.  Too late to figure that out now because I nuked them all.

Link to comment
Share on other sites

  • 2 months later...
BAlGaInTl

Has anyone taken a look at this?

The problem seems to be getting worse, and I think it may be because of all the Roku devices people got for the Holiday.  It stll seems to be tied to those devices somehow.

I have thousands of transcoding-temp files that build up.  850GB in just 2 days.

I have a 1TB cache drive in Unraid, so this eats it up really quickly.

I've had to update my script above to run every hour and delete anything older than 6 hours to try and keep up.

 

  • Sad 1
Link to comment
Share on other sites

We are looking into it. Thanks for reporting.

  • Like 2
Link to comment
Share on other sites

  • 1 month later...
BAlGaInTl
On 1/2/2021 at 8:00 PM, Luke said:

We are looking into it. Thanks for reporting.

Thanks for looking in to it.

For now, my cron job is keeping things cleaned up. 

When you think you have it fixed, let me know and I'll disable that and monitor.

Link to comment
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now
 Share

×
×
  • Create New...