Massive transcoding temp files

[Audiosleef 0]

Any update on this? I was just watching LOTR Blue Ray and it shut off, appearantly I only had 90 kb diskspace left...

[Ronstang 202]

The only update is going to be the new version of Live TV which I've been told should be in the next few months or so because the problem we are all experiencing has do do with the core of the current Live TV so it would require an entire rewrite....which is what the new version of Live TV is going to be.  So at the moment your only options are to do as I did and get a large SSD and dedicate it to the Emby transcoding temp file or not use Live TV at all.

[Jason'sEmby 29]

I hit 700 gigs tonight on my ssd lol, maxed that sucker out. Forgot to restart my server this morning:(. 

[Ronstang 202]

48 minutes ago, Jason'sEmby said:

I hit 700 gigs tonight on my ssd lol, maxed that sucker out. Forgot to restart my server this morning:(. 

That's because Emby eats resources like a fat lady eats pie.  I don't have to restart my server any more, I simply stopped using it to watch live TV, I only record and as soon as a recording stops it releases that tuner's temp file.  If you watch TV on the other hand it never lets go until you run out of space.  I keep hearing you have to back out of the channel properly which I always did but the tuner was never released.

[Carlo 4330]

I think this is Win TV software not clearing, not Emby?

@Jason'sEmbycan you confirm?

@Ronstang A lot of your problem likely is the old hardware being used through a proxy.  I'm surely not saying Emby shouldn't handle it better but most running HDHomeRuns don't have this issue.  On my personal system I typically won't have any hung sessions or if it happens it's like once a month. Every recording done via tuner is perfect.  I can't remember the last time I had a failed recording unless it was a Baltimore station 75+ miles away with a low signal.

With that said I do use SSD for transcoding and as my recording directory to keep IO fast.

[Jason'sEmby 29]

@cayars I don't use win TV:). It has to do with emby. Especially when a ghost stream gets stuck on the server. That's why I'm so focused on trying beta because of heartbeat. Once heartbeat is working and there's no streams being stuck open then the transcode folder won't be gigantic. At least in my case;).

[Carlo 4330]

OK thanks for clarifying as I was wrong.

Are you on .22 beta for the server?

[Jason'sEmby 29]

Once live TV is cleared up on beta. I run stable as my daily driver for now. 

Edited January 30, 2021 by Jason'sEmby

[lastwraith 4]

Yeah, same here as Ronstang. I do use an HDHR, never used WinTV and Emby's LiveTV behavior isn't great. I do primarily use a TCL Roku (I use the FireTV also, which works better) so not exiting it probably doesn't help, but Emby doesn't behave very well with regards to massive files being generated and locking (sometimes). 

It's okay though, we can wait for the new LiveTV release. But I'm eagerly awaiting, haha. Definitely appreciate the devs focusing their efforts in this area. 

[Ronstang 202]

Yeah, we can wait and are waiting patiently.  I'm not complaining just sharing so others that have the same problem realize it's not isolated and also that it is BEING RESOLVED in the new Live TV.  I surmise most of us here love Emby and support it even when it goes awry, I know I do.  I knew of the potential problems when I bought Premiere but I did anyway because I also knew there was a new version of Live TV around the corner which should resolve all the issues I'm having.....I even gladly bought a 1 GB SSD for the fat lady to chew on while I wait 

Before I was using Emby just to watch my content throughout the house on my Firesticks and I love it and it only gets better all the time, I was using NextPVR for recordings even though it was not perfect but once I tried Emby Live TV I knew the NextPVR was going to be in my rear view mirror.  Even though I have had lots of problems I know it is short term and even though I'm patiently waiting for the new version of LiveTV my personality is not going to allow me to not at least try and solve some of the issues I'm having.  Emby's integration and presentation on my Firestick makes it why I prefer it for recording even with the issues.  

Emby can't be beat because the team involved are great and very responsive even when they know there is no solution right away because the problem is at the core of the code....which they are working to replace.  The support for this product is top notch and I have received so much personal help that I am constantly learning and hope someday I can help others more as I understand more.  I feel so lucky to have found Emby.....I'm going nowhere.

[Dan_Austin 62]

I had a small script that I ran hourly that would delete any file whose creation time was over 90 minutes ago.  I don't record movies off TV and did not feel I needed to be ablt to rewind a live tv session more than 90 minutes.  Can't find it now, but wouldn't be hard to recreate.

On the hung HDHR streams, I  haven't had this in a long time, but I did find a small script I use to check to see if my HDHR has any tuners in use at a time I knew there shouldn't be any.  If there were any tuners in use at that time, restart emby.  This would free the locked HDHR tuners and obviously purging the ever-growing temp files.
It is a hack work-around, but I can share it if anyone is interested.

 

[go2amp 2]

So I got a half dozen notices today about this thread - I was in on it a month or two back.  

Like Dan, I just scripted a restart.  Also, like previous people suggested - I switched to SSD for caching (mine is 600GB). 

I did all this and it's been working fine - wife and family I shared with seem happy.

But I'm not happy - I banged out more $ for a higher end card and SSD when I'm told software will fix this issue.  And it should, or am I out of line?

WHEN IS THE MAGIC IPTV FIX coming out?

 

 

[lastwraith 4]

1 hour ago, Dan_Austin said:

I had a small script that I ran hourly that would delete any file whose creation time was over 90 minutes ago.  I don't record movies off TV and did not feel I needed to be ablt to rewind a live tv session more than 90 minutes.  Can't find it now, but wouldn't be hard to recreate.

On the hung HDHR streams, I  haven't had this in a long time, but I did find a small script I use to check to see if my HDHR has any tuners in use at a time I knew there shouldn't be any.  If there were any tuners in use at that time, restart emby.  This would free the locked HDHR tuners and obviously purging the ever-growing temp files.
It is a hack work-around, but I can share it if anyone is interested.

 

I would be interested in this. As you say, it doesn't happen often, but my script to clear things out obviously doesn't work if they are locked and it has happened twice in the last few months. I could just restart the emby server every night I suppose (and probably will), but I am curious to check out how the script parses the HDHR tuners anyway.

[lastwraith 4]

2 hours ago, Ronstang said:

Yeah, we can wait and are waiting patiently.  I'm not complaining just sharing so others that have the same problem realize it's not isolated and also that it is BEING RESOLVED in the new Live TV.  I surmise most of us here love Emby and support it even when it goes awry, I know I do.  I knew of the potential problems when I bought Premiere but I did anyway because I also knew there was a new version of Live TV around the corner which should resolve all the issues I'm having.....I even gladly bought a 1 GB SSD for the fat lady to chew on while I wait 

Before I was using Emby just to watch my content throughout the house on my Firesticks and I love it and it only gets better all the time, I was using NextPVR for recordings even though it was not perfect but once I tried Emby Live TV I knew the NextPVR was going to be in my rear view mirror.  Even though I have had lots of problems I know it is short term and even though I'm patiently waiting for the new version of LiveTV my personality is not going to allow me to not at least try and solve some of the issues I'm having.  Emby's integration and presentation on my Firestick makes it why I prefer it for recording even with the issues.  

Emby can't be beat because the team involved are great and very responsive even when they know there is no solution right away because the problem is at the core of the code....which they are working to replace.  The support for this product is top notch and I have received so much personal help that I am constantly learning and hope someday I can help others more as I understand more.  I feel so lucky to have found Emby.....I'm going nowhere.

I was going to quote a small section of your post but as I read through it I actually agree with every single thing you've said.

We are an Emby/Kodi house here because there are some things we like about each (and why not both anyway?) but I knew Emby was worth plunking down money for when I saw how helpful and involved the devs were. That sold me more than any amount of technical prowess the software could demonstrate on its own.

As you said previously, I am only voicing my history with the LiveTV portion to let others know that they are not alone and so that they have some other real-life experience to compare to. Even with some small flaws and idiosyncrasies to be ironed out currently, Emby is an excellent media companion and I have no doubt that Emby will be an ever greater product in the future. The latest iteration of the app on the Amazon ecosystem has some nice interface improvements for browsing seasons in a series, etc. Stuff keeps getting better and that is great to see. I have no doubts that the new LiveTV offering will be excellent!

Edited January 30, 2021 by lastwraith

[go2amp 2]

2 minutes ago, lastwraith said:

I could just restart the emby server every night I suppose (and probably will), but I am curious to check out how the script parses the HDHR tuners anyway.

This is what I do.  I'm a Linux guy and tried all these "scripts" plus my own touches. 

I finally just went Windows on a Physical server, high-end Nvidia card, and a 4:30AM (everyday of the week) script plus the SSD temp dir I mentioned.

Like this:

shutdown -r -t 0 scheduled task.  

2019 Server.  It's silly and more costly than expected, but it works.

I only hear bitching if that breaks a recording at 4-5am, anymore. 

 

 

 

[Dan_Austin 62]

Brute force and stupidity can overcome all sorts of obstacles.  This is my way of saying the following works.  There may be far more elegent, but perfect is the enemy of good, and this was good enough-

Things to note-
You'll need to change the ip of your HDHR in this script, mine was at .16
You'll need to change the IP of your emby server, mine is .1 in this script
You'll need to change the api_key to your own
If your HDHR has more than 4 tuners, change the '3' to your tuner count
The logic is to restart if the free tuner count is zero, it could be tweaked to restart if any tuner was active, but not what I wanted
Finally I have this set in cron(scheduled task) at 11:55 PM.  I never have anything scheduled to record at this time.

#!/bin/bash

start_count=$(curl -s http://192.168.1.16/tuners.html | grep use | wc -l)
count=$((3 - start_count))

if [[ $count == 0 ]]; then
        curl -X POST "http://192.168.1.1:8096/emby/System/Restart?api_key=da9a39f3cda6442e934a3b2d6e159d59" -d ""
        echo "Restarted Emby to free tuners"
else
        echo "No restart needed."
fi
 

Edited January 30, 2021 by Dan_Austin

[Jason'sEmby 29]

10 hours ago, Dan_Austin said:

I had a small script that I ran hourly that would delete any file whose creation time was over 90 minutes ago.  I don't record movies off TV and did not feel I needed to be ablt to rewind a live tv session more than 90 minutes.  Can't find it now, but wouldn't be hard to recreate.

On the hung HDHR streams, I  haven't had this in a long time, but I did find a small script I use to check to see if my HDHR has any tuners in use at a time I knew there shouldn't be any.  If there were any tuners in use at that time, restart emby.  This would free the locked HDHR tuners and obviously purging the ever-growing temp files.
It is a hack work-around, but I can share it if anyone is interested.

 

I'd be interested if you still have the script? I see you posted the hdhomerun script above. I wish @Luke would throw the plugin restart server option back so I can do my nightly restarts automatically through task manager in Emby. 

[Audiosleef 0]

I use this Powershell script now, deletes all the files in the temp folder that are older then 10 minutes, but you can adjust that if you want. 

$path = "C:\Users\YOURUSERNAMEGOESHERE\AppData\Roaming\Emby-Server\programdata\transcoding-temp"

while($true){
    $limit = (Get-Date).AddMinutes(-10)
    Start-Sleep -Seconds 30
    Get-ChildItem -Path $path -Recurse -Force | Where-Object { !$_.PSIsContainer -and $_.CreationTime -lt $limit } | Remove-Item -Force
}

You can remove the start-sleep if you want it to continuously look for files to delete, but this clears up the necessary diskspace for me to be able to watch the movie. 

[Dan_Austin 62]

That script appears to have been lost to the system upgrade gods.  But not too hard to recreate.

The +90 means any file whose last modify time is over 90 minutes.  Adjust to your needs.  -delete is self-explanitory, replace -print to test and validate in your environment.  Don't run as root.  Don't set path to /, or replace path with a variable that can evaluate to /  or "", badness will quickly manifest.  Adding to /etc/cron.hourly is one method to schedule.

#!/bin/bash
/usr/bin/find /var/lib/emby-server/transcoding-temp/ -mmin +90 -delete

Edited January 30, 2021 by Dan_Austin

[Ronstang 202]

Excuse my ignorance about such issues but does a script have more power than a batch file?  I currently have a batch file that empties the transcoding directory every so often and although it works and I stopped having lots of problems it cannot delete files in use by Windows because they are locked....so when you have an errant tuner that won't release the file just keeps growing until the entire SSD is consumed. Or if you have continuous recordings on a channel, like I do often, the temp file for that tuner cannot be deleted until there is a break in the recording.

[Dan_Austin 62]

Nope, 99% just different name for the same thing.  The 1% comes in that batch files are a specific Windows OS script using legacy Windows/DOS commands.  On Windows you can also write a powershell script that in theory could be much more powerful, with access to functions not available in a batch file.

Locked files will be an issue regardless of language or platform.  You could combine the the tuner in use script with a file size check, and if a file is over a reasonable size and a tuner is in use and both are true at a time that neither should be true, restart emby.  That will free the runer and delete the file.

[Ronstang 202]

@Dan_Austin Thanks for the info, that tells me what I am doing is working fine for now and no intervention is needed.  My temp directory never gets bigger than ~250Gb now that I don't use Live TV to actually watch TV as that is what was causing my tuners to never  be released.  My current use of LiveTV is just for recordings and I get at least one break every day or so on each channel I record from so my batch file is taking care of cleaning up the temp directory as best as it can.

[jacotec 7]

Hey @Luke,

just want to ask about any updates regarding the new live TV implementations. It's pretty silent around this ... so is there any news or ETA (or is it in the BETA version at any point)?

[Luke 37066]

On 8/20/2021 at 4:34 PM, jacotec said:

Hey @Luke,

just want to ask about any updates regarding the new live TV implementations. It's pretty silent around this ... so is there any news or ETA (or is it in the BETA version at any point)?

That's kind of a vague question. Please ask instead in terms of something specific that you're looking for. Thanks.

[jacotec 7]

33 minutes ago, Luke said:

That's kind of a vague question. Please ask instead in terms of something specific that you're looking for. Thanks.

Basically a built-in solution to prevent temp transcoding files growing to disc capacity if a user does not end a live stream session properly.