Massive transcoding temp files

[Masterz 2]

12 hours ago, jacotec said:

Sorry for capturing that old thread, but today my monitoring sent me a warning that my Emby VM is running low on disk space. Wondering why (I have all media files on a mounted external source) I've checked the /dev/sda1 contents and found a 195GB(!!!) .ts temporary transcoding file in /tmp.

Needed to wipe the file and reboot the server.

I'm running the current 4.4.3.0 build on Ubuntu 16.04 and I'm wondering how this could happen ...

I've created quick & dirty fix on Windows machine: scheduled daily batch file to clear transcoding files.

Using command like this: forfiles /p <PATH_TO_>\Emby-Server\transcoding-temp\ /m *.ts /d -3 /c "cmd /c del /S @PATH"

On Ubuntu similar "solution" can be established too.

[ebr 14913]

4 hours ago, Masterz said:

I've created quick & dirty fix on Windows machine: scheduled daily batch file to clear transcoding files.

Using command like this: forfiles /p <PATH_TO_>\Emby-Server\transcoding-temp\ /m *.ts /d -3 /c "cmd /c del /S @PATH"

On Ubuntu similar "solution" can be established too.

That's going to be a problem if someone is actually playing something at the time...

[Carlo 4330]

5 hours ago, Masterz said:

Using command like this: forfiles /p <PATH_TO_>\Emby-Server\transcoding-temp\ /m *.ts /d -3 /c "cmd /c del /S @PATH"

The /d -3 will make sure this only deletes files with a creation date older than 3 days.

You should be able to make this /d -1

That will will clean up files older than 1 day.  The chance of still using a temp transcode older than a day is very slim.

With something like this you could schedule it to run every hour as well.

Edited July 14, 2020 by cayars

[Sammy 735]

On 7/14/2020 at 7:52 AM, cayars said:

The /d -3 will make sure this only deletes files with a creation date older than 3 days.

You should be able to make this /d -1

That will will clean up files older than 1 day.  The chance of still using a temp transcode older than a day is very slim.

With something like this you could schedule it to run every hour as well.

Shouldn't Emby just take care of this by itself? 

Here's what I'm looking at this morning when I got a drive almost full error when firing up my machine. I have a 1Tb m.2 drive and should not be seeing these things.

 

A little bit excessive, don't you think @Luke @ebr?

[Sammy 735]

Having my drive this full was causing issues with PC performance probably because things were having to be swapped to RAM disk. After deleting this nearly 20 hour LiveTV temp file things were smooth as silk again. I think I'll add this script to a .bat file and have it run on a daily schedule but seriously, this should not be happening at all. A lot of users are not as adept at doing these sorts of things and really shouldn't have to do them.

Edited July 25, 2020 by Sammy

[Sammy 735]

I already have a .bat file to clean the MCEBuddy back ups on a schedule so I just added another line to it as follows:

forfiles /p "<PATH TO>\transcoding-temp" /s /m *.ts /d -2 /c "cmd /c del @PATH"

I think I'll add another to clean up all the abandoned .nfo files in my Emby Recordings Directory that are lingering after MCEBuddy does its thing. Can this be done at the Emby Recordings folder with a structure such as

• C:/Emby Recordings/Movies
• C:/Emby Recordings/TV Series

and still delete all the nfo's in the subfolders or is there a bit more tweaking to this script to make it happen? @cayars

Edited July 26, 2020 by Sammy

[kevinf63 3]

55 minutes ago, Sammy said:

Having my drive this full was causing issues with PC performance probably because things were having to be swapped to RAM disk. After deleting this nearly 20 hour LiveTV temp file things were smooth as silk again. I think I'll add this script to a .bat file and have it run on a daily schedule but seriously, this should not be happening at all. A lot of users are not as adept at doing these sorts of things and really shouldn't have to do them.

TLDR: I gave up on Emby and wrote my own solution as I personally don't need it to transcode.

I personally got sick of the wait and cut my Emby subscription, as sad as it makes me. It's nearly a year later since I reported similar findings in this and other threads regarding Live TV transcoding building up large files.

Given that my wireless network was capable of direct streams (Orbi mesh network) and my profession (software dev), I looked into alternative methods for the Samsung smart TVs and realized the best way for me was to simply build my own client/app.

I built a self-hosted Web API around the Media Station X (white-label) app on my Samsung Smart TVs that taps into TVHeadEnd's Web API to get EPG data and launches a video player running the MP2T (.ts) stream when you select the channel in the menu.

It's rough around the edges and limited, but it's primary function of live TV playback works consistently without issue. I started it months ago, but with COVID-19 displacing me for months I haven't been able to really sink my teeth into fleshing it out...

[Sammy 735]

Do you get Closed Captions?

Does it timeshift/pause/play live streams?

Does it work with HDHomeRun Prime and Connect Tuners?

Does it record? Have a DVR to schedule recordings?

Edited July 25, 2020 by Sammy

[Carlo 4330]

I don't see how you gained anything from this.

If all you want is super simple and being able to "tune tv" without pause/rw/ff, shared tuners, etc then why not just split the cable, boost the signal and run coax to each TV and not use software at all?

[kevinf63 3]

6 hours ago, Sammy said:

Do you get Closed Captions?

Does it timeshift/pause/play live streams?

Does it work with HDHomeRun Prime and Connect Tuners?

Does it record? Have a DVR to schedule recordings?

I get subtitles. I can pause/play live streams but not timeshift rw/ff unless I'm running it directly through TVHeadEnd stream on say an Android/Fire TV app (only one device in house for that). As I'm in the EU I use a slightly different HDHomeRun Quattro and a Megasat SAT>IP Server 3 (DVB-T2 and DVB-S2) for Irish terrestrial and satellite.

It's TVHeadEnd so it has the ability to record and schedule from the backend, I'm just fleshing out the front-end TV interface to do it easier and more intuitively, as well as obviously playback those recordings.

Technically it could transcode it server-side, for now I just use the pass profile as my parents needed something that works consistently at home, which unfortunately Emby just failed to play more than one frame before failing or as I'll explain in another reply other problems.

My solution is definitely lacking a lot feature wise, but has enough to please the older parents for the TV basics, so for now I'll just schedule recordings through TVH web portal and remote watch through alternative android apps on my phone for now. That will change in time! For now it's just a neat side project that's been fun, a learning experience and ended up being the right solution for my parents thus far (they don't record much, but I do!)

[kevinf63 3]

5 hours ago, cayars said:

I don't see how you gained anything from this.

If all you want is super simple and being able to "tune tv" without pause/rw/ff, shared tuners, etc then why not just split the cable, boost the signal and run coax to each TV and not use software at all?

Long story short, I had and want those features for myself like remote play, pausing, rewind... but my parents use Live TV for its simpler functions such as standard playback and the odd bit of recording and I simply couldn't achieve that with PLEX or Emby for various reasons. For now, I'll deal with scheduling recordings through the TVHeadend interface and simple playback that works is what they need most, so that's what I'll provide until I can spruce something up that "just works". 

The long story boils down to much more than just the unwieldly transcoding files thrashing the SSD. Originally had no dedicated server, and had individual LibreELEC boxes running TVHeadEnd server/clients through Kodi and they were not connected, so the recordings were all on separate machines and a bit disjointed. Got a media server for PLEX, heard it had a new native Live TV feature at time, tried it and it was just too clunky to setup with M3U tuners and didn't even have an EPG on Samsung Smart TVs (still doesn't) for some bizarre reason.

PLEX didn't pan out, so I went for Emby next, it functioned using the TVHeadEnd plugin for some time, but an official Emby update soon broke that plugin, so I had to use the native Live TV features for my HDhomeRun and my DVB-S2 M3U tuner, which worked great for the HDHomeRun but performed miserably for the M3U tuner. Default language set as english but would frequently playback narration track was a nuisance, playback would cut randomly, transcoding thrashing my SSD despite it being a direct stream was very odd, then one day it just refused to playback properly at all after an Emby server update as far as I could tell. it would play the first frame of the video/audio and die immediately. I've another thread on the forum for that, and other folks experience too unfortunately.

Eventually I settled on rustling something up simple to get the playback job done myself rather than relying on something else. I'm sure there was another way around it, a potential fix for Emby even. I just personally couldn't wait any longer. I can retrieve the list of TVH recordings, just need to build out the menus and get recordings playing back next. Then I'll figure out requesting scheduling recordings through TVHeadEnd API by pressing a red button of sorts in the menus/TV remote etc.

[Carlo 4330]

Is this from OTA or cable?

[kevinf63 3]

5 minutes ago, cayars said:

Is this from OTA or cable?

The HDHomeRun Quattro is DVB-T2 (Free terrestrial aerial/"Saorview") and the Megasat SAT>IP Server 3 is DVB-S2 (Over the Air Satellite/Free to Air/FreeSat).

[Carlo 4330]

OK because one of the things you can do with antenna based system is to run a direct coax cable to the TVs from a splitter.  Most TVs work pretty well with coax attached.  LOL

That way you have TV on the TV (old school) and can use Emby for DVR purposes.  That will save you tuner ports as well.

[kevinf63 3]

Just now, cayars said:

OK because one of the things you can do with antenna based system is to run a direct coax cable to the TVs from a splitter.  Most TVs work pretty well with coax attached.  LOL

That way you have TV on the TV (old school) and can use Emby for DVR purposes.  That will save you tuner ports as well.

Not a bad idea... Unfortunately the Samsung Smart TV EPGs are... Not very smart? They do this very odd thing were they split terrestrial and satellite into separate sections? So you can't see all your channels together, and have to jump through hoops to flick between both types. Very hard to explain on paper, but to technophobe parents, was a nightmare (where's TV3 gone!?! "Like I said last week...")

Also I guess the big problem would be all our terrestrial and satellite lines run into one room where the server and HDHomeRun/network tuners are so that would be a lot of undoing! I think the moral of the story is, test any new system you wish to install thoroughly before diving in, I didn't with network tuners, and paid the price dearly!

[jacotec 7]

In my opinion the transcoding temp files should no be anything the user needs to deal with by setting up cron jobs etc. Emby creates these temp files, so Emby needs to ensure that it does not leave abandoned files on the disk array.

Even in case a user closes his app or crashes his app while watching live TV!

If a temp transcoding file is not in active use anymore, Emby must wipe it from the disk. That can be a scheduled task inside Emby, but nobody else than Emby knows if the file is still being used or not.

Second, why does Emby create these ultra huge transcoding files? A much better strategy would be to create i.e. a temp file for max 15 mins of content and kind of "rotating" through their use. So the TV backend writes a temp file, once 15 mins are over it starts writing into temp file 2 (while the playback might still be from file 1). Once file 1 was watched on the client side and the client changed to file 2 for playback, Emby should erase file 1 and so on.

@Luke What do you think? There seems to be many users with that issue.

Edited July 26, 2020 by jacotec

[Carlo 4330]

As Luke has already mentioned a few times there is active development and testing on a new Live TV project/subsystem for Emby Server so stay tuned.

[jacotec 7]

1 minute ago, cayars said:

As Luke has already mentioned a few times there is active development and testing on a new Live TV project/subsystem for Emby Server so stay tuned.

 

Thanks, seems I've missed that statement.

[kevinf63 3]

1 minute ago, cayars said:

As Luke has already mentioned a few times there is active development and testing on a new Live TV project/subsystem for Emby Server so stay tuned.

 

I was actually unaware of this, although I haven't been active in the forums for the last few COVID-19 months to keep an eye. Thank you for the update.

[Sammy 735]

On 7/14/2020 at 7:03 AM, ebr said:

That's going to be a problem if someone is actually playing something at the time...

So can Emby clean itself?

The huge file I had was days old and was 19 hours of Cable News. Apparently a client was left running along and this was left behind. I cannot think of a case where LiveTV is steaming for nearly an entire day and someone is still watching unless they croaked on the couch and cutting off the live stream is the least of our worries!

Edited July 26, 2020 by Sammy

[Carlo 4330]

I've not had a problem highlighting the whole directory and killing things.  Anything in use is locked and can't be deleted.

[Carlo 4330]

On 7/25/2020 at 9:16 AM, Sammy said:

I already have a .bat file to clean the MCEBuddy back ups on a schedule so I just added another line to it as follows:

forfiles /p "<PATH TO>\transcoding-temp" /s /m *.ts /d -2 /c "cmd /c del @PATH"

I think I'll add another to clean up all the abandoned .nfo files in my Emby Recordings Directory that are lingering after MCEBuddy does its thing. Can this be done at the Emby Recordings folder with a structure such as

• C:/Emby Recordings/Movies
• C:/Emby Recordings/TV Series

and still delete all the nfo's in the subfolders or is there a bit more tweaking to this script to make it happen? @cayars

you have nfo's in your transcoding temp directory?

Edited July 26, 2020 by cayars

[Sammy 735]

1 minute ago, cayars said:

you have nfo's in your transcoding temp directory?

No.. I was thinking in the Emby Recordings Folder of using a similar script to delete these files left behind after MCEBuddy does it's thing. I know, off topic, but using a script to deal with this came to mind. Each recording gets in its own folder in Emby, even in seasons for Series (brilliant @ebr and @Luke). Even WMC didn't do this. Within each of these is at least an nfo and maybe more that's created by Emby before MCEBuddy gets to doing it's thing. I know these are small but as in differential calculus a lot of small things can add up to big things.. I'd like to delete these as they don't do anything and are just lingering along. Having it automated at the upper folder level but deleting these in the lower sub folders is what I'm talking about.

[Carlo 4330]

Yea, sure you could do similar and I'll help if needed but of course should do testing on a copy of data "just in case".

Something like this would delete all the NFO files.

forfiles /p "<PATH TO>\recording area" /s /m *.nfo /c "cmd /c del @PATH"

but if testing without destruction could use this

forfiles /p "<PATH TO>\recording area" /s /m *.nfo /c "cmd /c  echo @path will get deleted"

so you change it from "del" to "echo" and get a list on screen of what would be deleted.  Helps to avoid mistakes that way.

[Sammy 735]

On 7/26/2020 at 10:33 AM, cayars said:

Yea, sure you could do similar and I'll help if needed but of course should do testing on a copy of data "just in case".

Something like this would delete all the NFO files.

forfiles /p "<PATH TO>\recording area" /s /m *.nfo /c "cmd /c del @PATH"

but if testing without destruction could use this

forfiles /p "<PATH TO>\recording area" /s /m *.nfo /c "cmd /c  echo @path will get deleted"

so you change it from "del" to "echo" and get a list on screen of what would be deleted.  Helps to avoid mistakes that way.

I did the echo and it is echoing a very long list of .nfo files. I will change "echo" to "del" and run again. Thanks!

I also have a bunch of orphaned posters so I'd add another forfiles command to include .jpg files too..