Massive transcoding temp files

[Carlo 4330]

The "echo trick" is a great way to test prior as noted.

There is another format you can use if you want one command to handle multiple search masks.  For example to combine this to look for nfo, png, jpg you can do it this way:

for %G in (.nfo, .jpg, .png) do forfiles -p "<PATH TO>\recording area" -s -m *%G -c "cmd /c echo @path will get deleted"

This will only ECHO the results but you change it to delete the same way as the other given above.  Should be obvious how to add new extensions for this to look for.

Carlo

[Sammy 735]

11 minutes ago, cayars said:

The "echo trick" is a great way to test prior as noted.

There is another format you can use if you want one command to handle multiple search masks.  For example to combine this to look for nfo, png, jpg you can do it this way:

for %G in (.nfo, .jpg, .png) do forfiles -p "<PATH TO>\recording area" -s -m *%G -c "cmd /c echo @path will get deleted"

This will only ECHO the results but you change it to delete the same way as the other given above.  Should be obvious how to add new extensions for this to look for.

Carlo

Quick question.. Where would find all the ms command line commands for scripting like this?

[Carlo 4330]

Just now, Sammy said:

Quick question.. Where would find all the ms command line commands for scripting like this?

Quick answer: In my head.

Long answer:  Probably not a help, but what I do (nerd in me) is look at and study every command an OS has available at least to know it exists and to file it away for another day when it may be useful. Then when you get a light bulb moment and think, hey that forfiles command could be useful right now just google it for syntax. So a google search "DOS forfiles syntax" will give a list and the first link is to MS https://docs.microsoft.com/en-us/windows-server/administration/windows-commands/forfiles

That second one builds on the first by using the "for" command which can be for in, for in do, etc  Google "DOS for syntax"

https://docs.microsoft.com/en-us/windows-server/administration/windows-commands/for

Normally MS pages will show the syntax as well as a few examples. But in general syntax is similar with many commands so once you learn a few commands the rest get easier to learn.

Once you know basic command of one OS you can usually transfer them over to another OS and google something like "DOS equivalent of linux mount command" which would likely return results for mklink, subst and join which are symbolic link methods.

On windows you would have DOS command available like the ones used above but also have power script commands which rival linux/unix scripting and are sorting of like using Visual Basic to script with. PS is better to use for more complex things as you have full looping and programming like features for checking conditions (if, then, else) and whatnot.

For complex matching and filtering there is what is known as REGEX parsing (not used above) which is super powerful, but hard to grasp at first but really nice in scripting or full out programming languages like C++ or C#.

So I guess it helps to be a programmer at heart because you sort of learn ways of doing things like this to save time vs writing routines yourself.  Shuffling files around is something almost every programmer needs to do at some point so being able to delete things like above or move, copy, archive files is quite helpful.

Here's a list of most DOS commands with brief descriptions of what they do:

https://en.wikipedia.org/wiki/List_of_DOS_commands

[Sammy 735]

Nice! Thank you!

Jeez, in 1990ish I was pretty proficient in DOS because, well, you had to be. Windows has made me lazy over the years I suppose.

Earlier than that I was programming in Fortran 77 in high school and college at least so I understand the basic concepts..

And BTW, the word you were searching for when you said, "..for checking conditions (If, then, else)" is Boolean Logic..

[Carlo 4330]

Boolean Logic at it's heart is the fact a value is true or false compared to something using boolean operators such as "AND", "OR", and "NOT".

The commands like "if", "then", "else", "case", etc are know as conditional statements which can optionally use boolean operators.

[Sammy 735]

@cayars I ran 

C:\Users\tbeck>for %G in (.nfo, .jpg, .png) do forfiles -p "<C:\Emby Data\Emby Recordings" -s -m *%G -c "cmd /c echo @path will get deleted"

from a command line and got this

C:\Users\tbeck>forfiles -p "<C:\Emby Data\Emby Recordings" -s -m *.nfo -c "cmd /c echo @path will get deleted"
ERROR: The directory name is invalid.

C:\Users\tbeck>forfiles -p "<C:\Emby Data\Emby Recordings" -s -m *.jpg -c "cmd /c echo @path will get deleted"
ERROR: The directory name is invalid.

C:\Users\tbeck>forfiles -p "<C:\Emby Data\Emby Recordings" -s -m *.png -c "cmd /c echo @path will get deleted"
ERROR: The directory name is invalid.

 

[Carlo 4330]

Get rid of the less than sign you have in the pathes. <

[Sammy 735]

2 hours ago, cayars said:

Get rid of the less than sign you have in the pathes. <

Oops!

All fixed up now..

Did you see my PM?

[Carlo 4330]

2 hours ago, Sammy said:

Oops!

All fixed up now..

Did you see my PM?

Saw it and replied.   Glad you got the script fixed.

[go2amp 2]

So, when is this new LiveTV update due?  I bumped another thread (similar) before coming across this one.  Fancy scripts aside, it's a nuisance to have to implement them, or service restart scripts, or reboots to clear these directplay transcode caches full of orphaned, inactive stream files when Emby should be able to clean itself up natively.

[Carlo 4330]

When it's ready.  There is no estimated release date or timeframe that is being given.

[Luke 37057]

We are working hard on it.

[jacotec 7]

Is it already in the beta channel for Linux?

[Luke 37057]

5 minutes ago, jacotec said:

Is it already in the beta channel for Linux?

Not yet.

[go2amp 2]

On 8/13/2020 at 2:04 PM, cayars said:

When it's ready.  There is no estimated release date or timeframe that is being given.

 

On 8/13/2020 at 3:11 PM, Luke said:

Not yet.

Ok, thanks guys.  Keep us posted.  

[JERKBALL 1]

Fix for this would be very appreciated 

 

user@omv:/sharedfolders/Video/Recordings/transcoding-temp# ls -lah

	insgesamt 2,1T

	drwxrwxrwx+  2 emby emby 240K Aug 17 00:31 .

	drwxrwxrwx+ 10 root root  20K Aug 16 20:10 ..

	-rw-rw-rw-   1 emby emby  217 Aug 16 23:12 51F781.xml

	-rw-rw-rw-   1 emby emby  217 Aug  8 01:01 57F255.xml

	-rw-rw-rw-   1 emby emby  217 Aug 13 23:30 5B80A0.xml

	-rw-rw-rw-   1 emby emby 289G Aug 17 10:22 6449467e1a724032ac6e039f4558fd3b.ts

	-rw-rw-rw-   1 emby emby  217 Aug 12 23:42 67A961.xml

	-rw-rw-rw-   1 emby emby  217 Aug  4 22:40 A39249.xml

	-rw-rw-rw-   1 emby emby 451G Aug 17 07:34 a891f0d7acfe417c9fd0e87f6e6e066b.ts

	-rw-rw-rw-   1 emby emby  217 Aug  3 21:02 B79E3A.xml

	-rw-rw-rw-   1 emby emby 1,3T Aug 17 10:22 b7a7bcb5982a45c2b508c29e5282d0e7.ts

	-rw-rw-rw-   1 emby emby  217 Aug  8 10:58 D40D9C.xml

	-rw-rw-rw-   1 emby emby  58G Aug 17 10:22 d45c91bd065e4361a3e59927256b1edd.ts

	-rw-rw-rw-   1 emby emby  217 Aug  8 00:17 D6F57E.xml

	-rw-rw-rw-   1 emby emby  217 Aug  6 22:28 ED96DC.xml

	-rw-rw-rw-   1 emby emby  217 Aug  8 00:39 FDD0ED.xml

	-rw-rw-rw-   1 emby emby  217 Aug  8 01:24 FFB292.xml[/CODE]

[lastwraith 4]

Any further updates? I'm new to Emby and just ran into this. Running 4.5.2.0 and there were somehow 17k files in my transcoding-temp directory, one of which was 80GB in size. Many were 3 days old. Seems reasonable that Emby should at least clean up things 2 or more days old in this directory. We have TCLs with Roku built-in and there is no way I am going to get people in the house to stop hitting the Home button. In a perfect world sure, but it just isn't going to happen.

At least for now, some basic housekeeping for files older than 24hrs or so would be nice. I'll run a script for now but that file behavior can't be very good for smaller SSDs.

[Carlo 4330]

If you NAG your family members enough to back out of playback to the menu anytime they are done it will sink it eventually.  Been there, done that!  This isn't an Emby thing per say as some other software including Netflix and Hulu have issues with this at times and will keep streaming or keep things open as well when you use the home button and the app loses focus.

You can likely take your script and adjust it down to 1 day/24 hours and then run it every 3 to 4 hours via Task Manager or other scheduler on your OS and that will help.

For transcoding I like to tell people to purchase 256 to 512GB "throw away" SSDs for transcoding use.  They are dirt cheap these days so if you have to replace one every year or two it's worth it for the faster access.  My "throw away" drive is now going on 3 years use and it's constantly used all day long.

Luke has been looking at this and trying to improve this in the backend.  Still needs more work however but getting better.

[lastwraith 4]

I think you overestimate the cooperative desires of people in my household, haha. I understand it isn't necessarily an Emby thing but we don't really care if Netflix or Hulu stay open when the app loses focus since they aren't writing to disk on our media server. Also, it wouldn't be so bad if it were just a few files or if they were at least smaller. As it is, it's hard to deal with an 80GB file 2-3 days old (or more) that just sits there. I do appreciate your response though.

I have set the script to run daily and it will prune anything older than a day. That should be fine for me but I will adjust as needed. I manually ran it to check and it cleaned up the directory nicely. Thanks to "Sammy" so I didn't have to come up with something myself! I'm just not sure why there isn't a simple cleanup option in Emby for this, at least while the devs come up with a more elegant solution. I understand the app is basically at fault but a timed cleanup script to remove things 2 days or older (for example) would probably help out a lot of people, especially those who don't realize this is a concern until they find their hardware with a filled drive one day and have to track down the cause. I was expecting runaway log files or something initially.

I think I know the specific event that caused this and it definitely was my fault. I was trying to advance in an NFL game and Emby would not play the file if I tried to advance more than about 60 mins. I was trying a bunch of things before realizing that my media server was thrashing the disk and was starved for resources. Once I fixed that issue, Emby behaved. Of course this was all via the Emby app on the FireTV since our TCL Roku's can't advance more than a few minutes at a time because of the lack of onboard storage. Damn you TCL Roku! (Though they are otherwise pretty great and I realize this is an edge-case.

SSDs are pretty resilient at this point so I will probably go the route you suggest. Still, a timed reboot option or cleanup script would probably suffice for most people. People are picky though, so who knows.

I appreciate the efforts but I was just curious if any progress had been made.

[Carlo 4330]

@Luke Can you take a look at the transcoding cleanup routines and see if this can be improved more please?

[lastwraith 4]

Much appreciated, thanks again!

[tmock 0]

I have a 32GB ram disk (my Emby server is running on Win10 which has 64GB of RAM).  I'd prefer developers implement an option that would allow us to choose how much time live tv will retain for rewind on the fly.  This would give us plenty of flexibility since not all of us have the same amount of space to work with.

I'd probably set mine to an hour at most.  In my situation, this would be sufficient to allow two simultaneous live tv transcodes while also leaving room for other scheduled recordings.

Just providing an option to remove transcodes on a scheduled basis would not be ideal.

[Carlo 4330]

Hi, Ram discs are NOT the best use for this type of thing as they have a fixed amount of space available and not that big.  A 32GB ram disk would be quite easy to fill with normal playback activity.

You'd be much better off using an SSD drive for the transcode directory and allowing windows to use the memory for buffering/caching or other things.

[spoonchild 0]

Quote

A 32GB ram disk would be quite easy to fill with normal playback activity.

Respectfully I disagree with this statement. I have been doing the same to save on SSD and have much faster performance on transcodes. I have have 12-14 hours of full HD live TV(HD Homerun Prime) without an issue. Where I run into problems is that since Roku's don't disable, that after a few days of it not shutting down the transcode stream, it fills the drive up. With 32GB of transcode RAM disk I can have 15-20 people transcoding out of plex without an issue with space. 32GB is more than enough for transcodes IF you don't have this issue with them building and building up over 24+ hours of playback. This is why this thread exists is that it doesn't matter how much space you have, what type of drive you are using, it isn't enough due to this issue.

I 100% support the idea of for live TV limiting throttling the transcode in a reverse manor. Maybe it's time to look at a feature like Plex has that limits the transcode to x minutes ahead of playback, but also do this for x minutes of total playback. Live TV would benefit from the latter and better performance for playback from the first allowing admins to choose how much transcode sits ahead of the file that is being played.

Having an option to limit the playback transcode tp a time base would be perfect and solve this as well. I'm sorry but no one needs 48+ hours of Live TV rewind if they left the app connected. If you do, then you should just DVR that channel and have it record everything, and even then each "transcode" would only be the total time of that programs time slot.

[Carlo 4330]

Have a couple of users watch live TV from OTA for a few hours and watch what it does to the RAM Drive.  Two recorded OTA football games alone will fill that and that's with no other use. When you're using 15 to 20Mbps it doesn't take long to fill that. A Prime with HD video from most cable companies will be encoded in H.264 vs Mpeg2 and are much smaller but often the local channels like ABC, NBC, CBS, Fox are still broadcast in MPEG2 so on a Prime tuner with most cable providers it depends on channel.

Try playing back a 45GB Blueray rip that requires a remux.  A lot of people's transcode directories fluctuate in the 100 to 300GB per day depending on load, number of users and types of content. If you have any Roku users who don't back out of Live TV then Emby may not get notified they are not longer watching and it will fill up quickly.  There are many reasons why you want a couple hundred GB free for headroom in a dynamic environment like this to cover yourself.

I honestly don't see this changing with the present code since the New Live TV is around the corner and uses a different pipeline that won't do this.  So a "fix" is imminent for the Live TV issue but will come with the new Live TV.  Not sure that's good or not for you, but there is an end in site.

I still don't think 32 GB is a good idea for most systems and not something I'd recommend but maybe that's enough for your system which only you would know. 

I'm personally not a fan of limited transcode for reasons specific to Emby.  Emby for example can be set to NOT throttle and transcode the whole file to finish as quickly as possible.  This can be a huge thing for people using chapter or want to skip ahead OR for those with GPUs limited to 2 streams.  The faster you get that stream done, the faster it frees up the GPU.  But that's just my personal opinion ONLY.  What you mentioned is being considered and if it is implemented then a 32 GB Ram Drive would likely be fine in that situation if you don't mind giving up what you gain from not throttling and having the whole file transcoded and allowing FF/RW to react faster since it's already built and on disc ready to use.

You sort of covered some of this yourself and realize why 32GB at present isn't a good idea with the current software (Roku alone makes it not viable).  For all the aggravation of trying to get things to work with a RAM drive you can buy an SSD which will just works and are dirt cheap these days with sub $50 price tags.  You could then use the memory for delayed writes if you want to the SSD so most things will get stored in memory but flushed occasionally to SSD/Disc.  Another things you could do is setup a "disc" using something like SnapRaid with a combination of your RamDisk as the primary and an SSD or conventional disc to be used if the RAM disc fills up.  That's sort of a hybrid that would put most use on the RAM disc but give you additional storage when needed.