Transcoding-Temp folder not deleting files

[achalmers 3]

Hi.  I have migrated my Emby service to running on a Win10 VM.  I am noticing that the transcoding-temp folder is not deleting its files when users get done streaming a video.  Due to VM resources I only allocated 56GB of storage space as an E: drive for the EmbyVM specifically for the transcoding-temp folder.  I am finding that even with no users / active sessions there are still files taking up space in that folder.  I repetitively have to go in and manually clear that folder to allow Emby to continue working again.  What steps can I take to diagnose why this folder isnt deleting files ?  

[Carlo 4561]

I'd bump that way up as 56GB of space will often not be enough space. 
You don't want to be stingy on drive space for your media servers when you're storing Terabyte of media.

Many of us use dedicated SSD just for the transcode folder which of course is optional but helps with IO performance.  I'd not use a virtualized drive for this if possible as that will be hurting your performance compared to a raw drive.

[achalmers 3]

4 hours ago, cayars said:

I'd bump that way up as 56GB of space will often not be enough space. 
You don't want to be stingy on drive space for your media servers when you're storing Terabyte of media.

Many of us use dedicated SSD just for the transcode folder which of course is optional but helps with IO performance.  I'd not use a virtualized drive for this if possible as that will be hurting your performance compared to a raw drive.

Ok I'll up it, but I never saw my original bare bones server consume more than around that number and it always flushed files out when streams were done.  Are you saying day old files with no active streams is normal?

The logical drive is part of a RAID10 volume with 8 - 15k rpm drives.  Speed tests are like 2000 MBps read / write.  I can allocate more I just didnt want to.  I would think the folder should self delete after streams end.  

Edited March 22, 2021 by achalmers

[Happy2Play 9781]

3 minutes ago, achalmers said:

Are you saying day old files with not active streams is normal?

No this is not normal as a properly stopped session will/should clean up that session.  But a improperly stopped session will leave behind that sessions transcode and a Emby restart should clear out that abandoned session.

[achalmers 3]

Just now, Happy2Play said:

No this is not normal as a properly stopped session will/should clean up that session.  But a improperly stopped session will leave behind that sessions transcode and a Emby restart should clear out that abandoned session.

That makes sense but IDK how to tell what an improperly stopped session is or who is doing it.  Maybe I can come up with some kind of cron job to delete files older than 12 or 24 hours

[Happy2Play 9781]

7 minutes ago, achalmers said:

That makes sense but IDK how to tell what an improperly stopped session is or who is doing it.  Maybe I can come up with some kind of cron job to delete files older than 12 or 24 hours

Only way would be using the session id (the filenames) and compare to the server log or possibly ffmpeg log.

But others have done their own scripts in other topics.  

here is one example

The client this happens the most on are Roku devices as user click home instead of back arrow to send the proper stop command to the server.

But the dev has mentioned they are looking into this issue as it is a common one.

[achalmers 3]

2 minutes ago, Happy2Play said:

Only way would be using the session id (the filenames) and compare to the server log or possibly ffmpeg log.

But others have done their own scripts in other topics.  

here is one example

The client this happens the most on are Roku devices as user click home instead of back arrow to send the proper stop command to the server.

But the dev has mentioned they are looking into this issue as it is a common one.

The most used device (by far) for my system is an xbox one.  I dont have any Roku devices nor do any of my remote users.  

 

I just created this powershell script and it worked.  I'm going to schedule it to run every 12 hours and see how it goes.  

Get-ChildItem -path "E:\Emby\transcoding-temp" | where {$_.Lastwritetime -lt (date).addhours(-12)} | remove-item 

[Luke 42079]

We are looking into this. Thanks for reporting.

[Ronstang 294]

5 hours ago, Happy2Play said:

No this is not normal as a properly stopped session will/should clean up that session.  But a improperly stopped session will leave behind that sessions transcode and a Emby restart should clear out that abandoned session.

You guys keep saying properly stopped session like we are all idiots.  Do any of you have women or children...let alone a special needs one in your house?  Doesn't matter how many times you tell them they think it functions like the cable box....it doesn't....but since you aim to replace the cable box IT SHOULD.  Plus, I can't count the times I hit the wrong button on my Firestick remote laying in bed watching LiveTV only to realize I now had a tuner that would NEVER release until I shut down and restarted emby....which is not an option for me very often because I am usually recording something.  Because of this issue I had to stop using LiveTV all together except to record stuff I want to archive so 4 months after I purchase Premiere solely for LiveTV I still have a cable box because I can't use LiveTV because of these issues.

The fact that emby has to transcode TV recordings in progress when there should be no need to transcode as it should play them all natively is another matter all together.

[rbjtech 5284]

Personally, there is no need to use an SSD for the temp files used for transcoding.   Not only is the write/read of these files linear (it is not random, so seek performance is not a problem) but you are also massively wearing out your SSD by writing GB's worth of data to them for no performance gain.

For cache/db etc - then yes SSD gives a massive performance gain, but move the temp transcode area to a normal HDD if you can - you will see zero degraded performance vs SSD (unless it's the same HDD used by other high I/O system activity).  I use a general 'swap/junk' disk for this purpose.

 

 

[achalmers 3]

Thanks for the responses.  Powershell script is working perfectly so far.  I'll keep an eye on it.  I set it to run every 6 hours and delete any files older than 6 hours.

[Carlo 4561]

I so disagree with that rbjtech.  I think putting your transcoding on a cheap "throwaway" SSD is the best thing you can do to make Emby run faster with multiple transcodes or especially multiple people watch TV.  We see this time and time again here in the forums.  Watching TV can become IO bound on some system and moving the transcode directory to SSD really helps with this IO.

You can get cheap 250GB drives these days for $35 or so which to me makes them throwaway.  I'm on my 3rd year with my "throwaway" drive and have a spare ready to install when it finally wears out.  I use this drive ONLY for transcoding and DVR recording so no need to backup or be concerned when I finally pound it into the ground from heavy use.

I do not use high my hi end Samsung Pro SSD drives for transcoding but use them for the cache and metedata like you mentioned.

Now with that said, if you aren't having any performance issues or any issues with transcoding using a spindle drive then it likely doesn't matter. But if you already are using a more expensive SSD for multiple purposes like transcoding and cache I'd think hard about a throwaway drive just because transcoding does write a lot.

[achalmers 3]

I kind of agree with both of you.  Although I'm using a RAID 10 array of 15k rpm drives and performance isn't an issue here, I can see where multiple sessions / multiple users may start to push the limits of a single eco-friendly consumer drive.  Also, some of these SSD's are pretty impressive these days.  I have a 5 or 6 year old Samsung  950pro that has had like 250 TB written and it's still performing just fine and passing all tests.  

[rbjtech 5284]

20 minutes ago, cayars said:

I so disagree with that rbjtech.  I think putting your transcoding on a cheap "throwaway" SSD is the best thing you can do to make Emby run faster with multiple transcodes or especially multiple people watch TV.  We see this time and time again here in the forums.  Watching TV can become IO bound on some system and moving the transcode directory to SSD really helps with this IO.

That's fair enough - for extreme cases, then I guess an SSD is easily going to outperform a HDD.   

Being brutally honest, I've no idea why emby has to keep the .TS 2 second files post being transcoded anyway - once a reasonable buffer has been created to create the streaming .TS (say 5 mins) then why not simply delete the temporary TS files that are more than 5 minutes old - they server no purpose - they are never re-used ?

If this was engineered properly, then a RAM disk would be the best solution - bettering any form of SSD with zero wear issues ...  

Edited March 23, 2021 by rbjtech

[ebr 16184]

8 minutes ago, rbjtech said:

they server no purpose - they are never re-used ?

Unless you skip back further than that...

In the future we will look to optimize it to do a better job of managing the space but it currently does it on a per-session basis.

[rbjtech 5284]

-deleted-

 

Edited March 23, 2021 by rbjtech

[Carlo 4561]

1 hour ago, rbjtech said:

If this was engineered properly, then a RAM disk would be the best solution - bettering any form of SSD with zero wear issues ...  

 

1 hour ago, ebr said:

In the future we will look to optimize it to do a better job of managing the space but it currently does it on a per-session basis.

I think that would be very interesting.  These days with high end GPUs the cost to transcode isn't much and of course if it were only a remux then that can be done very easily again if needed.  To me the challenge would be non index base files like TS which could make rewinding to a specific spot non precise but livable.

It would be cool to be able to use a 16 or 32 GB ram drive and know Emby is making the best use of the space as possible.  With just a tad bit of "cleanup smarts" it would be really useful.  By this I mean keep ram disk 95% full (a bit of working room), delete remux packets first keeping only X amount of packet (5 to 10 mins worth), then cleanup GPU based transcodes and lastly cleanup packets from CPU transcodes but the whole thing dynamic. That would allow the greatest chance of having more "expensive to produce" packets in the RAM disc.

This would also solve Live TV issues where a person turns on the TV at night and falls a sleep or bing watches news channels all day long.

With some decent cleanup logic any small SSD like 100GB would be plenty of space and never run out of space!

[achalmers 3]

Just an update the powershell script is working nicely.  No issues since implementing that and every time I have logged in the drive is clear.  I would think one of the easiest things Emby could implement is the option to do this as a built in server function.  I would bet 98% of installs have no need for files older than 6 hrs in the transcoding-temp folder and an option for auto folder clearing would eliminate a lot of issues.  Here is the last version of the script that I've been using

 

Get-ChildItem -path "E:\Emby\transcoding-temp" | where {$_.Lastwritetime -lt (date).addhours(-6)} | remove-item