Jump to content

Throttle Cleanup


jaketame

Recommended Posts

jaketame
On 03/10/2020 at 04:51, Luke said:

No, we just haven't' gotten to it yet, but hopefully soon. Thanks.

Any luck on this one please? Thanks :)

Link to comment
Share on other sites

  • 3 months later...
Napsterbater

I'm fine with keeping the current entire transcoded session if it means helping with seeking around, but what really needs to be fixed is Emby not deleting every transcoding sessions temp files.

I noticed this when I created a RAMdrive for Temp Transcoding files, there will be days worth of transcoding temp files built up of session long closed and completed that I am having to manually clean up.

Link to comment
Share on other sites

Happy2Play
6 minutes ago, Napsterbater said:

I'm fine with keeping the current entire transcoded session if it means helping with seeking around, but what really needs to be fixed is Emby not deleting every transcoding sessions temp files.

I noticed this when I created a RAMdrive for Temp Transcoding files, there will be days worth of transcoding temp files built up of session long closed and completed that I am having to manually clean up.

Without specific example it is hard to say why the transcode session files were abandoned.  And the only way Emby currently cleans them up is with a restart.

Link to comment
Share on other sites

Napsterbater
13 minutes ago, Happy2Play said:

Without specific example it is hard to say why the transcode session files were abandoned.  And the only way Emby currently cleans them up is with a restart.

And that is the problem, There are apparently many reasons for them to be abandoned mainly because it seems a client MUST tell emby it is stopping, and some client apparently still do not, or there could be some network issue causing the stop message to get though.
But even if a stop message is not making to emby, seems emby is smart enough to remove the session from the active sessions, so maybe that threshold what ever it is should be used instead of the current implementation.

Link to comment
Share on other sites

4 hours ago, Napsterbater said:

And that is the problem, There are apparently many reasons for them to be abandoned mainly because it seems a client MUST tell emby it is stopping, and some client apparently still do not, or there could be some network issue causing the stop message to get though.
But even if a stop message is not making to emby, seems emby is smart enough to remove the session from the active sessions, so maybe that threshold what ever it is should be used instead of the current implementation.

It is, however there are clearly some situations in which this doesn't occur so we need to identify them.

Link to comment
Share on other sites

  • 4 weeks later...
On 4/18/2021 at 2:59 PM, jaketame said:

Eager beaver here, any progress on this part please? Thanks

Not yet, but it's on our to do list. Thanks.

  • Thanks 1
Link to comment
Share on other sites

  • 6 months later...

I mean if this would ensure the transcode folder stays at a manageable size (under 5 or 6 GB), then we could put it on a ram drive and keep those files on the fastest possible medium. The data is needed immediately, and it's not meant to be retained, so it seems silly to write it to any kind of long term storage device. No need to keep the whole transcode alive - I'm pretty sure my HW transcoding is fast enough to recreate anything needed when skipping back and forth through a video.

Link to comment
Share on other sites

Part of this would be to not have to store the whole file either. You would only need to store as much as the current client needs at any one time (at the minimum).  If you can store more of the old parts of the transcode that's fine in case the user rewinds it would be available but if not present the transcoder would redo those sections as needed.

A small routine would cycle to remove the oldest segments of all transcode freeing up space. When ever a specific media file is done playing all segment/files used by that session are immediately removed.

A RAM disc at that point would elevate a lot of problems people have with Live TV in general as it would substantially reduce disc IO on the system and wear and tear. With the price of memory these days a simple/cheap 16GB RAM upgrade would have an amazing speed increase for things like this.  Would be even better for people who can setup RAM discs of 32, 64 or 128GB.

Technically, there could be an option in Emby to use storage for the transcode or Memory.  The user could select the amount of memory used and Emby would allocate that itself and manage it.  This would allow it to be even more optimal vs going through the disc IO subsystem to access the RAM disc.

Edited by cayars
Link to comment
Share on other sites

I'm afraid, but I have to disagree on that whole subject.

With regards to using a RAM disk for transcoding-temp, I can only say: Don't ever do this! It's a very very bad idea!

You don't even need an SSD for transcoding-temp, but (important!): It should be a local disk - no NAS, no SAN or anything like that. 
For a perfect setup, you'd have an average magnetic disk - but dedicated only for that single purpose.

In a while, I'll come back and explain why.

  • Agree 1
Link to comment
Share on other sites

Napsterbater
35 minutes ago, softworkz said:

With regards to using a RAM disk for transcoding-temp, I can only say: Don't ever do this! It's a very very bad idea!

And the reason is?

I've been using a ram disk for my transcode temp for about a year now with a scheduled task to clean up files older than I believe I have it currently set for 6 hours. The average maximum RAM for the RAM disk is about 11 GB plus or minus but usually much less.

The system it's on has 64 gigs of RAM with about an average of 30 gigs free at any given time. Of course there's always the risk of running out of RAM disk but again in my setup I haven't had that problem.

Link to comment
Share on other sites

4 minutes ago, Napsterbater said:

And the reason is?

 

41 minutes ago, softworkz said:

In a while, I'll come back and explain why

 

  • Like 2
Link to comment
Share on other sites

1 hour ago, softworkz said:

With regards to using a RAM disk for transcoding-temp, I can only say: Don't ever do this! It's a very very bad idea!

Damn. TWO verys?? 😉

I mean it's not like Vegas with your parents, right? It can't be that bad.

Edited by C.S.
  • Haha 2
Link to comment
Share on other sites

It's a complex subject and I wanted to respond to some other things first and see whether I'd be still in the mood to explain 😉
In fact, the recent couples of posts are covering so many things that I'm not quite sure where to start. Let's go for the ramdisk part first:

First of all: Sure! We have all learned that we can improve performance of certain processing tasks by things like using SSDs instead of conventional HDs and ultimately even using RAM disks.

What are those "certain tasks" that can be accelerated by such measures?

=> IO-bound tasks, i.e. tasks where the limiting factor is IO throughput

Is this actually a problem that we're having with transcoding that we'd need to solve?

=> No - not at all. Even with the fastest hw accelerations or even when assuming one that would be infinitely fast - the source data needs to come from somewhere. It doesn't come from RAM - it comes from some disk. The output bandwidth when transcoding is usually much smaller than the input, so what comes from some disk storage (network or local) can easily be saved to another disk storage.

Are there any reasons at all, why we would want to accelerate disk IO?

Now, let's assume an odd case, like 5 transcodings of videos from 5 different source disks, with the results all landing at our single disk with the transcoding-temp folder.
Even in that case, we would need an extremely powerful hardware acceleration to saturate the transcoding-temp disk's, IO capacity. And there, we would have each of those 5 transcodings running at a multiple (e.g. 5x or even 50x) of the required transcoding speed (1x).
By default, we are running transcodings "as fast as possible". That's not really a good thing, because this will always cause to max out one of the following resources: CPU processing, system memory IO, disk IO, hwa IO or hwa processing - to its limit, even though that wouldn't be required in any way to improve the user experience. It's often a waste of resources - e.g. when a user stops watching after 5 minutes and we have already transcoded the whole video..

To deal with those things and use resources more wisely, we have introduced throttling - just the opposite of "as fast as possible", for all those cases where transcoding is running much faster than needed.

Looking at the other end: None of all those cases where transcoding is too slow, can be accelerated by using a RAM disk.

Actually, this has just the opposite effect: it can slow down your transcodings instead of accelerating.

And not just that: A ramdisk can even negatively affect Emby server operation in general.

The problem with the throttling we have is that it is binary - like a refrigerator: there's just ON or OFF switching over time. While it's on, it still runs with maximum speed. Transcoding itself requires a lot of memory IO - all this IO bandwidth will be taken away and not available for regular Emby operation.

And now you want to add a ramdisk on top of this, creating even additional memory IO for writing and reading? Very bad idea! 
The more shoulders on which we can put these things, the better - the right "shoulder" for transcoding temp is a disk. Even when the CPU does the transfer of memory in either case - it's still totally different: RAM is just not optimized for such access patterns. When you use the RAM in this way, you will permanently flush the memory caches (specifically L3) with other data, and this can in turn very badly affect sw filtering operations where the same memory needs to be accessed repeatedly.

Edit: And as mentioned, also Emby Server and all other server operations. 

In Summary

With a RAM disk, you would:

  • Either not accelerate anything or at best, accelerate something that is already running much faster than needed
  • Use memory IO bandwidth and CPU cycles (for memcopy)  - which are badly needed for other operations
  • Negatively impact overall server operation
Edited by softworkz
Added summary
  • Agree 1
  • Thanks 1
Link to comment
Share on other sites

19 minutes ago, C.S. said:

I mean it's not like Vegas with your parents, right? It can't be that bad

LOL - I'd say it's probably worse than that, but it might depend on your parents as well... 🤣

 

Link to comment
Share on other sites

Things are often different than expected with those high-throughput operations.
One of my favorite examples that I've mentioned at a number of other occasions is this:

"Please help - I have a new powerful GPU for hw acceleration, but now my CPU goes to 100% when transcoding"
(the explanation being that the new GPU is transcoding video so fast, that the CPU can't even keep up with audio transcoding)

  • Like 1
  • Agree 1
Link to comment
Share on other sites

Right on. Thanks for the info. I was thinking of it as a way to use excess ram. I think I'm just trying to justify a huge ram upgrade.

Link to comment
Share on other sites

11 minutes ago, C.S. said:

Right on. Thanks for the info. I was thinking of it as a way to use excess ram. I think I'm just trying to justify a huge ram upgrade.

I would rather invest in a dedicated magnetic HD for transcoding-temp and a separate (from the OS) M.2 or SSD storage for Emby Server logs, metadata and library (I mean one - not three.. .).

Edited by softworkz
Link to comment
Share on other sites

Napsterbater
1 hour ago, softworkz said:

What are those "certain tasks" that can be accelerated by such measures?

=> IO-bound tasks, i.e. tasks where the limiting factor is IO throughput

Is this actually a problem that we're having with transcoding that we'd need to solve?

=> No - not at all. Even with the fastest hw accelerations or even when assuming one that would be infinitely fast - the source data needs to come from somewhere. It doesn't come from RAM - it comes from some disk. The output bandwidth when transcoding is usually much smaller than the input, so what comes from some disk storage (network or local) can easily be saved to another disk storage.

I do it solely for the purpose of cutting down on IO on the Disk array, while yes the reads come from the disk, no reason the writes must back on it, which in my case is spinning rust but what still frees up IO time for other task, but even just cutting down on writes on SSDs is a good reason (minor but its not nothing), it is in no way is about speeding up the transcoding task itself.

You seem to mainly focus on "the speed of transcoding", there are other reasons, you also seem to assume all servers hosting emby are already close to their limits.

1 hour ago, softworkz said:

And now you want to add a ramdisk on top of this, creating even additional memory IO for writing and reading? Very bad idea! 

I VERY minor amount of bandwidth in comparison to system RAM bandwidth, and again IO time on spinning rust is more valuable in my case.

 

3 hours ago, softworkz said:

With regards to using a RAM disk for transcoding-temp, I can only say: Don't ever do this! It's a very very bad idea!

While you do point out some considerations that should be taken into account, they are far from the Doom and gloom warning you posted.

Link to comment
Share on other sites

22 minutes ago, Napsterbater said:

VERY minor amount of bandwidth in comparison to system RAM bandwidth,

That's the wrong assumption you are making. The transcoding operations itself are already taking a large amount of memory bandwidth, especially when some parts can't be done in hardware. As mentioned already, your disk IO might only approach its limits in situations with many simultaneous transcodings, at high speeds - but in those situations, your memory IO is of much higher concern and it would even be a good thing when it would be limited by disk write performance rather than hitting memory IO limits.

I agree on one thing: a disk array for general use is not a good place for transcoding temp (in high performance scenarios! - no problem for average setups), hence my suggestion for a separate local disk for this. That's much better than using RAM for it.
You can add more RAM, but you cannot increase the bandwidth for accessing neither the caches.

When you look for it, you'll find many articles about the pros and cons of using ram disks - the general conclusion, as you'll find out, is that ram disks are not a good recipe in general, but can help in certain very specific cases. 

What I'm saying is that Emby transcoding is not one of such cases.

Edited by softworkz
Link to comment
Share on other sites

6 minutes ago, softworkz said:
27 minutes ago, Napsterbater said:

VERY minor amount of bandwidth in comparison to system RAM bandwidth,

That's the wrong assumption you are making.

When you want to do some math, you must not forget that video processing (same as decoding-to and encoding-from) is done on uncompressed frames:

A 4k image has 3840×2160=8,294,400 pixels for simplicity, let's assume BGRA with 4 bytes per pixel, makes 32 MB per video frame. 
With 30fps, that already makes 960MB per each second of video that might need to be moved around. When it needs to be copied back and forth between GPU and CPU memory, that's already 2GBps. When we would have 5 simultaneous transcodings at 5x speed, this would make 50 GBps. And that is only for one transfer from GPU-to-system memory and back. It doesn't include all the other memory IO that is happening at the same time, so you could end up at an even higher value. All that is everything but "very minor".

Granted - it's an extreme example, but I hope it helps to understand the dimensions of these things a bit better.

  • Agree 1
Link to comment
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now
×
×
  • Create New...