One small step for ffmpeg... one giant leap for Emby! (Looking for C developer(s) to help with transcode throttling)

[jabbera 23]

I don't understand what you are asking. It doesn't actually save any cpu usage. It just allows ffmpeg processes that have transcoded less then <X> minutes ahead of the current viewing position priority on the CPU. Once they get X minutes ahead they are changed to idle (or some lower priority then normal) and all ffmpeg processes at that priority share the cpu equally.

[jluce50 118]

how much cpu savings?

 

Perhaps it might help to clarify what the overall goal is. Throttling via priority may not result in an overall CPU savings, but it could/would result in an improved user experience. If there's nothing else going on there might be no apparent change, but if there are other transcoding streams (or other important tasks) then those would benefit by having streams that have a larger buffer lowered in priority.

[jabbera 23]

The other nice thing about this approach is you aren't constant switching the thread states of the ffmpeg process every 5 seconds. Since it will continue to transcode while idle if the transcoding rate > framerate it will never need to change back from idle unless the workload on the server changes and the transcoding rate drops. You also aren't messing with another processes threads.

[jluce50 118]

One other thing I just thought of. There is something to be said for maintaining complete control over the throttling mechanism. Hypothetically, even if the pause functionality was accepted into ffmpeg, there's nothing to guarantee it wouldn't be removed or changed in the future. Then we'd be up a creek and have to scramble for another solution. It's probably not likely, but it's worth taking into consideration...

[jabbera 23]

Perhaps it might help to clarify what the overall goal is. Throttling via priority may not result in an overall CPU savings, but it could/would result in an improved user experience. If there's nothing else going on there might be no apparent change, but if there are other transcoding streams (or other important tasks) then those would benefit by having streams that have a larger buffer lowered in priority.

 

This is a really important point. The goal is to give more CPU time to streams with the least amount of time transcoded, not stop transcoding once you reach a certain buffer size. The server should always be transcoding if there is a stream to be processing. You could even have a sliding scale. Less then 1 minute Highest, less then 3 normal less then 5 below normal, > 5 Idle.

[Luke 37113]

The goal is that plus also to improve single stream cpu usage. when the transcoding position jumps out ahead of playback position, we can throttle and drop cpu usage to near 0.

 

So the process suspension definitely had to go because it wasn't working, but by changing the interface now the above goal isn't really possible anymore.

[jluce50 118]

This is a really important point. The goal is to give more CPU time to streams with the least amount of time transcoded, not stop transcoding once you reach a certain buffer size. The server should always be transcoding if there is a stream to be processing. You could even have a sliding scale. Less then 1 minute Highest, less then 3 normal less then 5 below normal, > 5 Idle.

 

Yeah, this approach does provide quite a bit of flexibility, which I like. That said, I think setting it to anything above "Normal" should probably be an option, disabled by default. Lot's of people use MB on machines that aren't dedicated HTPC's (including me).

[jabbera 23]

but by changing the interface now the above goal isn't really possible anymore.

What do you mean by the quoted text? I Think you mean that using the schedule makes this goal impossible, if not ignore everything that follows.

 

By continuing to transcode at idle there is an increased chance that when you do jump ahead the transcoding is finished. If the jump ahead is within the transcoding buffer, but less then Xminutes, priority should bump to normal.

 

If the file hasn't been transcoded the point which the client jumps ahead to the controller should restart ffmpeg with an approiate seek at normal process priority.

 

All of this is my opinion wen though it's written as fact. It's just too hard to say I think ithink ithink

[jabbera 23]

Yeah, this approach does provide quite a bit of flexibility, which I like. That said, I think setting it to anything above "Normal" should probably be an option, disabled by default. Lot's of people use MB on machines that aren't dedicated HTPC's (including me).

Agreed by default but it could certainly be configurable for people who run dedicated systems.

[jluce50 118]

I think what Luke is saying is that there are other considerations than just the relative performance of transcodes. A lot of people care about stuff like power consumption, heat, fan noise, etc. and being able to lower overall processor usage is important to them. I'm not one of those people, so I'd be thrilled with the priority-based approach, but I'm not everybody .

[jabbera 23]

I think what Luke is saying is that there are other considerations than just the relative performance of transcodes. A lot of people care about stuff like power consumption, heat, fan noise, etc. and being able to lower overall processor usage is important to them. I'm not one of those people, so I'd be thrilled with the priority-based approach, but I'm not everybody .

I've never even thought of that. My machine runs in my basement:-)

[jluce50 118]

I think this is implemented in 5537 Alpha. Although it dramatically improved the CPU usage it has broke the multi stream transcoding stability. If I start second stream on another device first one got hangs. Sometimes it completely stops, sometimes hangs for a long time etc. It is practically impossible to stream multiple files to multiple device. But one stream works stable.

 

 

can someone else confirm that? if that's the case then we'll have to just wait for ffmpeg.

 

Assuming you still want to pursue this route, here's some interesting info. It looks like there may be a bug in how the downloadedPositionTicks is incremented. For the second stream (the one that never resumes after being suspended), it looks like the downloadedPositionTicks isn't accurate.

 

I let the file play until it ran out of buffer. Since the ffmpeg process was suspended, it just froze:

 

Here's what I'm seeing in the code when this happens:

 

 

 

So, since it's reporting that amount downloaded is significantly less than the amount transcoded, this always returns true (which results in PauseTranscoding() being called over and over). 

[Luke 37113]

Assuming you still want to pursue this route, here's some interesting info. It looks like there may be a bug in how the downloadedPositionTicks is incremented. For the second stream (the one that never resumes after being suspended), it looks like the downloadedPositionTicks isn't accurate.

 

I let the file play until it ran out of buffer. Since the ffmpeg process was suspended, it just froze:

 

Here's what I'm seeing in the code when this happens:

 

 

 

So, since it's reporting that amount downloaded is significantly less than the amount transcoded, this always returns true (which results in PauseTranscoding() being called over and over). 

 

Ok, so this was an application bug and not actually related to the process suspension methods. in which case the change i just added from jabbera might not have been  necessary.

[Luke 37113]

i'll check into this but let's not lose sight of what the real ultimate goal is. we need that command in ffmpeg.

[jluce50 118]

Would it be more trouble than it's worth to allow users to choose which method they want? "High performance" vs. "Lower CPU usage" or something. Personally, I don't care about CPU usage, but I do care about relative performance when I have multiple transcodes going. Providing this choice, to me, would put MB ahead of Plex, not just on par...

[Luke 37113]

i was thinking of some ideas of prioritization yes, but i think that comes after this. first things first, which is making one single operation as efficient as it can possibly be.

[jluce50 118]

Okay, cool. I totally agree. I just wanted to throw that out there...

[Luke 37113]

ok, i think i've got that fixed. give it a shot again in the build going up. thanks.

[jabbera 23]

i'll check into this but let's not lose sight of what the real ultimate goal is. we need that command in ffmpeg.

 

It's totally your project and I'll respect your decision, but I don't think pausing ffmpeg is what people who run a dedicated server would want. They want it always transcoding if there is a stream playing that hasn't been done yet. They will want streams with less buffer given higher priority.

[Luke 37113]

Except with sync and scheduled tasks it isn't that simple. there's other work the server might be doing that will eat up cpu. plus you might transcoding a whole movie but only watching half of it. i would think most people would want it to "just work" above all else, and then the second factor is how much cpu does it use while working. I am basing that off of what I see from users reporting feedback and issues here in the community.

[techywarrior 688]

Also, a lot of people want to combine their MB server with their storage. This will greatly help with people using NAS devices as their server.

[jabbera 23]

You guys will laugh but the pause key on your keyboard will pause transcoding and any key resumes. (At least on windows)

[techywarrior 688]

You guys will laugh but the pause key on your keyboard will pause transcoding and any key resumes. (At least on windows)

lol.

 

Probably not as good a solution as having a pause added to ffmpeg but funny. Does that just pause the single thread or every instance of ffmpeg? (just curious)

[jluce50 118]

Also, a lot of people want to combine their MB server with their storage. This will greatly help with people using NAS devices as their server.

 

I would personally tend to agree with jaberra, but I trust that Luke has a much broader perspective as to what MB users in general want. That said, I would think that either solution will greatly help NAS users. jaberra had a good point about not wasting CPU cycles by pausing when lowering priority would give all of the same performance benefits when it comes to multiple transcodes (and might even be preferable on low powered systems).

Edited March 2, 2015 by jluce50

[CBers 6771]

well i put in a windows-based throttle via process suspension and can confirm that it works. this really should only be a temporary solution until the ffmpeg changes are complete. I know others have been able to pause ffmpeg processes with success, but it's generally not a recommended practice for multi-threaded apps because this only pauses the main thread i believe. So if ffmpeg hasn't been designed for this, it's child threads may continue to operate which could create unexpected results.

It doesnt work.

 

Please see: http://mediabrowser.tv/community/index.php?/topic/18853-transcoding-problems/