One small step for ffmpeg... one giant leap for Emby! (Looking for C developer(s) to help with transcode throttling)

[FredipusRex 12]

The 'c' technique can work (even if it spits out some error text every time you use it) as long as the input streams are 'blockable'.

 

The input threads will continue to read data when the 'c' causes the main loop to stop. If the input streams are considered 'blockable', the av_thread_message_send call will eventually block on a mutex/condition when the input buffer fills. However, if the stream is non-blockable, it will error out. It then tries again and if it errors out a second time, it's fatal.

 

The test for non_blockable is pretty opaque but has something to do with whether the stream is seekable. If it isn't, the non_blocking flag is set and the failure state is possible.

 

At this point, it's probably moot - throw it in the alpha channel and see if anyone complains. My guess is that this would primarily negatively affect live streams, which the 'p'/'u' method might not handle either. The 'c' method is crude but it has the benefit of needing no changes at all.

Edited March 17, 2015 by FredipusRex

[FredipusRex 12]

Luke - One difference between the 'c' method and the 'p' method is that the input threads aren't idled and memory consumption will grow to the max allowed. This might explain why you're not seeing the CPU drop as much as expected. It might drop more once the input buffer is filled (if that buffer is too large, it might cause some OS swapping, which could also cause issues).

[techywarrior 688]

If the ffmpeg dev asked why we don't use 'c' can't things like this be explained? Unless there is an actual command that you can pass along with 'c' then it's really not even close to the same as the 'p' method that was developed.

 

He didn't seem to like the pull request but perhaps explaining why it differs from using a hack like 'c' and fixing the issue he described (not having an output to say "press p to unpause") would suffice to get him to approve the pull request.

Edited March 17, 2015 by techywarrior

[jluce50 118]

If the ffmpeg dev asked why we don't use 'c' can't things like this be explained? Unless there is an actual command that you can pass along with 'c' then it's really not even close to the same as the 'p' method that was developed.

 

He didn't seem to like the pull request but perhaps explaining why it differs from using a hack like 'c' and fixing the issue he described (not having an output to say "press p to unpause") would suffice to get him to approve the pull request.

 

I had the same thought. I suppose at this point there's no harm in asking, although I'm starting to get the feeling he's already made up his mind and is just looking for reasons to not be bothered with it. I'm a little wary of pouring more time into this if it turns out to be an exercise in futility.

 

Edit: I'm not terribly hopeful, but I sent him the following response. If he shoots this down, I think we've done all we can...

 

It appears that c/C doesn't pause the input threads, which results in

higher CPU usage than the 'p' pause method. Also, since the input

threads continue to read data, the input buffer will continue to grow,

which may lead to undesirable results. I'm not sure if this is by

design or not, so I'm hesitant to attempt to "fix" it. These are the

main reasons why my method was implemented differently than c/C.

 

If I were to add some visual feedback for 'p', is there anything I

could tweak that would make you more comfortable with this approach?

If not, would pausing the input threads on c/C be a viable option?

 

 

Edited March 17, 2015 by jluce50

[Luke 37090]

maybe rename the input to words such as pause/unpause to ease his concerns of the user inadvertently pausing with the p key

[jluce50 118]

Just got this response:

 

i think pausing the input threads with c/C is what should be done

 

Perhaps we can put this on the roadmap. Seeing as the current solution is still a big improvement I don't see this as critical at this point, unless testers experience problems (read: I could use a little break from ffmpeg ).

[Luke 37090]

Understandable Yes as you say, it's a nice win even as currently done. We can look to improve later.

[dark_slayer 103]

Sad to see them reject the work you guys did (at least that's how I'm interpreting the outcome)

 

That being said, running the beta server and none of my family have called me to complain yet. Nice improvement for me, because they rarely finish movies. Many will get paused for 24+ hrs, so I am glad that ffmpeg won't just finish up every time anymore. I've asked them not to leave things paused, but it's like asking a wall to talk. Speaking of that, how does this change something that sits paused long enough for ffmpeg to idle?

[dark_slayer 103]

Also, meant to mention my utilization hovers in the 10-18% range for ffmpeg when my family is watching a transcode. I haven't combed through different file sizes and codec combinations to get a real idea, just pulled up task manager and looked in on ffmpeg earlier today when someone happened to be watching. Clearly it's early to call, but this puts my utilization on emby at parity with what I used to see in LexLuthor

[MSattler 387]

Also, meant to mention my utilization hovers in the 10-18% range for ffmpeg when my family is watching a transcode. I haven't combed through different file sizes and codec combinations to get a real idea, just pulled up task manager and looked in on ffmpeg earlier today when someone happened to be watching. Clearly it's early to call, but this puts my utilization on emby at parity with what I used to see in LexLuthor

 

The only thing I have noticed is on occasion if a client disconnects without stopping playback first, is the ffmpeg will continue running?  Is it possible that it's paused when the client exists so it never get's the notice to stop transcoding?

[dark_slayer 103]

The only thing I have noticed is on occasion if a client disconnects without stopping playback first, is the ffmpeg will continue running? Is it possible that it's paused when the client exists so it never get's the notice to stop transcoding?

Can't say for sure. I've only seen this once so far, and I had to close ffmpeg from task manager (ran a couple hours beyond the client concluding playback)

 

All three of my family transcoders watched a movie yesterday and no hanging ffmpeg was there this morning. Can't conclude it's persistent, but if I see it again I'll try to capture the log this time

[MSattler 387]

Try just unplugging an android box during playback.

 

Thanks!

[richardap1 10]

The only thing I have noticed is on occasion if a client disconnects without stopping playback first, is the ffmpeg will continue running?  Is it possible that it's paused when the client exists so it never get's the notice to stop transcoding?

I see this a lot too. I have posted about it in the alpha -> server section

Edited March 25, 2015 by richardap1

[BAS 225]

Well I havent't been around to weigh in on this, I got bit with this new implementation and it doesn't blend well with how my server gets hit at prime time hours. Since its mainly staying at 25% on each transcode and no transcode ever finishes, my CPU still hits 100% and I get even more ffmpeg process hangs which make the server unreachable. While I didn't like that the old process hit 90% it still split the CPU percentage fine when other users wanted to start something with the big advantage of the copies finishing so if another user hits play the cpu is available again. Maybe I'm in a different camp then most, with 6 users playing at once and needing to transcode those streams due to limited upload bandwidth in my area.... regardless getting text messages about the server being down is becoming more and more of a problem over the last couple days and it comes down to this option, if my server can't ever complete the transcode then I have to many users transcoding at one time to use this new method.

 

Did this already get rolled into the official release?

[MSattler 387]

The official release does not include it yet. I thought there was supposed to be an option to disable this.

[BAS 225]

The official release does not include it yet. I thought there was supposed to be an option to disable this.

 

Ok I will try a rollback to official for now then as I'm not even at home to take care of this and where I'm at has limited internet most days. I tried the different toggles for transcode options with no difference in results and I saw nowhere to choose to disable this, which is probably more suited for people in my scenario.

[MrWebsmith 598]

an option on the transcoding panel would be good for those who want to keep the "race ahead and finish sooner" method that it used to do.

[BAS 225]

Is there some other place I should have put my post? If this rolls into official I guarantee there will be problems.

[Luke 37090]

the issues of stranded ffmpeg processes related to throttling are resolved. i think the issue is more that your perception of the final result is based on your experience during early testing stages, and that's just not an accurate assessment. there could still be ffmpeg processes stranded but most likely it is due to the roku thumbnail plugin as there is work that needs to be done there.

[BAS 225]

the issues of stranded ffmpeg processes related to throttling are resolved. i think the issue is more that your perception of the final result is based on your experience during early testing stages, and that's just not an accurate assessment. there could still be ffmpeg processes stranded but most likely it is due to the roku thumbnail plugin as there is work that needs to be done there.

 

Well I believe my initial post wasn't read, the problem is the new setup never allows a transcode process to finish so I'm still hitting 100% cpu and because of limited bandwidth almost all my users are transcoding. I know there is more users out there in my use case scenerio that never post and never will. I'm going to agree with Smith on the fact that a toggle of some sort should allow certain users to choose between these new throttling concepts and the way it stands in official now where it just burns through to get them finished which works perfectly fine for me and all my users.

 

and i dont use the roku thumbnail plugin.

Edited March 30, 2015 by BAS

[Luke 37090]

the transcoding will finish. if for some reason it's not, then it needs to be reported and looked at. it could be completely unrelated. 

[Maudite 7]

I'll chime in on my experience with this new work, as I have come across an issue.  My situation is probably a rather limited one, but perhaps some other xbox one users can let me know if they are having this issue.

 

So a little background:

 

Watching on Xbox One via the Browser.

Long time ago, when first starting a show, the frame rate is really bad (10 frames/ second approx).  Switching the quality up or down, to ANY other quality, even max, would re-start the transcode and everything is fine.  Then this issue disappeared for some time, and was good for probably 6 months (guessing now).  And then it re-appeared in the last 1 or 2 releases (prod releases).

 

The BIG problem.

 

Well, since the framerate issue was back anyway, and I wanted to see how things developed here in the dev line, I was allowing my server to run dev for the last month or so.  Things were looking good until I believe it was release 30000.  Now when starting a show, get the expected low frame rate (and I now see on the server CPU only goes to about 25% with new FFMPEG changes).  However, changing the quality does not fix the framerate, and framerate is much worse (2-3 fps).

 

 

I don't know if you guys care about this too much, as hopefully with win10 hitting xbone you will have an official app.  However, other than a Roku and some mobile devices, the xbox one is where we match most of our shows, so this is a big deal to me.  If you could please do some testing with web client on xbox one before going live with these changes, unless as mentioned by the above request, this feature can be turned off.

 

Thanks.

[Luke 37090]

Yea we do care, and that sounds unrelated. Please see 

http://mediabrowser.tv/community/index.php?/topic/790-how-to-report-a-problem/

[Vicpa 559]

+1 to have option to disable

[Maudite 7]

Yea we do care, and that sounds unrelated. Please see 

http://mediabrowser.tv/community/index.php?/topic/790-how-to-report-a-problem/

 

Thanks Luke, I will report it officially.