One small step for ffmpeg... one giant leap for Emby! (Looking for C developer(s) to help with transcode throttling)

[Carlo 4330]

The math is super easy to do this way and it allows jumps in movies without having to waste cpu time transcoding stuff that was skipped.

 

In Plex on my systems I have 120 set as the amount of time to transcode in each cycle and 60 seconds to buffer before throttle.  This works really well for me with a 1st gen i7 with 6 to 10 streaming sessions without running into issues.

[Luke 37056]

The math is super easy to do this way and it allows jumps in movies without having to waste cpu time transcoding stuff that was skipped.

 

In Plex on my systems I have 120 set as the amount of time to transcode in each cycle and 60 seconds to buffer before throttle.  This works really well for me with a 1st gen i7 with 6 to 10 streaming sessions without running into issues.

 

Ok i think we are talking about two different things here.

 

You saying segment media into chunks - we already do for HLS, no problem there. Rejoining those is perfectly fine.

 

It's the extension on top of that that I think you aren't getting right. You're suggesting stopping the process after say, the first 20 chunks, then later instructing ffmpeg to start again at an offset. and seeking to that offset is the part that is inexact and could be off by an entire second (or worse).

 

Maybe it could be made closer to exact if we were to seek to a byte offset instead of time, but we don't have that information.

[Carlo 4330]

Ok i think we are talking about two different things here.

 

You saying segment media into chunks - we already do for HLS, no problem there. Rejoining those is perfectly fine.

 

It's the extension on top of that that I think you aren't getting right. You're suggesting stopping the process after say, the first 20 chunks, then later instructing ffmpeg to start again at an offset. and seeking to that offset is the part that is inexact and could be off by an entire second (or worse)

 

I understand what you are saying but I'm saying back. Don't assume.  Test it yourself by creating 5 second chunks.  Do the first 5 minutes of a movie.

Take the chunks created and now combine them back into one 5 minute video.

Use vlc or anything to now watch/listen to it.

[Carlo 4330]

are you really sure that's what they do? if so then that would mean you can't throttle when the media is unseekable or you don't know the duration (e.g., internet channel content). 

[Luke 37056]

ok, thanks. so they are pausing altogether. so we can play with a couple different methods, one being pausing the process altogether, another being separate ffmpeg instances.

[jluce50 118]

If that's all Plex is doing, I wonder why they would bother with a custom ffmpeg build? 

[Luke 37056]

maybe their solution has evolved to where it's not as necessary as it used to be. or maybe they use it to get more detailed reporting out of the transcoding process than just what the regular log provides.

[techywarrior 688]

I am glad we are talking about this BEFORE trying a custom ffmpeg build. Sounds like there is a good possibility that we can get the same, or similar, experience without the hassle and without the need to rebuild everytime there is a fix or improvement in ffmpeg.

[Carlo 4330]

If any of you guys would like to try this grab a movie from your collection grab and mp4 rename it movie.mp4

 

create a batch file "test.bat" and put this in it:

ffmpeg -ss 00:00:00.00 -i "movie.mp4" -t 00:00:05.00 -c:v libx264 -c:a libvo_aacenc -avoid_negative_ts 1 "1.mp4"

ffmpeg -ss 00:00:05.00 -i "movie.mp4" -t 00:00:05.00 -c:v libx264 -c:a libvo_aacenc  -avoid_negative_ts 1 "2.mp4"

ffmpeg -ss 00:00:10.00 -i "movie.mp4" -t 00:00:05.00 -c:v libx264 -c:a libvo_aacenc  -avoid_negative_ts 1 "3.mp4"

ffmpeg -ss 00:00:15.00 -i "movie.mp4" -t 00:00:05.00 -c:v libx264 -c:a libvo_aacenc  -avoid_negative_ts 1 "4.mp4"

ffmpeg -ss 00:00:20.00 -i "movie.mp4" -t 00:00:05.00 -c:v libx264 -c:a libvo_aacenc  -avoid_negative_ts 1 "5.mp4"

ffmpeg -ss 00:00:25.00 -i "movie.mp4" -t 00:00:05.00 -c:v libx264 -c:a libvo_aacenc  -avoid_negative_ts 1 "6.mp4"

ffmpeg -ss 00:00:30.00 -i "movie.mp4" -t 00:00:05.00 -c:v libx264 -c:a libvo_aacenc  -avoid_negative_ts 1 "7.mp4"

ffmpeg -ss 00:00:35.00 -i "movie.mp4" -t 00:00:05.00 -c:v libx264 -c:a libvo_aacenc  -avoid_negative_ts 1 "8.mp4"

ffmpeg -ss 00:00:40.00 -i "movie.mp4" -t 00:00:05.00 -c:v libx264 -c:a libvo_aacenc  -avoid_negative_ts 1 "9.mp4"

ffmpeg -ss 00:00:45.00 -i "movie.mp4" -t 00:00:05.00 -c:v libx264 -c:a libvo_aacenc  -avoid_negative_ts 1 "10.mp4"

ffmpeg -ss 00:00:50.00 -i "movie.mp4" -t 00:00:05.00 -c:v libx264 -c:a libvo_aacenc  -avoid_negative_ts 1 "11.mp4"

ffmpeg -ss 00:00:55.00 -i "movie.mp4" -t 00:00:05.00 -c:v libx264 -c:a libvo_aacenc  -avoid_negative_ts 1 "12.mp4"

ffmpeg -f concat -i concat.txt -codec copy output.mp4

 

Next create a text file name "concat.txt" and put this in it:

file '1.mp4'

file '2.mp4'

file '3.mp4'

file '4.mp4'

file '5.mp4'

file '6.mp4'

file '7.mp4'

file '8.mp4'

file '9.mp4'

file '10.mp4'

file '11.mp4'

file '12.mp4'

 

run the batchfile

 

It will break the mp4 down into 12 chunks of 5 seconds then will combine them back into output.mp4

 

We would actually want to do it slightly different to be even more precise and to explicitly set the frame to start on but this give you an idea.

[Carlo 4330]

I think this was already mentioned but we should easily be able to change the priority and/or affinity of ffmpeg in real time on any operating system. We could essentially set to low priority or remove cores  This method should be a slam dunk but wouldn't offer the same benefits as "chunking" when a user seeks to a different part of the file.

[Carlo 4330]

Was eating dinner and happen to think of another way.  Linux is easier as you can pause a process/program but windows typically doesn't give you that option easily.  We could however use:

NtSuspendProcess

NtResumeProcess

 

 

We could also use:

SuspendThread

ResumeThread

Example here: http://stackoverflow.com/questions/71257/suspend-process-in-c-sharp

 

Both options would easily work on windows but I'm not sure about mono as I don't use it.

 

Just thought I'd throw those options out while I was thinking about it.

Edited February 26, 2015 by cayars

[thefirstofthe300 291]

Has anyone looked heavily into gStreamer as a possible replacement backend?

 

I know this has been mentioned before but it seems like if you are looking for C coders, it might be worth looking at gStreamer too since it is built using C. gStreamer would also add the bonus of GPU decoding and a completed DASH implementation from a cursory look.

 

I would help but I just don't know C well enough. If however, the gstreamersharp bindings were finished, I might be able to help here.

[Luke 37056]

 

Was eating dinner and happen to think of another way.  Linux is easier as you can pause a process/program but windows typically doesn't give you that option easily.  We could however use:

NtSuspendProcess

NtResumeProcess

 

 

We could also use:

SuspendThread

ResumeThread

Example here: http://stackoverflow.com/questions/71257/suspend-process-in-c-sharp

 

Both options would easily work on windows but I'm not sure about mono as I don't use it.

 

Just thought I'd throw those options out while I was thinking about it.

 

 

i would rather start with the pausing process approach instead of separate ffmpeg processes. In theory, you could grab the Thread from the Process object and then there's a Suspend method. Only question is whether mono has implemented it for linux, bsd and linux. Worst case scenario we could have different implementations for different operating systems.

[Carlo 4330]

Yea ideally you want to explore the different methods available for each operating system.  If they all have a suspend feature then that's probably easiest.  If not then the thread method which is easy in a loop.  We could then encapsulate/hide this and just create a simple interface to it.

 

Ironically, I stumbled upon this: http://mediabrowser.tv/community/index.php?/blog/1/entry-104-media-browser-305366-released/

Which says you had already implemented throttling.

 

What happened?  Which implementation did you try with that?

 

Carlo

[jluce50 118]

i would rather start with the pausing process approach instead of separate ffmpeg processes. In theory, you could grab the Thread from the Process object and then there's a Suspend method. Only question is whether mono has implemented it for linux, bsd and linux. Worst case scenario we could have different implementations for different operating systems.

Not sure if you saw this as I added it later in an edit...

 

Edit: SuspendThread comes from kernel32.dll. It looks like Mono.Unix.UnixProcess.Signal *might* be the equivalent (using SIGSTOP and SIGCONT). I'm not familiar with Mono (or unix/linux), though, so take this with a giant grain of salt.

 

Not sure about bsd, but this is at least worth looking into.

 

Just curious, do you see process pausing as the end game for this? How much effort is it worth in the long-term to have granular control over the encoding process? It would require a larger work effort on the MBS side as well, since that's where all the logic would be.. 

 

The pausing approach does solve the problem of unnecessary encoding (if a user quits part way through), but it only partially resolves the issue of competing threads. Something analogous to QOS on a router would be ideal in my opinion. The priority of a given encoding process could be determined dynamically based on a number of factors, such as encoding framerate vs. display framerate, current buffer size vs. target buffer size, etc.; all relative to other threads. I suppose cayars suggestion of modifying the thread priority/affinity might be another, potentially much easier, way to address this (although with less granularity). Either way, I think it would be ideal if this aspect was addressed as well.

 

Basically, I'm trying to figure out if I should continue looking into modifying ffmpeg or if we have enough other (easier) options to pursue for the time being.

[jluce50 118]

Ironically, I stumbled upon this: http://mediabrowser.tv/community/index.php?/blog/1/entry-104-media-browser-305366-released/

Which says you had already implemented throttling.

 

What happened?  Which implementation did you try with that?

 

Carlo

That was the API method I mentioned in my second post:

 

Plex uses a custom build of ffmpeg to achieve control over the speed of transcoding. When MB attempted to implement this, rather than bother with ffmpeg, they passed an API endpoint to ffmpeg and used that to control how fast data was fed in. This worked fine on most machines, but struggled on lower powered ones. There were some other less severe drawbacks as well. 

]

[Luke 37056]

If suspending the process works well, then it is possibly a high value, big bang for the buck change. Do I think it would be better to go down into ffmpeg source code and add the ability to change encoding speed on the fly? Yes, but as the developer of this project I have think about every aspect of it, including level of effort to get into our build process, whether all the package maintainers would be able to do it, etc. 

 

I like suspending the process better than separate processes because I think seeking is going to wind up being an inexact science that will be prone to problems with different video files. Byte based seeking could help, but we don't have that level of information up front about the files.

 

As far as changing process priority, yea sure but I don't think it will have as much of an impact as a true pause or throttle.

[jluce50 118]

If suspending the process works well, then it is possibly a high value, big bang for the buck change. Do I think it would be better to go down into ffmpeg source code and add the ability to change encoding speed on the fly? Yes, but as the developer of this project I have think about every aspect of it, including level of effort to get into our build process, whether all the package maintainers would be able to do it, etc.

Wouldn't the ffmpeg executable be an artifact from its own package that could then be included in other packages? Why would all the package maintainers need to worry about building it with each release?

 

I like suspending the process better than separate processes because I think seeking is going to wind up being an inexact science that will be prone to problems with different video files. Byte based seeking could help, but we don't have that level of information up front about the files.

Totally agree.

 

As far as changing process priority, yea sure but I don't think it will have as much of an impact as a true pause or throttle.

No, it won't. But I see this as a compliment to pausing, rather than an alternative. In the absence of true throttling it would help to address the situation where you have competing encoding processes, which will happen even with pausing.

[Luke 37056]

Currently ffmpeg is a dependency for linux packages that can be resolved by using the system installed version. If we create our own build, that won't be possible anymore. Since the code is unmanaged it will have to be built separately for every target architecture. Probably this will lead to them each having to create it as separate, publicly available package that can then become a dependency for the MBS package. But some of the maintainers aren't even doing any building at all, they're just taking the managed mono build I publish with every release, and wrapping that into something executable. So for them the impact will be really high because build systems are going to be needed for every possible architecture.

[jluce50 118]

Ah, I see. Well damn...

[Luke 37056]

Yea now you see why I tried to find a quick win with a managed solution. Don't get me wrong, if a custom ffmpeg build is hands down the best solution, then we should do it. (And submit a pull request so that it gets added to the official release). But the idea of suspending a process is probably worth a little investigation. You could try running ffmpeg from say a console app, then see if you can pause and resume it, and see that it still completes successfully.

[jluce50 118]

I get that the separate build processes for each target architecture is unavoidable. I don't know what SCM solution you guys are using, but I would think there's a way to abstract that away from the individual package maintainers. A build of ffmpeg should only have to be done when there's a new release (of ffmpeg) and all the artifacts for the various architectures are created at that point. Then any other packages can grab the latest version from there. My point of reference is Perforce, so I'm guessing you'd know better than I how it would work for you.

 

Check this out. This guy is using SuspendThread/ResumeThread with ffmpeg. Not sure how dependent this is on the specific commands being used, but still...

https://psycodedeveloper.wordpress.com/2013/01/28/how-to-suspend-and-resume-processes-in-c/
 

I use this in my application code, while converting videos, to pause and resume ffmpeg.exe processes, where they are processes that I launched programmatically,

Edited February 26, 2015 by jluce50

[Luke 37056]

If suspension works then we're golden just with that. Can you play with it in a console app?

 

My current throttling solution already keeps track of the following:

• Current encoding position
• Current download position

It then compares those values to decide when to throttle and when to run at full blast. So for example, throttle while the download is say, 120 seconds ahead of the transcoder. (just an example). So we already have that. We just need to be able to pause and resume.

[jluce50 118]

I'll see if I can test that tomorrow. I don't even think a console app is necessary. I should be able to use Process Explorer to suspend ffmpeg.exe.

[jluce50 118]

Okay, I couldn't wait. Using the web player I set the bitrate low enough to ensure than the file would transcode. I pushed play, let it go for a minute, and then suspended ffmpeg.exe using Process Explorer. When the playback exhausted the buffer it just froze. I let it sit there for a minute and then resumed the ffmpeg process. Playback resumed immediately like nothing had happened.

 

Not the most thorough test, but it's a very good sign. I can't imagine SuspendThread would be any different...

Edited February 26, 2015 by jluce50