Parallelize Bulk Conversion Jobs (Season, Show, etc.)

[awkimball 3]

Hi all,

Recently, in order to more efficiently use my storage space I have been enjoying using emby's conversion feature to convert and replace less frequently watched content with lower bitrate hevc/mkv files. Although I have the conversion thread count and performance settings set to max, one conversion job running at a time barely uses 15% of my R730's 48 CPU cores.

 

Would it be possible to add a setting in the conversion menu called "Maximum parallel conversion jobs allowed" that defaults to 1 (one)? This would maintain the existing behavior but would allow us with many spare CPU cores to have more than 1 conversion job running and take advantage of our hardware.

 

I've attached a screenshot of an example of a batch conversion job for a season of television that I feel would benefit from this added feature.

 

Thanks for your consideration

 

[rbjtech 4265]

This feature is meant for occasional conversion - if you are serious about converting an entire library or large parts of it - then may I suggest you use a tool that is designed for that purpose .. ie

Tdarr

 

Edited January 4, 2023 by rbjtech

[Spaceboy 2493]

yeah tdarr is great. i don't use it to convert but all my media get processed by it before it gets added to emby

[awkimball 3]

I see your point but based on the size of my library, I would personally consider my use-case to be "occasional conversion". I have no interest in converting an entire library, just specific seasons of shows, periodically.

The fact that emby already includes the feature to batch a whole show, a season of a show, or manually select multiple shows/movies for conversion would indicate to me that my use-case is not too far off from emby's intended purpose. Adding the ability to run those conversion jobs in parallel would only make an existing feature more efficient.

 

As for Tdarr, while it does seem pretty awesome, I explicitly *don't* want to convert most media that I import to emby. It would seem somewhat overkill to set up a complete transcoding server in order to pick and choose a few seasons or shows to convert, but maybe that's just me.

*EDIT* Spelling

Edited January 4, 2023 by awkimball

[Luke 37065]

5 hours ago, rbjtech said:

This feature is meant for occasional conversion - if you are serious about converting an entire library or large parts of it - then may I suggest you use a tool that is designed for that purpose .. ie

Tdarr

 

The conversion feature is fully capable of converting your entire library.

[rbjtech 4265]

9 hours ago, Luke said:

The conversion feature is fully capable of converting your entire library.

Not using all available resources it's not as per the OP's query - the conversions feature in emby is single pipeline only.

[rbjtech 4265]

15 hours ago, awkimball said:

I see your point but based on the size of my library, I would personally consider my use-case to be "occasional conversion". I have no interest in converting an entire library, just specific seasons of shows, periodically.

The fact that emby already includes the feature to batch a whole show, a season of a show, or manually select multiple shows/movies for conversion would indicate to me that my use-case is not too far off from emby's intended purpose. Adding the ability to run those conversion jobs in parallel would only make an existing feature more efficient.

 

As for Tdarr, while it does seem pretty awesome, I explicitly *don't* want to convert most media that I import to emby. It would seem somewhat overkill to set up a complete transcoding server in order to pick and choose a few seasons or shows to convert, but maybe that's just me.

*EDIT* Spelling

A fair response.

The issue primarily is that by opening up parallel workloads - emby quickly open themselves up to support calls on 'broken' systems when users set unrealistic parameters for their systems without realising where the bottlenecks really are.

In your example above -  'cpu' probably isn't the bottleneck for lets say 4 parallel encodes - but could the I/O keep up as an example.  What if you were playing a movie at the time - would the heavy I/O interrupt the playback ?

btw - I'm not against the request at all - I'm all for it - but I can see why it needs to be restricted.

Edited January 5, 2023 by rbjtech

[awkimball 3]

6 hours ago, rbjtech said:

In your example above -  'cpu' probably isn't the bottleneck for lets say 4 parallel encodes - but could the I/O keep up as an example.  What if you were playing a movie at the time - would the heavy I/O interrupt the playback ?

This is definitely a good point. I understand Emby's mission to be user friendly and not introduce features that could easily mess up a less-advanced user's experience.

Speaking selfishly though, I would certainly love a checkbox buried in the Conversion settings menu titled "Allow parallel conversion jobs" that defaults to unchecked, with a dropdown number picker titled "Number of allowed parallel conversion jobs:" that defaults to 1.

Shown here is the menu I am picturing it as a part of:

 

22 hours ago, rbjtech said:

then may I suggest you use a tool that is designed for that purpose .. ie

Tdarr

Btw, I am thinking about trying out Tdarr anyways. The more I think about it, it seems like it might be pretty useful for me. Like you said above though, I'd be curious to see how quickly I hit a disk I/O bottleneck with parallel conversion jobs. Thanks for the suggestion

[ebr 14913]

Hi.  This probably warrants some testing at some point but my suspicion is you will become I/O bound very quickly and then render the server useless for its primary function (serving content).

@softworkz may have some thoughts.

[Qowy 0]

I was redirected here from this topic.

My use case would be Download conversions, and as far as I can see, there is no way to do that easily with another program since emby they are only temporary created by emby anyway.

I have a dedicated p400 in my server to do this, but only 20% of the encoder is used.

 

Interestingly starting a manual FFMPEG conversion with nvenc for testing purposes results in much higher load and speed. So I wonder what the holdup is?
Maybe some scaling filter.

 

I/O could be an issue, but so far I am on 5% HDD usage on a software RAID.

I think some manual value of parallelism would work best with a default of 1 so that we can override it when needed.

 

Anyway my question seems to be answered, as it is intended behavior to only convert one file at a time.