Jump to content


Photo

Scale subtitle stream to target dimension instead of the original dimension

transcoding subtitle 4K

  • Please log in to reply
34 replies to this topic

#21 Luke OFFLINE  

Luke

    System Architect

  • Administrators
  • 138075 posts
  • Local time: 07:19 AM

Posted 20 July 2018 - 12:26 AM

can you give an example of how force_original_aspect_ratio would be added to your scale2ref commands, and why not just always use it?



#22 Waldonnis OFFLINE  

Waldonnis

    Advanced Member

  • Members
  • 652 posts
  • Local time: 07:19 AM

Posted 20 July 2018 - 01:00 AM

can you give an example of how force_original_aspect_ratio would be added to your scale2ref commands, and why not just always use it?

 

Sure, but it'll have to wait until tomorrow.  I try to post copy-and-pasted command lines I've actually used/tested to avoid typos, so I want to do that first...but I need to start an HEVC encode for the night in a few minutes and need to start closing everything running on the machine now.

 

As for why not always using it, do you mean force_original_aspect_ratio or the scale2ref?  If you mean scale2ref, sure, you could use it since it wouldn't do jack if no scaling was required.  My tendency is to avoid including things in command lines that aren't useful, hence why I wouldn't personally include a scale-type filter that does nothing (programmer in me always says don't burn cycles on stuff that does nothing)....but it's not like it's a "costly" operation in the grand scheme so there's no harm in doing so that I can think of.

 

If instead you mean if you could use force_original_aspect_ratio all of the time...you'd probably want to, but the value may change if you need to stretch the sub frame for some reason rather than keeping it's normal AR (e.g. in case subs are stretched to match an anamorphic DAR width which is likely wider than the sub's native dimensions/AR).  If it turns out that subs on anamorphic videos are normally just rendered without any "stretching", then the force_original_aspect_ratio would never need to change since we'd always want to maintain the sub frame's AR when scaling.



#23 Luke OFFLINE  

Luke

    System Architect

  • Administrators
  • 138075 posts
  • Local time: 07:19 AM

Posted 20 July 2018 - 01:07 AM

Thanks that would be great, and yea I was asking about force_original_aspect_ratio. Thanks.



#24 Waldonnis OFFLINE  

Waldonnis

    Advanced Member

  • Members
  • 652 posts
  • Local time: 07:19 AM

Posted 20 July 2018 - 08:32 PM

Bah, force_original_aspect_ratio apparently doesn't have the effect on scale2ref that I expected, so I had to preserve the sub frame's AR the hard way.  Note: this is really rough still and the expressions probably need to be rounded or truncated to avoid floating point errors, but it seems to work no matter what I've thrown at it so far:

ffmpeg -canvas_size 1920:1080 -i input.mkv -filter_complex "null[video];[0:2][video]scale2ref=ih*mdar:ih[sub][ref];[ref][sub]overlay=(W-w)/2:(H-h)/2" -c:v libx264 -c:a copy output.mkv

A little explanation:

  • null[video]
    This is just creating a "passthrough" pad so I can create a named pad for the video input.  If you're going to scale the video for any reason or do some other operation on the video, this can be replaced by that operation as long as you keep the output pad name the same as the input pad name being fed to scale2ref.  Really, the names don't matter as long as you can understand which one goes to which (you could use something like [foo] if you really wanted, but it may be too vague and make it harder to figure out what it is if you have to fix something with it in the future).
  • [0:2][video]scale2ref=ih*mdar:ih[sub][ref]
    Takes the subtitle ([0:2]) and reference video ([video]), then scales the subtitle frame while keeping the sub frame's aspect ratio.  Width is determined by multiplying the height of the video by the DAR of the subtitle frame, and height is the height of the video.  The filter then outputs the scaled sub frames and video frames to [sub] and [video] respectively.
  • [ref][sub]overlay=(W-w)/2:(H-h)/2[v]
    Pretty standard overlay operation, but since the sub frame may not be the same AR as the video, we centre the sub frame on the video frame.

If you try it and get a ton of "Changing frame properties on the fly..." warnings, make sure canvas_size is set to the sub stream's frame dimensions and those should go away.  I'll run some tests in a bit to see if truncation is needed on the expressions, but the above should be enough for local testing in the meantime to see if I missed something.  Oh, and if you have problems with external IDXs again, post an ffmpeg report and I'll look at it (don't have any IDX's laying around and haven't converted/copied one yet).

 

There is one case I still need to test: 4:3 videos that are cropped to remove pillarbox padding.  I doubt the current scale2ref expression would work out if the original subs had text that was rendered into the padding area.  Should be easy enough to fix, but I haven't done the math on it yet and need to dig up a sample to verify.



#25 Luke OFFLINE  

Luke

    System Architect

  • Administrators
  • 138075 posts
  • Local time: 07:19 AM

Posted 20 July 2018 - 10:35 PM

Great, thanks !



#26 Waldonnis OFFLINE  

Waldonnis

    Advanced Member

  • Members
  • 652 posts
  • Local time: 07:19 AM

Posted 21 July 2018 - 02:32 AM

Still working on it.  I have it working perfectly now, but think the scale2ref expression is ridiculously complicated and can be simplified.  I'll finish it up tomorrow.



#27 Waldonnis OFFLINE  

Waldonnis

    Advanced Member

  • Members
  • 652 posts
  • Local time: 07:19 AM

Posted 21 July 2018 - 03:34 PM

Well, I do have it working, but after loading up a Japanese movie that splashed rendered captions all over the screen, it just didn't look good and I had a hard time figuring out a good way to keep positional elements in place while also keeping the sub frame AR.  Long story short, I concluded it's probably just easier to "stretch to fit" the sub frame to the video (which is super-easy).  Yeah, it'll probably squish or stretch captions a bit, but barring drastic AR changes, it would be the same behaviour that Emby seems to do now and at least it would keep text that was intentionally positioned to be directly above some character's head in a scene exactly where it's supposed to be.

 

So, we're back to simple with no real explanation needed other than the sub frame scaling happens after any desired video scaling:

ffmpeg -canvas_size 1920:1080 -i input.mkv -filter_complex "null[v];[0:2][v]scale2ref[sub][ref];[ref][sub]overlay" -c:v libx264 -c:a copy output.mkv

scale2ref by default can mangle the AR of the sub frame if it's different than the video's AR (think of it like a 16x9 movie on a 4x3 television set up to "stretch to fill the screen"; everyone looks taller/thinner).  As mentioned above, though, it's pretty much the best way to go IMO so positional intent is maintained and the distortion is relatively minor in all but the most extreme cases that are too rare to care about (e.g. 4:3 sub frame on a 36:10 video).  I also changed the [video] pad naming to [v] because I was tired of typing [video] (lol) and because I didn't have a reason to explicitly name the output pad for overlay any longer now that I don't have any debugging/info filters in the filtergraph (I never post those bits; they just confuse things, but leave the pad name in place until I'm done).

 

Oh, and it's safe to use scale2ref whenever you overlay subs if you want, even if the video isn't scaled or the sub/video frame sizes are equal.  If the frame sizes match, it's just a 1:1 scale anyway.  It's probably wasteful to do so from a CPU cycles perspective, but it won't cause output issues.

 

Here's a really quick benchmark comparison for a 2160p video being downscaled to 1080p and overlaying subtitles for 30s using the original arrangement (as noted in the OP) vs. the above on my admittedly-busy machine (using a hardware encoder, but software decoding...and outputting to null)...

 

Original:

frame=  720 fps= 20 q=20.0 Lsize=N/A time=00:00:30.03 bitrate=N/A speed=0.823x
video:7620kB audio:0kB subtitle:0kB other streams:0kB global headers:0kB muxing overhead: unknown
bench: utime=111.781s stime=5.891s rtime=36.528s
bench: maxrss=779288kB 

New:

frame=  720 fps= 30 q=20.0 Lsize=N/A time=00:00:30.03 bitrate=N/A speed=1.26x
video:7598kB audio:0kB subtitle:0kB other streams:0kB global headers:0kB muxing overhead: unknown
bench: utime=74.219s stime=1.391s rtime=23.880s
bench: maxrss=662488kB

Edited by Waldonnis, 21 July 2018 - 03:36 PM.


#28 Luke OFFLINE  

Luke

    System Architect

  • Administrators
  • 138075 posts
  • Local time: 07:19 AM

Posted 21 July 2018 - 04:11 PM

Can you try that when the subtitles are external sub/idx? thanks.



#29 Waldonnis OFFLINE  

Waldonnis

    Advanced Member

  • Members
  • 652 posts
  • Local time: 07:19 AM

Posted 21 July 2018 - 05:03 PM

Yep, works fine.  Same command line, except for the stream IDs obviously (in this case, the IDX is [1:0] since i specified it as the second input file, and the input of the first filter would be [0:0] for the movie file's video track stream ID).



#30 Luke OFFLINE  

Luke

    System Architect

  • Administrators
  • 138075 posts
  • Local time: 07:19 AM

Posted 21 July 2018 - 06:33 PM

I'll play with it some more but when i tested this the other day the transcode was successful but i just couldn't see the subtitles.



#31 Pittiplatsch OFFLINE  

Pittiplatsch

    Advanced Member

  • Members
  • 38 posts
  • Local time: 01:19 PM

Posted 28 September 2018 - 09:09 AM

Hi,

 

I'd like to know what the status of the issue is? I've a problem with misaligned/scaled subtitles (vobsub) with some of my media and it seems to be very related to the issue described in this thread.

 

i figured out, that the issue only come up when I crop letterboxed media to get the actual or intended aspect ratio in my media files while rendering. I also tested if anamorphic rendering has any influence to it, but the issue seems to be not affected by this, only the cropping.

 

I'm pretty sure this issue come up with a Emby update a while ago, but I cannot reproduce this anymore.

 

Thanks,

Peter



#32 Luke OFFLINE  

Luke

    System Architect

  • Administrators
  • 138075 posts
  • Local time: 07:19 AM

Posted 28 September 2018 - 01:56 PM

Hi @Pittiplatsch, this topic is not about fixing any alignment issues, this topic is about finding optimization in our subtitle transcoding command line. Let's look at your issue from scratch. please see how to report a media playback issue. thanks !



#33 cayars OFFLINE  

cayars

    Advanced Member

  • Alpha Testers
  • 2945 posts
  • Local time: 07:19 AM

Posted 29 September 2018 - 12:39 AM

Sounds like this might be related based on the only media he sees this issue on.

#34 Pittiplatsch OFFLINE  

Pittiplatsch

    Advanced Member

  • Members
  • 38 posts
  • Local time: 01:19 PM

Posted 30 September 2018 - 06:30 AM

Hi @Pittiplatsch, this topic is not about fixing any alignment issues, this topic is about finding optimization in our subtitle transcoding command line. Let's look at your issue from scratch. please see how to report a media playback issue. thanks !

 

Hi @Luke. Sorry for the misunderstanding. The title and some discussion of this thread suggested that this might be related to my issue. Anyway, I created a new post: https://emby.media/c...bsub-subtitles/

 

Thanks,

Peter



#35 Luke OFFLINE  

Luke

    System Architect

  • Administrators
  • 138075 posts
  • Local time: 07:19 AM

Posted 30 September 2018 - 11:17 AM

Thanks.







Also tagged with one or more of these keywords: transcoding, subtitle, 4K

0 user(s) are reading this topic

0 members, 0 guests, 0 anonymous users