Subtitles come in too EARLY when resuming video player?

[Protected 86]

I spent some time looking into this and so far I suspect this problem is specific to the web application's video player. It's a somewhat reproducible client-side problem that likely depends on the attributes and length of specific videos used for testing (and maybe available device resources?) It can be very subtle or it can be several seconds. Results are inconsistent between tests.

When resuming a video such that a position in the middle of the video is passed to the player, it seems subtitles don't always come in exactly when they should. Instead, they come in some amount of time earlier than they should. The subtitle track is not dilated - subtitles also disappear earlier than they should, it's just that the whole subtitle track seems to render too early as a whole.

I observe this problem both using a remote server and a local (no lag) server, with servers running on windows and linux.

My friend observes this problem on his (completely different) computer (he's in fact the one who pointed it out to me).

You can see this in multiple different videos with subtitles.

Differences between web browsers are inconclusive so far.

This problem doesn't seem to happen if you start playback from the beginning of the video (ignoring the available resume) and then immediately seek to the resume position using the seek bar. It only happens if you use the "resume" options in emby.

Any ideas?

[Protected 86]

I had only another hour to work on this so far. I've made the following additional observations:

- This problem happens exclusively with .ass subtitles, not with .srt .

- It's much easier to reproduce this when accessing an emby server remotely, which seems consistent with the theory that this is caused by video loading delays? In fact I can reproduce this very reliably on the remote server for certain videos. (Makes it harder to debug though!)

[Luke 42077]

 

Hi there, let's look at an example. Please attach the information requested in how to report a media playback issue. Thanks!

 

[visproduction 315]

Pro,

Try remuxing the media.  It sounds like the timeline metadata has errors.  This happens to a lot of media files.  Remuxing can be handled with ffmpeg or a 3rd party video editor.  Since both video and audio are set to copy, the remuxing fixing timeline issues on a 2 hour media takes, perhaps, 30 seconds to copy out to a new version.  You need correct timeline numbers throughout the media file for subtitles to work.  If you don't havea good timeline, then subtitles only work when you play from the start of the media. 

Another way to test this is to get a designated test media file with subtitles from an test site online source and run that through your server.  If the subtitles work for the test media, then it's your media that has the issue.  I think it's not a good use of your time to keep trying to get a media file to work, when it may be the cause of problem.  By the way, playing media back with a 3rd party player like VLC is not a test.  VLC will fix a lot of issues.  The log files can show errors, but playback doesn't prove the file is good.  It's much easier to use a test media and if that works, you know it's your media that has the issue.

Anyway, hope that makes sense.

[Protected 86]

Hello visproduction. I'm sorry if this was unclear from my earlier posts, but this happens with many (most?) videos with .ass subtitles, to multiple people and multiple browsers. It's not practical or resource effective to preprocess every video compared to having a functioning client side subtitle rendering solution. I did peek at the subtitle files and found nothing obviously wrong; regardless, I have to side with VLC here - I'd like to have a video player that's capable of handling the subtitles that are effectively used in thousands of videos out there, even if they're "wrong"!

Could you please directly link or attach a suggested test video source (mkv with ass subtitles ideally)? I like the idea of using a standardized video test file but I'm not sure what I should use. I googled it, but all I got were people asking for help with various issues on different websites.

Luke: That section seemed to refer to the server (it has a direct link to the server forum). I didn't include some of that information because I thought this was a different kind of issue, web client specific (still do!) If visproduction provides a test video I could try a controlled test session to produce a server log for you and attach it here (it's the only relevant missing information after this post). I figure some kind of client log would be more useful, though. I made extensive use of browser console logs while debugging the issue.

I did fix the problem to the best of my ability after putting some more time into it today (for a certain definition of fix - modifying the mangled javascript we have access to can be difficult, and I think this may have broken certain types of playback, but it works as a proof of concept).

I noticed after careful examination that seeking from the UI functions differently from seeking on playback start: The BaseHtmlPlayer uses <video>.currentTime directly, however PlaybackManager.seek uses (for this player) a function called changeStream to set the stream with the new position in ticks. My hypothesis was that SubtitlesOctopus internally fails to account for some kind of potential delay when the video currentTime is changed directly (this might have to do with the decoding process or not, I have no idea).

Since seek works fine when you don't resume, I figured the solution would be to always start at 0 ticks and seek to the resume position, even if it has the potential to cause some extra real time transcoding.

In playbackmanager.js, playAfterBitrateDetect (this is the bad part):

... startPosition = 0, ..., (streamInfo = createStreamInfo(...), streamInfo.seekAfterPlaybackStart = playOptions.startPositionTicks ...

In htmlvideoplayer/plugin.js :

function onPlaying(e) {
    if (!self._started) {
        (async () => {
            self._started = true;
            if (self._currentPlayOptions.seekAfterPlaybackStart) {
                await _playbackmanager.default.seek(self._currentPlayOptions.seekAfterPlaybackStart);
            }
            if (null != audioTrackIndexToSetOnPlaying) setInitialAudioTrack();
            if (null != subtitleTrackIndexToSetOnPlaying) setInitialSubtitleTrack();
        })();
    }
    
    _events.default.trigger(self, "playing");
}

This seems to solve the problems, resulting in perfectly synced subtitles in my local tests so far. I'll have another session with multiple users on saturday and any lingering issues are sure to pop up at that time.

Edited October 3, 2024 by Protected
typo

[visproduction 315]

Prot,

Nice to see a code option.  I am just cautious that this is not trying to fix bad metadata in a video file.  That's why I suggested use a demo video to rule that out.  If you are dealing with bad metadata, then some code might work on one file and not on the next one.  I hear that you don't like fixing video files.  OK.  It takes me about 30 seconds per 2 hour media to remux the file.  If you tweak a command line in ffmpeg correctly, it can do, say 120 movies an hour when you click Enter.  It's not too tough to do.  Just wanted to mention that.

VLC playback: check that the timeline looks correct and does not end at something like 42 minutes for a 2 hour media or have ending time of 0:00. Either situation means you have invalid timeline metadata. Playing back with VLC is not a good test for problem metadata in the file.  VLC will ignore issue on the fly that browsers or TV will fail on playback.

Repair video timeline using a full copy of ffmpeg:

ffmpeg -err_detect ignore_err -i video.mkv -c copy video_fixed.mkv

It is difficult to find embedding subtitles in demo videos.  There are a lot of different videos here: 
https://www.demolandia.net/
https://getsamplefiles.com/sample-video-files/mkv

The only subtitle creation software, seems to make separate .ass files and not embed them.  I use https://aegisub.org/
This is a nice software program, but has a learning curve to use.
 

Hope that helps.
 

[Protected 86]

Thank you for the clarifications! I have an idea for the test video, and I hope to be able to work on it after the weekend. I'll update here.

[Protected 86]

Here's a test video that can be used to experience the subtitle offset on resume issue. It's a mkv containing a trim of A Conversation With Death by Khemmis and soft karaoke subtitles in .ASS format. Custom fonts and weird symbols have been removed (I'm only using Arial). As Khemmis have been kind enough not to copyright claim on youtube, and this is only a partial video (it's only this long to ensure it's resumable on default settings), I'm hoping it's fine to distribute here (but check out the band if you're interested in their music!)

Karaoke subtitles on a slow song make it easy to intuitively check the synchronization between the subtitles and audio.

At time of writing I still have the vanilla installation I mentioned in another thread, meaning this should be an ideal test environment. I'm on the Windows version, web client and Firefox. I can clearly see a significant offset when I resume the video, with the subtitles coming in ahead. Although, even when not resuming, I see in this test video that the subtitles come in slightly ahead, when compared to a native video player.

Note that I'm experiencing a second issue for the first time here; the color wipes (which are a feature of .ASS) seem to overwhelm the subtitle renderer on playback, causing stutters. If you experience this, you can correct it by pausing and unpausing the video (you can seek back to the beginning to observe the beginning without stutters, but it doesn't seem to affect the timing offset between the subtitles and audio, so it's not a significant hindrance). Maybe the default subtitles octopus framerate of 24fps is too optimistic, at least on my setup?

In the attached server log, I started the server, played the test video from the beginning, after a few seconds paused and resumed, and after a few more seconds left the video player. Then I resumed the video, noting a significant offset between subtitles and audio; after a few seconds paused and resumed, and a few more seconds later I left the video player.

testvideo.mkv embyserver.txt

[visproduction 315]

The timeline on the testvideo is corrupt.  This video is a H.264 video encapsulated inside .mkv wrapper to allow the subtitle.  The video will close if you move the play control to 2:11.  The timeline states 2:30.  When I remux the file it becomes 2:10 long.  You need to use a demo file that does not have timeline issues.

[Protected 86]

It doesn't really matter, but I guess that was because when I trimmed the video with -t the final displayed subtitle was cut before its ending timestamp, which was at around 2:30. The player (or emby's server side ffmpeg-based analysis I guess) must have used the longest stream to determine length but can't render beyond 2:10 because there's no video for it. Here's a version with the ending timestamps of final subtitles adjusted in the .ass so they are positioned before the end of the video stream. The issue remains unchanged.

testvideo2.mkv

[visproduction 315]

This new #2 video plays back correctly using VLC and jumping around to check the subtitle sync works fine.  You do have to turn on the subtitle.  I don't have a way to check it with a browser.  Perhaps having the timecode works better.  Pro, see if your transcoding test looks better.

Perhaps part of these issues with out of sync subs, is that there are a lot of videos with metadata errors everywhere you look.  I just remux or reencode everything and use separate .srt or .ass sub files and that gives me media that plays correctly in sync all the time.  All the professional online streaming media sites are reencoding to fix all these issues.  I don't think any site online, just copies and tries to play media.  Social media sites I've worked on always transcode anything coming in.

Hope that works now.  

[Protected 86]

I think Luke understood the problem, which is that the subtitles come in too early in the web application's video player when you resume from a position beyond the beginning of the video. Seeking or pausing/unpausing after resuming/playback does not affect the fact that subtitles remain slightly off when starting from the resume action compared to starting from position zero. The subtitles are rendered by the browser on a canvas element using subtitle octopus. That's what we're testing here. If you can't test on the web client you have no visibility over the problem.

[Luke 42077]

What about a resume that direct plays? How does that compare for you?

[Luke 42077]

@Protected?

[Protected 86]

Hello. I haven't actually had the time to set up an instance for testing this yet (it should be an unmodified instance and one wasn't set up when you asked). I should do that eventually!

[Luke 42077]

That would be great. Thanks.