ffmpeg remains post-playback, pegging a core at 100%

[Walter_Ego 1]

hi team,

i spent some time looking into why my emby server sometimes leaves ffmpeg processes pegging 100% of a core post-playback, and i figured out why and how to fix it (it's already fixed upstream).

it seems that this occurs when ffmpeg seeks beyond the start of the last cue of the subtitle track.  if a video is 1 minute long, but the subtitles end 30 seconds in, and you seek to 31 seconds, this will cause the process to remain, and spin its wheels.  in fact, it occurs on any filtered subtitle output that receives zero frames; i reproduced it with a sparse forced track and a seek to a point in the first half of a video.

looking at https://mediabrowser.github.io/embytools/ffmpeg-2023_06_25-u1.tar.gz i can see that this is due to the emby patches on top of ffmpeg.  in fftools/ffmpeg_filter.c:758 is this comment:

    if (ist->st->codecpar->codec_type == AVMEDIA_TYPE_SUBTITLE && ist->dec_ctx) {
        // For subtitles, we need to set the format here. Would we leave the format
        // at -1, it would delay processing (ifilter_has_all_input_formats()) until
        // the first subtile frame arrives, which could never happen in the worst case
        ifilter->format = (uint16_t)ist->dec_ctx->subtitle_type;
    }

so emby presets the format so a subtitle graph configures with zero frames, which breaks the invariant that early return was written against - af1761f7b5 - "init filtergraphs only after we have a frame on each input".  this means choose_output() can select this output stream that is already finished as is seen in fftools/ffmpeg.c:3855:

static OutputStream *choose_output(void)
{
    for (i = 0; i < nb_output_streams; i++) {
        OutputStream *ost = output_streams[i];
...
        if (!ost->initialized && !ost->inputs_done)
            return ost->unavailable ? NULL : ost;

        if (!ost->finished && opts < opts_min) {
            opts_min = opts;
            ost_min  = ost->unavailable ? NULL : ost;
        }
    }
    return ost_min;
}

inputs_done is never set for a configured graph, and a subtitle sink with zero frames never sets initialized either.  both flags stay 0 permanently so choose_output()'s early return fires forever, resulting in a livelock.  emby tries to send "q" to ffmpeg's stdin to gracefully shut it down, but it never acts on it due to the livelock, so the process just spins its wheels forever.

the fix is to extend the test for the return: if (!ost->initialized && !ost->inputs_done && !ost->finished)

this is probably best done by cherry picking e1d12aaa45 onto the 5.1-emby source

you could also harden this in reap_filters() in fftools/ffmpeg.c:1522:

                } else if (flush && ret == AVERROR_EOF) {
                    if (av_buffersink_get_type(filter) == AVMEDIA_TYPE_VIDEO)
                        do_video_out(of, ost, NULL);
                    else
                        init_output_stream_wrapper(ost, NULL, 1);
                }

that's pretty much the whole story.  i see that there are other posts likely about this as well.

also, i can't seem to find the p6 version source for ffmpeg, could you please point me towards it?