SubtitleCreator - Application to create subtitles.

[BillOatman 565]

SubtitleCreator is an open source NET8 console application to create subtitles either automatically after recording or manually.

It has versions that run under Windows as well as Linux, RasPi, and OSX ... though I have only ever tried Windows.

It can create a .SRT file, and optionally merge the video file and the SRT file into a MKV container. It can also optionally translate Spanish or French audio into English subtitles.

It uses Whisper.net and ffmpeg to make the magic happen.

set drive=%~d1
set folder=%~p1
set filename=%~n1
c:
cd\SubtitleCreator
SubtitleCreator -ffmpegpath="C:\ffmpeg\bin" -infile="%drive%%folder%%filename%.ts" -model=medium -nomerge

I have mine setup to run comskip first, then SubtitleCreator.

The link to the binaries and source are here.

Note that based on my testing, using the medium model does a great job. However, depending
upon the strength of your machine running this app, it can take quite a while to generate the
subtitles. My box is several years old and by no means a supercomputer but not a boat anchor
either, a recording that was 66 minutes in length took 60 minutes to generate the SRT file.

SubtitleCreator.pdf

Edited September 30, 2024 by BillOatman

[rbjtech 4888]

Thanks Bill.

Does it only work on TS files - as I'm getting an error if I try and pass it a MKV using EAC3 Audio as input for example ?

From the log -

Unrecoverable exception extracting audio from the video file. The Media format is recognized but is invalid. (0xC00D3E8C)

 

 

[BillOatman 565]

On 10/4/2024 at 8:42 AM, rbjtech said:

Thanks Bill.

Does it only work on TS files - as I'm getting an error if I try and pass it a MKV using EAC3 Audio as input for example ?

From the log -

Unrecoverable exception extracting audio from the video file. The Media format is recognized but is invalid. (0xC00D3E8C)

Right now it wont work with a container.  I'll look into adding that capability.

Edit:  I took a quick look and that shouldn't be a problem.  I should have it in a day or two.

 

 

Edited October 6, 2024 by BillOatman

[rbjtech 4888]

Thanks Bill - that would be perfect for me trying to add SRT's for old movies etc that never had them.

[BillOatman 565]

Updated to work with MKV input files.  Source at same link.

Direct Link to Binaries here.

[rbjtech 4888]

Thanks Bill - I'm a step closer but it's now reporting I need the Whisper library ?   Do I need to install that first ?

I do not have C:\SubtitleCreator\runtimes\win-x64\whisper.dll - but I do have C:\SubtitleCreator\ggml-medium.bin

output below -

I'm going to try and install the DLL and report back ..

edit

Yes - I copied the runtime DLL from https://github.com/GewoonJaap/WinWhisper/releases/download/v1.4.0/WinWhisper-1.4.0-standalone.zip to the location specified in the error below - and it's now working.. 

C:\SubtitleCreator>SubtitleCreator.exe -ffmpegPath="c:\ffmpeg\bin" -inFile="M:\Films\Wolfs (2024) [tmdbId=877817]\Wolfs
(2024) - WEBDL-1080p.mkv" -nomerge -language=en -noRepair
SubtitleCreator version 1.1.0.0

ffmpeg Path: c:\ffmpeg\bin
Input File: M:\Films\Wolfs (2024) [tmdbId=877817]\Wolfs (2024) - WEBDL-1080p.mkv
Audio Language: eng
Translate to English: False
Attempt to Repair: False
SRT File: M:\Films\Wolfs (2024) [tmdbId=877817]\Wolfs (2024) - WEBDL-1080p.en.srt

Extracting audio from the video file ...
Input file is in a MKV container.  Extracting the video.
The video file has been extracted.  Now let's get back to extracting its audio.
Audio extraction complete in 00:02:42.55.
Creating subtitles file. Please be patient ...
Unhandled exception. System.IO.FileNotFoundException: Native Library not found in path C:\SubtitleCreator\runtimes\win-x64\whisper.dll. Verify you have have included the native Whisper library in your application, or install the default libraries with the Whisper.net.Runtime NuGet.
   at Whisper.net.LibraryLoader.NativeLibraryLoader.LoadNativeLibrary(String path, Boolean bypassLoading)
   at Whisper.net.WhisperFactory.<>c.<.cctor>b__11_0()
   at System.Lazy`1.ViaFactory(LazyThreadSafetyMode mode)
   at System.Lazy`1.ExecutionAndPublication(LazyHelper executionAndPublication, Boolean useDefaultConstructor)
   at System.Lazy`1.CreateValue()
   at Whisper.net.WhisperFactory..ctor(IWhisperProcessorModelLoader loader, Boolean delayInit, String libraryPath, Boolean bypassLoading)
   at SubtitleCreator.CreateTheSubtitles.SetupProcessor(String modelPath, String languageCode, Boolean shouldTranslate, String audioLanguage, OnSegmentEventHandler OnNewSegment)
   at SubtitleCreator.CreateTheSubtitles.DoWorkGenerateSubtitles(String wavFilePath, ModelType modelType, String workingDir, String srtFile, String languageCode, Boolean shouldTranslate, String audioLanguage)
   at SubtitleCreator.SubtitleCreator.RealMain(String[] args)
   at SubtitleCreator.SubtitleCreator.Main(String[] args)

 

Edited October 8, 2024 by rbjtech

[BillOatman 565]

3 hours ago, rbjtech said:

Thanks Bill - I'm a step closer but it's now reporting I need the Whisper library ?   Do I need to install that first ?

I do not have C:\SubtitleCreator\runtimes\win-x64\whisper.dll - but I do have C:\SubtitleCreator\ggml-medium.bin

output below -

I'm going to try and install the DLL and report back ..

edit

Yes - I copied the runtime DLL from https://github.com/GewoonJaap/WinWhisper/releases/download/v1.4.0/WinWhisper-1.4.0-standalone.zip to the location specified in the error below - and it's now working.. 

C:\SubtitleCreator>SubtitleCreator.exe -ffmpegPath="c:\ffmpeg\bin" -inFile="M:\Films\Wolfs (2024) [tmdbId=877817]\Wolfs
(2024) - WEBDL-1080p.mkv" -nomerge -language=en -noRepair
SubtitleCreator version 1.1.0.0

ffmpeg Path: c:\ffmpeg\bin
Input File: M:\Films\Wolfs (2024) [tmdbId=877817]\Wolfs (2024) - WEBDL-1080p.mkv
Audio Language: eng
Translate to English: False
Attempt to Repair: False
SRT File: M:\Films\Wolfs (2024) [tmdbId=877817]\Wolfs (2024) - WEBDL-1080p.en.srt

Extracting audio from the video file ...
Input file is in a MKV container.  Extracting the video.
The video file has been extracted.  Now let's get back to extracting its audio.
Audio extraction complete in 00:02:42.55.
Creating subtitles file. Please be patient ...
Unhandled exception. System.IO.FileNotFoundException: Native Library not found in path C:\SubtitleCreator\runtimes\win-x64\whisper.dll. Verify you have have included the native Whisper library in your application, or install the default libraries with the Whisper.net.Runtime NuGet.
   at Whisper.net.LibraryLoader.NativeLibraryLoader.LoadNativeLibrary(String path, Boolean bypassLoading)
   at Whisper.net.WhisperFactory.<>c.<.cctor>b__11_0()
   at System.Lazy`1.ViaFactory(LazyThreadSafetyMode mode)
   at System.Lazy`1.ExecutionAndPublication(LazyHelper executionAndPublication, Boolean useDefaultConstructor)
   at System.Lazy`1.CreateValue()
   at Whisper.net.WhisperFactory..ctor(IWhisperProcessorModelLoader loader, Boolean delayInit, String libraryPath, Boolean bypassLoading)
   at SubtitleCreator.CreateTheSubtitles.SetupProcessor(String modelPath, String languageCode, Boolean shouldTranslate, String audioLanguage, OnSegmentEventHandler OnNewSegment)
   at SubtitleCreator.CreateTheSubtitles.DoWorkGenerateSubtitles(String wavFilePath, ModelType modelType, String workingDir, String srtFile, String languageCode, Boolean shouldTranslate, String audioLanguage)
   at SubtitleCreator.SubtitleCreator.RealMain(String[] args)
   at SubtitleCreator.SubtitleCreator.Main(String[] args)

 

There is a runtime download in the github release, did you put that on your system?

[rbjtech 4888]

53 minutes ago, BillOatman said:

There is a runtime download in the github release, did you put that on your system?

yes, after posting (I edited my original post) - I guess you maybe need to say that on the read me as people are not going to have that.

btw - it all worked ok - I tested it on a film 

...
Audio extraction complete in 00:02:43.73.
Creating subtitles file. Please be patient ...
Subtitle Generation Complete.
Subtitle creation complete in 01:16:06.22.
Merge process bypassed. Subtitle file M:\Films\Wolfs (2024) [tmdbId=877817]\Wolfs (2024) - WEBDL-1080p.en.srt created.

It's fairly CPU intensive (my i7 12700K was hitting 27% cpu for this entire peroid..)

I guess it's just less than realtime for a 1h 47 duration, it took 1h 16

Output appears to be SDH - I wonder if there is any switch to turn that off (even if it is very impressive!) and just have only spoken dialogue ?

770
00:50:22,740 --> 00:50:24,740
Come back here!

771
00:50:26,740 --> 00:50:28,740
(grunting)

772
00:50:30,740 --> 00:50:32,740
(tires screeching)

773
00:50:32,740 --> 00:50:34,740
(dramatic music)

774
00:50:34,740 --> 00:50:36,740
Stop!

775
00:50:36,740 --> 00:50:38,740
Run in it!

 

[BillOatman 565]

1 hour ago, rbjtech said:

yes, after posting (I edited my original post) - I guess you maybe need to say that on the read me as people are not going to have that.

btw - it all worked ok - I tested it on a film 

...
Audio extraction complete in 00:02:43.73.
Creating subtitles file. Please be patient ...
Subtitle Generation Complete.
Subtitle creation complete in 01:16:06.22.
Merge process bypassed. Subtitle file M:\Films\Wolfs (2024) [tmdbId=877817]\Wolfs (2024) - WEBDL-1080p.en.srt created.

It's fairly CPU intensive (my i7 12700K was hitting 27% cpu for this entire peroid..)

I guess it's just less than realtime for a 1h 47 duration, it took 1h 16

Output appears to be SDH - I wonder if there is any switch to turn that off (even if it is very impressive!) and just have only spoken dialogue ?

770
00:50:22,740 --> 00:50:24,740
Come back here!

771
00:50:26,740 --> 00:50:28,740
(grunting)

772
00:50:30,740 --> 00:50:32,740
(tires screeching)

773
00:50:32,740 --> 00:50:34,740
(dramatic music)

774
00:50:34,740 --> 00:50:36,740
Stop!

775
00:50:36,740 --> 00:50:38,740
Run in it!

 

I thought I put  that in the pdf but I will double check.  There is no way I know of to not have it not be descriptive. I was surprised the first time I saw it do 
770
00:50:22,740 --> 00:50:24,740
[Speaking in Spanish]

 

Edited October 8, 2024 by BillOatman

[rbjtech 4888]

You can pass through subtitle edit and remove all the SDH, so there is an easy solution if it's not possible via the command line.

[Neminem 775]

I know bazarr also uses whisper to transcribe audio to text.

There might be some pointers in there

morpheus65535/bazarr: Bazarr is a companion application to Sonarr and Radarr. It manages and downloads subtitles based on your requirements. You define your preferences by TV show or movie and Bazarr takes care of everything for you. (github.com)

  

[BillOatman 565]

1 hour ago, rbjtech said:

You can pass through subtitle edit and remove all the SDH, so there is an easy solution if it's not possible via the command line.

Hmmm there might be a way to make that an option, I'll play with it tonight and see,

I think only descriptive text has ( and ) ???

Edited October 8, 2024 by BillOatman

[BillOatman 565]

New release.  

Added -noSDH flag to stop descriptive text such as

3
00:00:05,600 --> 00:00:08,600
[audience applauding]

and

21
00:00:59,620 --> 00:01:06,660
♪ ♪

Link to binaries.

[rbjtech 4888]

Very cool !  

[BillOatman 565]

Updated to work with the latest Whisper.net.
Note that you will need to update the runtimes as well.

Link to binaries

[Oracle 84]

Any opportunities this will translate japanese audio to english subtitles?

[BillOatman 565]

On 10/19/2024 at 4:13 PM, Oracle said:

Any opportunities this will translate japanese audio to english subtitles?

I will see if that is possible and get back to you here.

[BillOatman 565]

On 10/19/2024 at 4:13 PM, Oracle said:

Any opportunities this will translate japanese audio to english subtitles?

I have it coded but my provider doesn't have any streams with Japanese audio.
Do you have a link to either a stream or a file that has that audio so I can test it?

Edit:  I found one.

Edited October 22, 2024 by BillOatman

[BillOatman 565]

@Oracle  It generates English subtitles for Japanese audio, but I have no clue how well   Give it a try.  Use flags -translate -audiolanguage=jap

Latest Release

Edit: I am in the process now of making this application work with all the languages that Whisper supports. I should be releasing that later today.
Note that the code for Japanese above will need to be jpn after that release.

Edited October 23, 2024 by BillOatman

[BillOatman 565]

Latest Release

- Updated to latest Whisper
- Support for all languages that Whisper supports translating to English.

You will need to update the runtime for this release.

Run SubtitleCreator -languagelist  to get a list of the supported languages.

Edited October 29, 2024 by BillOatman

[BillOatman 565]

Latest Release

- Updated to latest Whisper
- Progress showed in the console window

You will need to update the runtime for this release.

 

[BillOatman 565]

Latest Release

Added parameter -forceModelUpdate that will force an update of the Whisper model.

[Embodiment 15]

I stumbled upon this post after literally 5 hours research solutions to my close captioning and solutions beyond CCExtractor. This looks promising and I would beg your advance forgiveness that a someone new to some of this, I may need to bend an ear.

In the process of finding another command line solution, I did install FFmpeg into my Windows environment. It runs. I have downloaded the Windows SubtitleCreator-WIN.zip from the host server and have uncompressed it. I see the SubtitleCreator.exe.

From there I am lost exactly what to do next. I know it will run from a Dos command line interface. 

Where I lack the knowledge is where do I configure its settings so it can point to a specific network storage space so I can extract the close captions out of my mpg video files. Does a text document turned batch file come into play here?

Thank you in advance!

[BillOatman 565]

3 hours ago, Embodiment said:

I stumbled upon this post after literally 5 hours research solutions to my close captioning and solutions beyond CCExtractor. This looks promising and I would beg your advance forgiveness that a someone new to some of this, I may need to bend an ear.

In the process of finding another command line solution, I did install FFmpeg into my Windows environment. It runs. I have downloaded the Windows SubtitleCreator-WIN.zip from the host server and have uncompressed it. I see the SubtitleCreator.exe.

From there I am lost exactly what to do next. I know it will run from a Dos command line interface. 

Where I lack the knowledge is where do I configure its settings so it can point to a specific network storage space so I can extract the close captions out of my mpg video files. Does a text document turned batch file come into play here?

Thank you in advance!

You need to install the runtime folder as well.  Read the PDF it spells it out.  If not feel free to ask.
However, it does not extract existing closed captions, it generates the subtitles SRT file from the audio.
Also, I will be updating it shortly.

Edited December 27, 2024 by BillOatman

[BillOatman 565]

Latest release

Updated to the Latest Whisper, so you will need to update the runtimes.