Live-Video-Translation Plugin Beta using OpenAI Whisper

[bobo99 10]

Hey All,

After a few weeks of work I'm happy to present a new plugin Beta, called "Live Video Translation".

The plugin uses Whisper, Open Ai's translation/transcription service (https://github.com/openai/whisper), to generate translated subtitles on the fly. This allows you to watch TV with live generated subtitles. If the original source is detected as English you'll get transcribed subtitles (English to English). If the original language is any of the supported languages, you'll get English subtitles.

Languages supported are:

https://github.com/openai/whisper?tab=readme-ov-file#available-models-and-languages

How to Install

1. Run the following Whisper docker container somewhere that emby can access it. Make sure to run this docker container using the "GPU" flag. I have not tested CPU mode, but we do need good performance here which may not be available with just a CPU and LLMs.

https://github.com/ahmetoner/whisper-asr-webservice

2. Run the docker container using either the "small" or "medium" models. In my testing at 1070 was good enough to keep up with near real time translation using small or medium models. The model download took about 10 minutes the first time.

3. Place the attached dll into the plugin directory.

4.  Configure your plugin like so, with the full URI to the Whisper service, and with an EMBY API key. There was one thing that can't be accomplished with just a plugin. (a rapid way to add a library and scan just that library for new files).

5. Click on "Enable" at the top, and restart emby.

 

How it works

Currently the plugin only supports LiveTV. There are free LiveTV providers on the first few results of google that can allow you to test this for free.

1. User finds a Live TV Channel in Emby and Clicks "Play"

2. Plugin detects that the user wants to watch a Live TV Channel (and only live tv for now)

3. Emby starts playing the Channel, while in the background the plugin also starts consuming the same channel using ffmpeg. (For a short period of time there are 2 separate connections consuming the same channel)

4) The plugin adds a temporary library to emby that contains a single file. That file is a link to play the translated Channel Stream. It does not scan all the files that ffmpeg is creating, just a static stream file which points to our Plugin API, which serves files for that translation session.

5). The plugin starts creating chunks of the video and translating it, and generating a valid stream that emby can play.

6) Once a buffer of a few files has been created, the Plugin stops the initial stream, and starts the newly translated stream and immediately:

- Pauses it

- Displays a message to the user to wait for ~20 seconds until a bigger buffer is created

- Restarts playback about 20-30 seconds passes

These delays are very conservative to allow for any LiveTV ingestion hiccups. I have been able to watch near real time, with just in time delivery of the subtitiles, but have noticed certain IPTV providers hiccup and don't serve video in a timely manner, so this is a way of riding out those hiccups.

7)The new TV stream uses WEBVTT subtitles, which means emby properly parses them, and you can adjust the subtitle offsets, even though they should already be pretty accurate.

8 ) When the user stops playback (only on stopping playback, not closing an app, or closing a browser window), the ffmpeg consumer is killed, the folder deleted, and the newly added library is deleted. The library may still show up if you navigate home immediately, but if you refresh the page it'll be gone.

 

Caveats/Future Improvements

- The logs for this plugin, for this Beta, are very verbose in order to help troubleshooting.

- Sometimes an IPTV provider is rocky and does not serve video files in a timely manner. Which means you'll be watching the translated stream and it'll overrun the buffer. This happens infrequently in my testing.

- I have two test instances

a) Windows with CUDA , I had to disable hardware transcoding to get this to work, unsure why.

b) Linux with CUDA, works fine with hardware trans-coding enabled.

- If there is interest in this I could add live translation to TV Shows/Movies.

- I don't control the translation. Sometimes AI hallucinates, but overall it is very good.  Sometimes if there is no audio, it seems like the docker container returns "Thanks for watching".

- Sometimes the pausing/message on screen does not work. If it's been longer than 30 seconds feel free to click play in the translated stream.

Questions for the devs

- Sometime the plugin consuming FFMPEG receives corrupted packets from the provider and isn't able to handle it, and it stops.

- Sometimes emby's consuming FFMPEG (the ones that consumes the plugin's translated stream) consumes a corrupted packet, and just stops consuming the stream. I'm unsure which one is the problem first here. Seeking guidance from the devs on how I can modify my translation ffmpeg command to account for this. You'll see what the translation ffmpeg command is in the logs shortly after a playbackstart. The translation ffmpeg command just "consumes" the stream to output video files, translation is done separately.

- Sometimes sending a command like this, just does not work. Almost as if my code skips it, but it should be called each time.

 var pause_request = new PlaystateRequest
 {
     Command = PlaystateCommand.PlayPause
 };
 _sessionManager.SendPlaystateCommand(null, sessionId, pause_request, CancellationToken.None);

 

Troubleshooting

If this isn't working for you, please provide:

embyserver.txt - this will have a link to your IPTV in it, and maybe a username, you can obfuscate that if you'd like.

The most recent ffmpeg logs which represent the emby's ffmpeg that is trying to play the translated stream.

What OS are you using, is hardware trans-coding enabled?

Occasionally monitor the buffer health (look at how much video is "left" in the stream). It should be growing as you watch) If it isn't that means either the translation or consuming ffmpeg has encountered an error and I'll need to see the logs as to why.

If you watch that same TV channel, with the plugin disabled, does it fail in a similar way? Please provide an ffmpeg log of that.

Thanks!

 

 

LiveVideoTranslation.dll

[akacharos 15]

This is a great idea for providing subtitles on-the-fly when there are no subtitles available.
I started using gui-subtrans to generate subtitles with LLMs but it's a slow process. One cool feature of gui-subtrans is that it feeds small batches instead of line by line so the LLM has bigger context to provide a more accurate translation.Is there a similar feature in Whisper ASR that can be configured?

One more thing: you noted that "Windows with CUDA , I had to disable hardware transcoding to get this to work, unsure why". Does that mean that we need to disable Cuda transcoding in Windows Emby server for this to work?

[bobo99 10]

6 minutes ago, akacharos said:

This is a great idea for providing subtitles on-the-fly when there are no subtitles available.
I started using gui-subtrans to generate subtitles with LLMs but it's a slow process. One cool feature of gui-subtrans is that it feeds small batches instead of line by line so the LLM has bigger context to provide a more accurate translation.Is there a similar feature in Whisper ASR that can be configured?

One more thing: you noted that "Windows with CUDA , I had to disable hardware transcoding to get this to work, unsure why". Does that mean that we need to disable Cuda transcoding in Windows Emby server for this to work?

Currently the batch size isn't exposed to the user,  but something I can try making selectable. Currently it's 5 second chunks as a compromise between performance and buffer size.

Regarding CUDA I think it's just a bug in my environment, but give it a try and let me know.

[raptorjr 14]

I give a vote for also including movies/Tv-shows, if possible. As a non native speaker of Egnlish, and subtitles isn't always available, getting things texted helps a lot. And maybe also translation could be possible one day.

[bobo99 10]

On 30/01/2025 at 15:11, raptorjr said:

I give a vote for also including movies/Tv-shows, if possible. As a non native speaker of Egnlish, and subtitles isn't always available, getting things texted helps a lot. And maybe also translation could be possible one day.

Noted!

I notice there have been a few downloads. Does anyone have any feedback on the plugin as it stands now ?

[Luke 40018]

@raptorjr?

[bobo99 10]

5 hours ago, Luke said:

@raptorjr?

Let me get up a new version for test today. That DLL expired after 2 weeks

[bobo99 10]

19 hours ago, Luke said:

@raptorjr?

@raptorjrI've now added support for all video types, that are selectable in the plugin options.

The plugin currently handles Live TV very well.

For this plugin to work I need chunks of video that provide enough context for the LLM to translate/transcribe the subtitiles. This is currently 5 second chunks. In order to get 5 second chunks I am re-encoding the videos on the fly to force keyframes.

If your video is an HDR or high bitrate file, the plugin does not support that. Non-hdr lower bitrate files are easy to process in a timely manner.

If there are issues please post the full embyserver.txt log as well as the corresponding ffmpeg log that emby will generate while consuming the stream

Attached is the new plugin dll file. It is only good for 14 days from today.

Download the dll, and put it in your plugins folder.

Restart emby, fill out the plugin details and check what type of video you want to translate. I recommend you start with LiveTV.

Feedback is very welcome!

 

LiveVideoTranslation.dll

Edited March 2 by bobo99

[AXIANT 4]

I would love to use this on media in my library. Lots of things I have don’t have subtitles or they are not timed correctly. This would be fantastic!

[bobo99 10]

On 02/03/2025 at 12:26, AXIANT said:

I would love to use this on media in my library. Lots of things I have don’t have subtitles or they are not timed correctly. This would be fantastic!

Great please give it a try and let me know how it works for you.

 

[bobo99 10]

All, if you'd like to see this as a proper plugin, the only way for it to make it to the app store is if you provide feedback on its current usage. This is what the emby devs use to indicate demand for a plugin

 

So if you'd like to see this as a proper app please try and provide feedback !

[raptorjr 14]

So, I just started to try this on TV Shows. It is so cool, even if I can't get it to work properly  But I see a few translated lines and that was enough for me to be excited.

Anyway, my problems can probably be many. And I will try and help and test things as much as possible if that is of any use.

So the things that I have noticed is:

I sometimes get the message to wait 20 seconds for the stream to buffer. That message is on the screen for a very short time. It could stay longer, or if possible stay and show a countdown of the 20 seconds?

But even after waiting for more than 20 seconds the stream doesn't always restart. But when I exit the playback, because I don't want to wait anymore, the new stream starts with the translations.

But that new stream only works for about 60 seconds then everything stops. But I see in the log that there are 5 seconds chunks being processed/translated. But no playback on screen.

I also noticed that when the new stream starts, the show/episode information is gone and replaced by the text Stream.

 

But for that 60 seconds, even if translations is not perfect, so cool!!! I'm not natively speaking English, so sometimes it is hard to hear/understand everything when there is no text. So this is just amazing.  I don't know if my problems have anything with just using CPU power and not GPU? I haven't found out how to use the iGPU in my Lenovo M720q, if possible. But the CPU is only on around 40% during the translation so I hope it is enough power anyway. Also, would it be possible to have this as a background task for tv shows/movies that don't have any subtitles at all? Or does everything need to happen real time? Just a thought. 

 

Is there anything I can help with when it comes to logs or testing I will try and help.

 

[bobo99 10]

1 hour ago, raptorjr said:

So, I just started to try this on TV Shows. It is so cool, even if I can't get it to work properly  But I see a few translated lines and that was enough for me to be excited.

Anyway, my problems can probably be many. And I will try and help and test things as much as possible if that is of any use.

So the things that I have noticed is:

I sometimes get the message to wait 20 seconds for the stream to buffer. That message is on the screen for a very short time. It could stay longer, or if possible stay and show a countdown of the 20 seconds?

But even after waiting for more than 20 seconds the stream doesn't always restart. But when I exit the playback, because I don't want to wait anymore, the new stream starts with the translations.

But that new stream only works for about 60 seconds then everything stops. But I see in the log that there are 5 seconds chunks being processed/translated. But no playback on screen.

I also noticed that when the new stream starts, the show/episode information is gone and replaced by the text Stream.

 

But for that 60 seconds, even if translations is not perfect, so cool!!! I'm not natively speaking English, so sometimes it is hard to hear/understand everything when there is no text. So this is just amazing.  I don't know if my problems have anything with just using CPU power and not GPU? I haven't found out how to use the iGPU in my Lenovo M720q, if possible. But the CPU is only on around 40% during the translation so I hope it is enough power anyway. Also, would it be possible to have this as a background task for tv shows/movies that don't have any subtitles at all? Or does everything need to happen real time? Just a thought. 

 

Is there anything I can help with when it comes to logs or testing I will try and help.

 

Thanks for feedback

 

- the message on screen can be improved, I'll look into that. However if the plugin can translate it will eventually start the translated stream whenever the buffer is big enough , if it can't it just won't start.

- I am calculating in the plugin what a healthy buffer is, so youll have to be patient. That being said I've never tested this with cpu translation, but that could be your problem. Are you running the docker container in cpu mode ?

Please restart emby, try a playback of one file. Post embyserver.txt and an ffmpeg log of the last playback. Screenshot of how you're running your docker container, and how much ram /what cpu your laptop is running. What GPU do you have in your laptop?

What I suspect is happening is this can't be run on CPU. You're waiting a long time because CPU translation is slow using the docker container. That means it takes a long time to generate the buffer. Then when you play, you consume the buffer faster than the CPU can generate the subtitles and refresh the buffer.

 

The logs will answer that question

 

 

[raptorjr 14]

I'm using a Lenovo M720q, with 16MB memory, Intel® Core™ i7-8700T CPU @ 2.40GHz, with iGPU UHD Graphics 630. I have Unraid installed and the whisper container is running in that environment. Around 40% CPU when playback with trying to translate.

This time the stream restarted and started to show subtitles. I had to press play on two occasions for the stream to continue, but after 60 seconds playback was halted.

If it is because I use CPU, maybe there could be other settings to the container that can be used to make it faster? Or if I should dedicate cores or memory to that whisper container. If that is possible, I'm new to Unraid.

embyserver.txt ffmpeg-directstream-b20e298c-a936-41d2-aa7e-fa906285fe6c_1.txt

Edited March 10 by raptorjr

[Neminem 878]

@raptorjrand @bobo99If you need pre-created subs look into Bazarr it can use whisper as an subtitle provider and create english subs from any audio tracks.

Then I use Lingarr to translate them into other languages.

Using whisper with CPU is slow.

Did some testing. 

45min episode.

CPU time : around 50min too far behind real time.

13th Gen Intel® Core™ i5-13500 @ 2475 MHz using all cores. 

GPU time : around 3 min really great.

GeForce GTX 1080 at 100% usage, but only for those 3 min.

I short real time with only a CPU will get you in trouble, even with 1 stream

Hope this info is useful

[raptorjr 14]

Yes, maybe that is the route I have to take. Not really necessary to translate TV Shows and Movies in real time. Although it would be nice if in the future this could be added as a Task for TV Shows and Movies so we don't need Bazarr. But if my testing in someway can help anyway I can continue ant test things and send logs. If there is any other issues besides me using CPU power?

[bobo99 10]

On 10/03/2025 at 08:48, raptorjr said:

I'm using a Lenovo M720q, with 16MB memory, Intel® Core™ i7-8700T CPU @ 2.40GHz, with iGPU UHD Graphics 630. I have Unraid installed and the whisper container is running in that environment. Around 40% CPU when playback with trying to translate.

This time the stream restarted and started to show subtitles. I had to press play on two occasions for the stream to continue, but after 60 seconds playback was halted.

If it is because I use CPU, maybe there could be other settings to the container that can be used to make it faster? Or if I should dedicate cores or memory to that whisper container. If that is possible, I'm new to Unraid.

embyserver.txt 542.29 kB · 1 download ffmpeg-directstream-b20e298c-a936-41d2-aa7e-fa906285fe6c_1.txt 46.54 kB · 1 download

So looking at the logs it looks like you're trying to skip ahead/scrub through the timeline ? This is currently not supported. Also I just tried it with my CPU (Ryzen 5800x3D) and it ran successfully, although was slower, which required some buffer adjustments.

I have made some adjustments to the  how the pausing/starting works. In short it will will automatically start the new stream. It will then pause it, then restart when ready. If you're on CPU mode for translations, you'll have to wait longer. I've added a new chunk size in the options, but you can ignore that for now, just set it to 5. I'll post a new dll in a post after this. Please try it again and post your logs again. Try a couple files, and remember you have to cleanly click "back" to stop all transcoding/translation.

 

 

On 10/03/2025 at 13:30, Neminem said:

@raptorjrand @bobo99If you need pre-created subs look into Bazarr it can use whisper as an subtitle provider and create english subs from any audio tracks.

Then I use Lingarr to translate them into other languages.

Using whisper with CPU is slow.

Did some testing. 

45min episode.

CPU time : around 50min too far behind real time.

13th Gen Intel® Core™ i5-13500 @ 2475 MHz using all cores. 

GPU time : around 3 min really great.

GeForce GTX 1080 at 100% usage, but only for those 3 min.

I short real time with only a CPU will get you in trouble, even with 1 stream

Hope this info is useful

This plugin was really just intended to translate LiveTV. I added to options to do movies/tv shows as it wasn't that much more code.

[bobo99 10]

LiveVideoTranslation.dll
New version of the plugin:

Change log:

- added option for changing "chunk size", it doesn't yet work as intended

- Cleaned up the routine on starting the playback of the translated stream.

[Neminem 878]

1 hour ago, bobo99 said:

This plugin was really just intended to translate LiveTV. I added to options to do movies/tv shows as it wasn't that much more code.

Yes, sorry about the mess I have made. 
Just wanted @raptorjrto know he/she's options.

But I do get that my way would not work in a live tv transmission

and only work after the fact, when transmission ended and saved to disk.

[axelsl 17]

This is so exciting! Will try this plugin asap, thanks @bobo99! I will let you know how it goes. I am a fan of watching international TV so this could be a game changer for me.

PS: So far I was using a chrome plugin to transcribe/translate, but of course only works in Chrome. I can't wait to try this on different apps/platforms!

[axelsl 17]

Hi @bobo99is the latest dll not valid anymore? I've been trying to install the plugin without success. Would you be able to upload a new one? Thanks!

 

Edit: I get the following error in the embyserver logs when it fails to create the plugin: SimpleInjector.ActivationException: SimpleInjector.ActivationException: No registration for type Plugin could be found and an implicit registration could not be made. The type initializer for '<Module>' threw an exception. This assembly is protected by an unregistered version of Eziriz's ".NET Reactor"! This assembly won't further work.

Full log file attached.

embyserver (4).txt

Edited April 20 by axelsl

[bobo99 10]

3 hours ago, axelsl said:

Hi @bobo99is the latest dll not valid anymore? I've been trying to install the plugin without success. Would you be able to upload a new one? Thanks!

 

Edit: I get the following error in the embyserver logs when it fails to create the plugin: SimpleInjector.ActivationException: SimpleInjector.ActivationException: No registration for type Plugin could be found and an implicit registration could not be made. The type initializer for '<Module>' threw an exception. This assembly is protected by an unregistered version of Eziriz's ".NET Reactor"! This assembly won't further work.

Full log file attached.

embyserver (4).txt 56.23 kB · 0 downloads

Hey yea the DLL was only valid for 2 weeks. I'll have to upload a new one.

 

I'll get a new one uploaded by Wednesday

[axelsl 17]

18 hours ago, bobo99 said:

Hey yea the DLL was only valid for 2 weeks. I'll have to upload a new one.

 

I'll get a new one uploaded by Wednesday

Thank you! Can’t wait to test this. I’ll let you know what I find out!

[axelsl 17]

Hi @bobo99! Any chance you'd be able to upload the new dll? I have some time I could use to test the plugin today and report back.

Thanks!

[bobo99 10]

2 hours ago, axelsl said:

Hi @bobo99! Any chance you'd be able to upload the new dll? I have some time I could use to test the plugin today and report back.

Thanks!LiveVideoTranslation.dll

Here you are, it should work for 14 days.

Please let me know how it works for you!