Jump to content

Offset transcoding - Separate Transcoding Server


Bobba86

Recommended Posts

Bobba86

I would like to see a plug-in developed which allows for transcoding to be offset.

 

My thoughts are for a transcoding server (high powered pc), running cut-down/transcoding only version of Emby.

Then on my main Emby server a plug-in which offsets transcoding to the high powered pc (if available)

 

Originally I had thought of this as being a high availability server

https://emby.media/community/index.php?/topic/41443-emby-server-high-availability

But thinking about this more, my requirement isn't for high availability but simply to offset the transcoding elsewhere.

  • Like 26
  • Agree 1
Link to comment
Share on other sites

Deathsquirrel

In this scenario why aren't you running the server on the more powerful machine?  It couldn't really be powered down since the delay of booting and starting the transcoding process would confuse and annoy users.  If it's running all the time and you need that level of muscle why not run the services there?

  • Like 2
Link to comment
Share on other sites

Bobba86

Deathsquirrel,

I dont run services there because I only need the muscle sometimes.

My plan is to send WOL commands to wake my Muscle at 6pm and a scheduled task to shutdown the muscle at midnight.

Doing this will cover 80% of my watching needs, but I want my Emby server on 24/7.

Edited by Bobba86
  • Like 1
Link to comment
Share on other sites

browned

You could take it a step further and create a Emby Transcode Client" that monitors resources on each client and the server could send work to each client based on what it is capable of and what resources are being used.

 

I have 3 Nvidia Shields and two are asleep most of the time, My main server is OK but does not have hardware transcoding capability and uses a lot more power than my two shields. I could install the "Emby Transcode Client" on each shield and the server could send the jobs as required.

  • Like 4
Link to comment
Share on other sites

MSattler

You could take it a step further and create a Emby Transcode Client" that monitors resources on each client and the server could send work to each client based on what it is capable of and what resources are being used.

 

I have 3 Nvidia Shields and two are asleep most of the time, My main server is OK but does not have hardware transcoding capability and uses a lot more power than my two shields. I could install the "Emby Transcode Client" on each shield and the server could send the jobs as required.

 

While this will work in theory, how do we account for a transcode on that Shield that is 50% complete when someone then wants to watch a show on that same Shield.  Can it handle doing a transcode and say 4k playback at the same time?

 

I have always liked the thought of having transcoding slave servers.  The only worry I have is that with the Server being on one host and the transcoder on another, error handling becomes a nightmare.   Also if we now have a transcoding farm, but our main Emby server falls over, we are dead in the water.

 

If Emby heads down this route, I think the more logical option would be to move Emby from Sqlite to a dedicated database.  This would then allow you to build additional Emby servers, which can also transcode.  

 

You then place a HAProxy instance in front of it to balance the load.  

 

What you now have is redundant Emby servers, where your user requests are spread among multiple servers, and by spreading your users out this way you are also dividing the transcoding. 

Link to comment
Share on other sites

browned

No real world tests on Shield performance but some on the Plex forums have indicated the Shield can transcode 2 x 1080p streams while playing a direct stream.

 

Whatever is developed, if it is developed, it could accommodate most use cases. I see it working like someone wants to watch remotely so If the server needs to transcode hand that off to a shield. If someone else needs to transcode hand that to another shield, both would post back to the server about the users watch progress in case of failure. Both would then accept up to one more transcode or the server would start one to balance the load.

 

Priority order or limits could be set on each transcoding client.

- Only do x remote transcodes on this device/these devices.

- Use this device for local TV recording transcodes only.

- Use this device for all transcoding and fall back to the server or other devices if required.

 

Not sure I see the benefit of HA when my main server has all the storage. To suit most use cases I think the transcoding client would be a more favorable option, unless a majority have network storage devices?

Link to comment
Share on other sites

MSattler

 

Not sure I see the benefit of HA when my main server has all the storage. To suit most use cases I think the transcoding client would be a more favorable option, unless a majority have network storage devices?

 

For those of us with very large libraries <55TB for myself>, we will never be able to contain all the media in one rig.  That and the beauty of having 2 storage servers is that I can perform maintenance on one, and still have half of my content up during that time.   

Link to comment
Share on other sites

  • 6 months later...
fbartels

Theoretically speaking: since you can specify the path to you ffmpeg binary, why not put a script into the path emby knows that takes care of waking up you rig and sending the file to transcode over?

Link to comment
Share on other sites

viceversa

Would love to see this feature.  At minimum, the ability to have a specified remote machine take care of all transcode requests with ability to send WOL to said machine when needed.

 

EDIT: Apologies in advance for the "hey look, you can do that on plex" plug.  Found a project that accomplished just that.  Basically having a single installation of the server then slave nodes which take care of transcoding requests.

 

https://github.com/wnielson/Plex-Remote-Transcoder

 

Possible to adapt this model for Emby?

Edited by viceversa
Link to comment
Share on other sites

  • 1 year later...
lazyboy0172

found this thread via search and unfortunately don't feel great about the odds of a feature coming from reviving a 2+y/o thread, but hopeful something like this could happen.

 

Alternatively, how much larger would it make emby apps if they included transcoding software where available? many emby app platforms also have server support, so at least in those instances it may be possible to add, just at the expense of increased app space. The point here would be there could then be an additional option on the server user settings to allow playback of media if locally transcoded.

 

As someone with multiple Nvidia Shields around the home and one acting as a server, it feels silly that one machine may be asked to transcode multiple files at once, while someone may be trying to use that device for other purposes, all while the clients playing the media are using the very same hardware and being under-utilized.

  • Like 1
Link to comment
Share on other sites

Baenwort

If Emby used or easily supported using Ripbot264 instead of ffmpeg you could use Ripbot264's distributed encoding feature. https://forum.videohelp.com/threads/386325-Ripbot264-Distributed-Encoding-Guide 

 

You can do it currently if you are just wanting to use it to transcode recorded TV or batch convert something before it is requested.  Using the "LiveTV" post recording processing option to specify a script file that calls Ripbot264 to do the cluster encoding you could leverage it that way.

 

Before Emby added the ability to use the "Convert" function there were several scripts that people made for batch converting libraries that could be modified to use ripbot264 instead of Handbrake.  Perhaps Luke would consider enabling the specification of a encoding engine in the "Folder Sync" plug in. That would able a GUI method to batch convert using a distributed encoding process.

 

It won't help with live transcoding but it's a start?

Link to comment
Share on other sites

crusher11

In this scenario why aren't you running the server on the more powerful machine?  It couldn't really be powered down since the delay of booting and starting the transcoding process would confuse and annoy users.  If it's running all the time and you need that level of muscle why not run the services there?

Is it possible to run the server off a PC and store the files on a network-connected NAS? If not, that could be one circumstance.

Link to comment
Share on other sites

Deathsquirrel

Is it possible to run the server off a PC and store the files on a network-connected NAS? If not, that could be one circumstance.

 

Of course it is, that's how I do it myself.

Link to comment
Share on other sites

found this thread via search and unfortunately don't feel great about the odds of a feature coming from reviving a 2+y/o thread, but hopeful something like this could happen.

 

If Emby used or easily supported using Ripbot264 instead of ffmpeg you could use Ripbot264's distributed encoding feature. https://forum.videohelp.com/threads/386325-Ripbot264-Distributed-Encoding-Guide 

There really isn't a big need for this type of thing anymore.

 

You actually used to be able to do this if you were creative but once HW transcoding was introduced it made this sort of thing really difficult to setup.  On the other hand the ability to use HW transcoding has really made the need for distributed transcoding a thing of the past since a single HW GPU can handle 20+ transcodes. Need more transcodes then that? Add another HW GPU.  That's way "higher end" then what 99.9% of Emby admins need but can be done.

 

Ripbot264 wasn't that good a program and why it never really cought on.  It was basically just a scheduler that allowed multiple computers to queue up transcoding jobs for offline processing which is completely different then our current needs.

 

Besides the HW GPU route there are several other ways to solve the need/problem:

1) Pre-convert/process your media so it can direct play on all clients.  The only exception would be low bandwidth situations.

2) Split your libraries to 2 or more servers which could be Shield TVs.  For example TV Shows on one box and Movies on the 2nd.  This would require switching servers but would allow more overall transcodes.

3) Similar to #2 is to make use of multiple servers but set them up the same pointing to the same file locations.  The difference in this setup is you split your users between your different servers (shield TVs).

4) Run your servers in a load balanced environment (needs a bit more complex setup).

 

There are other methods similar setups to the above but the first 3 are the easiest to anyone to setup and use.  None of those require any additional code work in Emby to function and can be scaled out as needed.

 

Also Emby itself supports syncing to cloud providers which can also alleviate the need to real-time transcode and helps with limited upload bandwidth situations as well.

 

Non of this is meant to say this wouldn't be a cool feature to have built in, but it's limited in need/scope and there are surely more important features that would be usable to most admins that are far more important.

  • Like 1
Link to comment
Share on other sites

I would like to see a plug-in developed which allows for Xender Discord Omegle transcoding to be offset.

My thoughts are for a transcoding server (high powered pc), running cut-down/transcoding only version of Emby.

Then on my main Emby server a plug-in which offsets transcoding to the high powered pc (if available)

Originally I had thought of this as being a high availability server

https://emby.media/community/index.php?/topic/41443-emby-server-high-availability

But thinking about this more, my requirement isn't for high availability but simply to offset the transcoding elsewhere.

at the expense of increased app space. The point here would be there could then be an additional option on the server user settings to allow playback of media Edited by jojo220
Link to comment
Share on other sites

  • 10 months later...

There really isn't a big need for this type of thing anymore.

 

Non of this is meant to say this wouldn't be a cool feature to have built in, but it's limited in need/scope and there are surely more important features that would be usable to most admins that are far more important.

 

Is this your personal opinion, or are you deciding whats needed and what not?

 

Do you have Family or Friends? Are they using the SmartTV app? Do you have space saving x265 Rips?

 

I can tell you that most of the older TVs are not supporting HEVC.

 

Please, if you just look on your own Setup and not try to take a bigger look, just keep your opinions for yourself.

 

Transcode Offset is an must have feature.

Link to comment
Share on other sites

acnp77

I still think offset transcoding would be a great idea and a great way to use your HW.

 

The problem is that many servers typically do not have a decent GPU and also have no possibility to add one. When using a VM it is even worse.

To convert 1500 Movies is certainly not an option.

Also splitting up your servers seems like a really bad idea in terms of HW usage, configuration and maintenance - not a viable option IMHO.

Link to comment
Share on other sites

  • 8 months later...

I'm also bumping this thread, that I think it's better approach then moving Emby installation from NAS to dedicated box (mounting a share from NAS to eg. NUC will disable inotify/real time monitoring). It would be nice to just install daemon/agent on transcoding box and add it to Emby as transcoder. Alternative - it would be nice to have some small package for Synology which just watch directories for real time monitoring, and notify the server installed somewhere else - for me both solutions will work.

Link to comment
Share on other sites

  • 3 weeks later...
istvan.kovacs

I also agree it would be a great functionality to have offset transcoding.
Using your a low power consumption SBC to be an always-on media center which can handle up to 1080p transcoding, but offset the transcoding to your PC when it comes to 4k is something I' would love to see in Emby Premiere.

My use case would be something like: when it comes to a 4k transcoding it would trigger a WOL signal to the sleeping PC and run the transcoding client after that in a really short time it is available for transcoding, after the movie ends and there's no active transcoding on the client then trigger another signal to put the PC back to sleep.

What I could imagine  as a functionality in emby server is to enable offset transcoding on a given remote machine based on the resolution or bitrateand maybe and also an Emby weebhook if this before mentioned criteria is met.  Based on that in Home Assintant would be able to wake up , run the client and - since the 'Playback stop' webhook already available -  put the PC back to sleep making sure it only happens when 4k transcoding was in place. Hopefully everything could work before timeout on the playback :)

I hope offset trranscoding becomes a reality in Emby soon.

Link to comment
Share on other sites

  • 1 month later...
cyrus104

This would be a great feature. I would like to use a shield or a media pc with a nice card to do offset transcoding for devices that need it because my NAS with Emby is underpowered.

Link to comment
Share on other sites

rbjtech

Slightly offtopic - but related ... 🤪

One aspect that has puzzled me - is if a file is being transcoded anyway - why is there not an option to 'keep' that transcode for future use ?  (*)

It will then add as a multi-version - naming it with a suitable bandwidth ready for next time.   

I completely get that a 'real time' transcoded version will not be at the quality vs an offline encode at the same bandwidth - but it seems such a shame to 'throw away' all that CPU/GPU processing !

(*) I'm not even sure if this is technically possible - I presume it would need to concatenate all the .ts file back into a single file and would obviously only happen if the media was played 100% with no user intervention ... Maybe that is the reason ?

 

Edited by rbjtech
  • Agree 1
Link to comment
Share on other sites

8 minutes ago, rbjtech said:

Slightly offtopic - but related ... 🤪

One aspect that has puzzled me - is if a file is being transcoded anyway - why is there not an option to 'keep' that transcode for future use ?  (*)

It will then add as a multi-version - naming it with a suitable bandwidth ready for next time.   

I completely get that a 'real time' transcoded version will not be at the quality vs an offline encode at the same bandwidth - but it seems such a shame to 'throw away' all that CPU/GPU processing !

(*) I'm not even sure if this is technically possible - I presume it would need to concatenate all the .ts file back into a single file and would obviously only happen if the media was played 100% with no user intervention ... Maybe that is the reason ?

 

We have this capability with our Convert feature.  Trying to do it automatically like this would potentially create so many different versions of something that it just isn't practical.  Plus, we don't always transcode the whole thing for immediate playback.

In short, the convert feature will do what you are wanting in a much more effective and efficient manner.

Link to comment
Share on other sites

rbjtech
1 minute ago, ebr said:

We have this capability with our Convert feature.  Trying to do it automatically like this would potentially create so many different versions of something that it just isn't practical.  Plus, we don't always transcode the whole thing for immediate playback.

In short, the convert feature will do what you are wanting in a much more effective and efficient manner.

That is offline so not really what I was suggesting... 

Link to comment
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now
×
×
  • Create New...