Jump to content


Photo

Offset transcoding - Separate Transcoding Server


  • Please log in to reply
16 replies to this topic

#1 Bobba86 OFFLINE  

Bobba86

    Member

  • Members
  • 13 posts
  • Local time: 06:05 PM

Posted 27 December 2016 - 05:05 AM

I would like to see a plug-in developed which allows for transcoding to be offset.

My thoughts are for a transcoding server (high powered pc), running cut-down/transcoding only version of Emby.
Then on my main Emby server a plug-in which offsets transcoding to the high powered pc (if available)

Originally I had thought of this as being a high availability server
https://emby.media/c...gh-availability
But thinking about this more, my requirement isn't for high availability but simply to offset the transcoding elsewhere.
  • Mibok, Baenwort, otispresley and 7 others like this

#2 Deathsquirrel OFFLINE  

Deathsquirrel

    Advanced Member

  • Members
  • 1994 posts
  • Local time: 11:05 AM

Posted 27 December 2016 - 12:58 PM

In this scenario why aren't you running the server on the more powerful machine?  It couldn't really be powered down since the delay of booting and starting the transcoding process would confuse and annoy users.  If it's running all the time and you need that level of muscle why not run the services there?


  • Bobba86 likes this

#3 Luke OFFLINE  

Luke

    System Architect

  • Administrators
  • 135842 posts
  • Local time: 02:05 PM

Posted 27 December 2016 - 02:54 PM

I think this is a great idea.



#4 Bobba86 OFFLINE  

Bobba86

    Member

  • Members
  • 13 posts
  • Local time: 06:05 PM

Posted 27 December 2016 - 05:03 PM

Deathsquirrel,
I dont run services there because I only need the muscle sometimes.
My plan is to send WOL commands to wake my Muscle at 6pm and a scheduled task to shutdown the muscle at midnight.
Doing this will cover 80% of my watching needs, but I want my Emby server on 24/7.

Edited by Bobba86, 27 December 2016 - 05:04 PM.


#5 Deathsquirrel OFFLINE  

Deathsquirrel

    Advanced Member

  • Members
  • 1994 posts
  • Local time: 11:05 AM

Posted 27 December 2016 - 06:28 PM

Cool, I was curious about the use case is all.



#6 browned OFFLINE  

browned

    Advanced Member

  • Members
  • 161 posts
  • Local time: 08:05 AM

Posted 27 December 2016 - 09:47 PM

You could take it a step further and create a Emby Transcode Client" that monitors resources on each client and the server could send work to each client based on what it is capable of and what resources are being used.

 

I have 3 Nvidia Shields and two are asleep most of the time, My main server is OK but does not have hardware transcoding capability and uses a lot more power than my two shields. I could install the "Emby Transcode Client" on each shield and the server could send the jobs as required.


  • Mibok likes this

#7 MSattler OFFLINE  

MSattler

    Advanced Member

  • Alpha Testers
  • 1449 posts
  • Local time: 02:05 PM

Posted 28 December 2016 - 01:34 PM

You could take it a step further and create a Emby Transcode Client" that monitors resources on each client and the server could send work to each client based on what it is capable of and what resources are being used.

 

I have 3 Nvidia Shields and two are asleep most of the time, My main server is OK but does not have hardware transcoding capability and uses a lot more power than my two shields. I could install the "Emby Transcode Client" on each shield and the server could send the jobs as required.

 

While this will work in theory, how do we account for a transcode on that Shield that is 50% complete when someone then wants to watch a show on that same Shield.  Can it handle doing a transcode and say 4k playback at the same time?

 

I have always liked the thought of having transcoding slave servers.  The only worry I have is that with the Server being on one host and the transcoder on another, error handling becomes a nightmare.   Also if we now have a transcoding farm, but our main Emby server falls over, we are dead in the water.

 

If Emby heads down this route, I think the more logical option would be to move Emby from Sqlite to a dedicated database.  This would then allow you to build additional Emby servers, which can also transcode.  

 

You then place a HAProxy instance in front of it to balance the load.  

 

What you now have is redundant Emby servers, where your user requests are spread among multiple servers, and by spreading your users out this way you are also dividing the transcoding. 



#8 browned OFFLINE  

browned

    Advanced Member

  • Members
  • 161 posts
  • Local time: 08:05 AM

Posted 28 December 2016 - 06:20 PM

No real world tests on Shield performance but some on the Plex forums have indicated the Shield can transcode 2 x 1080p streams while playing a direct stream.

 

Whatever is developed, if it is developed, it could accommodate most use cases. I see it working like someone wants to watch remotely so If the server needs to transcode hand that off to a shield. If someone else needs to transcode hand that to another shield, both would post back to the server about the users watch progress in case of failure. Both would then accept up to one more transcode or the server would start one to balance the load.

 

Priority order or limits could be set on each transcoding client.

- Only do x remote transcodes on this device/these devices.

- Use this device for local TV recording transcodes only.

- Use this device for all transcoding and fall back to the server or other devices if required.

 

Not sure I see the benefit of HA when my main server has all the storage. To suit most use cases I think the transcoding client would be a more favorable option, unless a majority have network storage devices?



#9 MSattler OFFLINE  

MSattler

    Advanced Member

  • Alpha Testers
  • 1449 posts
  • Local time: 02:05 PM

Posted 28 December 2016 - 11:53 PM

 

Not sure I see the benefit of HA when my main server has all the storage. To suit most use cases I think the transcoding client would be a more favorable option, unless a majority have network storage devices?

 

For those of us with very large libraries <55TB for myself>, we will never be able to contain all the media in one rig.  That and the beauty of having 2 storage servers is that I can perform maintenance on one, and still have half of my content up during that time.   



#10 fbartels OFFLINE  

fbartels

    Member

  • Members
  • 29 posts
  • Local time: 07:05 PM

Posted 26 July 2017 - 01:44 AM

Theoretically speaking: since you can specify the path to you ffmpeg binary, why not put a script into the path emby knows that takes care of waking up you rig and sending the file to transcode over?



#11 viceversa OFFLINE  

viceversa

    Newbie

  • Members
  • 7 posts
  • Local time: 12:05 PM

Posted 26 July 2017 - 03:06 PM

Would love to see this feature.  At minimum, the ability to have a specified remote machine take care of all transcode requests with ability to send WOL to said machine when needed.

 

EDIT: Apologies in advance for the "hey look, you can do that on plex" plug.  Found a project that accomplished just that.  Basically having a single installation of the server then slave nodes which take care of transcoding requests.

 

https://github.com/w...mote-Transcoder

 

Possible to adapt this model for Emby?


Edited by viceversa, 26 July 2017 - 03:24 PM.


#12 lazyboy0172 OFFLINE  

lazyboy0172

    Member

  • Members
  • 19 posts
  • Local time: 06:05 PM

Posted 07 February 2019 - 09:43 AM

found this thread via search and unfortunately don't feel great about the odds of a feature coming from reviving a 2+y/o thread, but hopeful something like this could happen.

 

Alternatively, how much larger would it make emby apps if they included transcoding software where available? many emby app platforms also have server support, so at least in those instances it may be possible to add, just at the expense of increased app space. The point here would be there could then be an additional option on the server user settings to allow playback of media if locally transcoded.

 

As someone with multiple Nvidia Shields around the home and one acting as a server, it feels silly that one machine may be asked to transcode multiple files at once, while someone may be trying to use that device for other purposes, all while the clients playing the media are using the very same hardware and being under-utilized.


  • Mibok likes this

#13 Baenwort OFFLINE  

Baenwort

    Advanced Member

  • Members
  • 148 posts
  • Local time: 07:05 PM

Posted 14 February 2019 - 01:09 AM

If Emby used or easily supported using Ripbot264 instead of ffmpeg you could use Ripbot264's distributed encoding feature. https://forum.videoh...Encoding-GuideĀ 

 

You can do it currently if you are just wanting to use it to transcode recorded TV or batch convert something before it is requested.  Using the "LiveTV" post recording processing option to specify a script file that calls Ripbot264 to do the cluster encoding you could leverage it that way.

 

Before Emby added the ability to use the "Convert" function there were several scripts that people made for batch converting libraries that could be modified to use ripbot264 instead of Handbrake.  Perhaps Luke would consider enabling the specification of a encoding engine in the "Folder Sync" plug in. That would able a GUI method to batch convert using a distributed encoding process.

 

It won't help with live transcoding but it's a start?



#14 crusher11 OFFLINE  

crusher11

    Advanced Member

  • Members
  • 490 posts
  • Local time: 02:05 AM

Posted 14 February 2019 - 02:42 AM

In this scenario why aren't you running the server on the more powerful machine?  It couldn't really be powered down since the delay of booting and starting the transcoding process would confuse and annoy users.  If it's running all the time and you need that level of muscle why not run the services there?

Is it possible to run the server off a PC and store the files on a network-connected NAS? If not, that could be one circumstance.



#15 Deathsquirrel OFFLINE  

Deathsquirrel

    Advanced Member

  • Members
  • 1994 posts
  • Local time: 11:05 AM

Posted 14 February 2019 - 11:58 PM

Is it possible to run the server off a PC and store the files on a network-connected NAS? If not, that could be one circumstance.

 

Of course it is, that's how I do it myself.



#16 cayars OFFLINE  

cayars

    Advanced Member

  • Alpha Testers
  • 2935 posts
  • Local time: 02:05 PM

Posted 16 February 2019 - 12:51 AM

found this thread via search and unfortunately don't feel great about the odds of a feature coming from reviving a 2+y/o thread, but hopeful something like this could happen.

 

If Emby used or easily supported using Ripbot264 instead of ffmpeg you could use Ripbot264's distributed encoding feature. https://forum.videoh...Encoding-GuideĀ 

There really isn't a big need for this type of thing anymore.

 

You actually used to be able to do this if you were creative but once HW transcoding was introduced it made this sort of thing really difficult to setup.  On the other hand the ability to use HW transcoding has really made the need for distributed transcoding a thing of the past since a single HW GPU can handle 20+ transcodes. Need more transcodes then that? Add another HW GPU.  That's way "higher end" then what 99.9% of Emby admins need but can be done.

 

Ripbot264 wasn't that good a program and why it never really cought on.  It was basically just a scheduler that allowed multiple computers to queue up transcoding jobs for offline processing which is completely different then our current needs.

 

Besides the HW GPU route there are several other ways to solve the need/problem:

1) Pre-convert/process your media so it can direct play on all clients.  The only exception would be low bandwidth situations.

2) Split your libraries to 2 or more servers which could be Shield TVs.  For example TV Shows on one box and Movies on the 2nd.  This would require switching servers but would allow more overall transcodes.

3) Similar to #2 is to make use of multiple servers but set them up the same pointing to the same file locations.  The difference in this setup is you split your users between your different servers (shield TVs).

4) Run your servers in a load balanced environment (needs a bit more complex setup).

 

There are other methods similar setups to the above but the first 3 are the easiest to anyone to setup and use.  None of those require any additional code work in Emby to function and can be scaled out as needed.

 

Also Emby itself supports syncing to cloud providers which can also alleviate the need to real-time transcode and helps with limited upload bandwidth situations as well.

 

Non of this is meant to say this wouldn't be a cool feature to have built in, but it's limited in need/scope and there are surely more important features that would be usable to most admins that are far more important.



#17 jojo220 OFFLINE  

jojo220

    Newbie

  • Members
  • 5 posts
  • Local time: 07:05 PM

Posted 17 February 2019 - 09:25 AM

I would like to see a plug-in developed which allows for Xender Discord Omegle transcoding to be offset.
My thoughts are for a transcoding server (high powered pc), running cut-down/transcoding only version of Emby.
Then on my main Emby server a plug-in which offsets transcoding to the high powered pc (if available)
Originally I had thought of this as being a high availability server
https://emby.media/c...gh-availability
But thinking about this more, my requirement isn't for high availability but simply to offset the transcoding elsewhere.

at the expense of increased app space. The point here would be there could then be an additional option on the server user settings to allow playback of media

Edited by jojo220, 17 February 2019 - 03:12 PM.





0 user(s) are reading this topic

0 members, 0 guests, 0 anonymous users