Jump to content

Emby Server - ffmpeg external transcoding, plugin + app


Cheeseburger

Recommended Posts

Cheeseburger

Hi!

 

I suggest a Premium plugin/feature that would make Emby more powerful and not require expensive high end systems that are hard to scale, just to manage transcoding or syncing a few streams simultaneously, especially going down the 4K/HDR/x265 etc. paths... Most common use case would be if you run Emby server on a weak NAS that isn't powerful enough to transcode, maybe at all. This would definitively make Emby a more economical and feasible solution to many, being able to use already available hardware and not worry about getting a powerful enough server. So, lets make some use of that old laptop or desktop computer, that could be hiding anywhere in the house assuming a decent connection (i.e wired).

 
  • Create a plugin for Embyserver. Why not call it "DisTrac" or similar?
  • Create a standalone application/app, that basically just runs in the background, in the tray or as a service on another LAN connected PC.
    OR embed the functionality in to the Theater apps or as a plugin for them!
     
  • Don't do any fancy splitting/joining, just hand out full jobs to each "node".

In theory, it's plug and play with the defaults, get the server plug in, run Emby Theater or the app on a machine in the local network, and you now have more transcoding power, to be used for any client or sync job.

 

 

 

Detailed suggested way for doing it

  1. The standalone app/or Theater will basically have no settings except for a prompt that asks for the Emby server name/IP (with Theater even simpler as the server is already known). It will be built with an identical ffmpeg setup as what comes with the Emby server, and once registered as a transcoding client with the server, it'll be configured by and execute the commands it receives. Maybe a local setting to choose hardware acceleration, but most settings from current Transcoding page, and any extra features from the plugin page.
  2. In theory maybe existing ways of streaming/sending files from the server can be used, and the transcoding client could actually have a reduced copy of the Emby server functionality for making the result available to the main server (both ends client and server but the Emby Server still overall boss)
  3. The plugin, once enabled (toggle on/off), will have a simple priority system where all clients are listed, including the server itself. Disconnected clients will be remembered, but removable so that it's easier to manage. It will always fall back to local processing if there is no external client available. 
  4. Each client (and the server) will have to be assigned a relative "capacity" number for how many typical streams it can handle. Default is 2, and if all devices have this, they will share jobs equally.  Emby server running on PC hardware will also default to 2, but if run on NAS or similar hardware, will default to 1.
    1. ​There should also be a "hard limit" toggle per device.
    2. Higher rated devices will be assigned jobs first, and when there are as many jobs going as there is total available "capacity", any new jobs will keep being handed out relative to the capacity numbers, i.e. 4 jobs for the devices rated 2, and 2 jobs for the devices rated 1.
    3. Devices with the "Hard Limit" parameter set, will not be assigned more jobs than the capacity parameter set for it.
    4. A sync/conversion job (not live transcoding) will run up to a set number of parallel tasks, but always limited by the total "capacity" number, regardless of hard limit set or not set.
  5. Use WOL, like already exists between Emby Apps and the server. A checkbox per device. Choice for how much "capacity" to keep available (0-x no. of unused) before sending the WOL-command to the next computer in line. Prevent sleep if transcoding. If Theatre App is closed, keep running ffmpeg part in the background/tray.

 

 

In reality probably lots of hurdles -_- . But I think this could be a real deal-winner if it was made possible.  B)

 

 

  • Like 2
Link to comment
Share on other sites

Hi, I don't now if embedding into client apps is the approach we'd take, but the idea of being able to offload transcoding to slave machines is something we're interested in for the future. Thanks.

  • Like 3
Link to comment
Share on other sites

Waldonnis

Basically a distributed encoding setup, except with full files rather than segments?  If so, the big question for me is how to get the file(s)/data needed to the nodes efficiently.  There are lots of ways to do it, but they all have drawbacks that would either limit the overall capability/speed of the pseudo-cluster or cause additional initial playback delay.  This isn't a new problem, but it's one thing you wrestle with when designing clusters and figuring out how to distribute/manage a workload to X nodes.  Then there's the throughput aspect: you'd need a pretty capable network connection on a file server/NAS if you plan on using shares and employing more than a couple of nodes at once, especially with high bitrate sources....and a switch/router that can handle that much concurrent traffic.  For 1-2 nodes, that's probably not an issue, but considering how much some folks around here have put into their home server infrastructure, I suspect you'd see a few trying to run a dozen nodes or more.  Heck, I'd try it with a bunch of RPis just for grins  :P

 

Neat idea, though.  It's definitely workable, but it isn't gonna be easy to make something like that into a scalable "turnkey" solution that would work well on potentially older off-the-shelf desktops, IMO.  This would be considerably easier if you wanted to split a video into chunks, then distribute that to nodes since there are already scripts and other solutions for that, but farming an entire transcoding job to a single node makes it a bit tougher to do well (file sizes alone make it complicated).

Link to comment
Share on other sites

Cheeseburger

Sorry, I wasn't clear.

 

 

 

Basically a distributed encoding setup, except with full files rather than segments?

 

I meant that each transcode should be dealt with by just one node in order to reduce complexity and produce even quality. And each node would have to start downloading whilst serving the chunks to the server as they finish, like ffmpeg and the server does today.

 

Regarding scaling, that is gonna be a challenge from a design point but also from a hardware point of view. But primarily lets make it work well for the average user, and let us geeks accept that it will require more hard core kit and maybe some tinkering if we want many (powerful) nodes to work together. Network could easily be a bottleneck, likely so would most storage, as reading multiple files on mechanical drives is so much slower overall than just reading them one by one.

 

 

My intent/goal/vision

 

Short version of all below: First make it work with 1-2 extra normal computers aiding a normal Emby server on a NAS or PC, then let those who want go serious/get crazy.

 

Initial steps (simple mode)

  1. A solution that easily extends the performance of a weak setup, that is, boosting a NAS/RPi with a simple computer connected on the same switch running 100Mbps. That should be turnkey to help the average John Doe's setup, and make an Emby setup more economical to run (I.e. not have to buy new expensive hardware just to get started on something with good performance).
  2. A decent (i3 - i7 CPU, maybe a GPU or even a Xeon) Emby server (PC) should be able to have it's transcoding ability doubled if adding another equal computer to a 1000Mbps switch. This would help even the more serious Emby admins create a powerful solution assuming their setups are healthy and have fast enough storage in a thoughtful setup.

When the above is achieved, I think it would be a success and a lot of people would be happy, but some would still want more  B). But if there are hurdles to go further, well so be it. Primarily make sure 1 and 2 can work in an easy and stable way as that is what the majority would actually use and benefit from. If more is not considered stable enough at the time, limit the simple mode, and let the advanced mode or parts of it be limited to a config file and call that "unsupported/beta".

 

 

For those with a little bit to much free time or as my wife would say, obsessive interest in all things technical :huh: (advanced mode)

  1. Handling many nodes, both strong and weak, mixing servers or workstations or gaming rigs with laptops, RPis and NAS. Using gigabit and big or multiple switches, main server with multiple NICs, maybe 10GigE or dual Gigabit between Emby server and library NAS. Not necessarily very powerful but a complex setup.

Above is the most challenging setup, a multitude of odd hardware, which will probably be the hardest to make work properly. These users from a developer standpoint should not really be prioritised but their setups will be excellent testbeds for developing and improving upon the system. And of course great fun.

 

The professional users

  1. Professional user, using powerful servers and dedicated NAS, on 10GigE or greater, maybe supplying a school or company with a video library. Those will be powerful, starting from a quite simple setup with dual nodes (Emby server + extra transcoding node, could often be run in simple/default mode), that easily scale with a few extra nodes assuming network and storage keeps up (which usually can be upgraded easily in such an environment, where CPU and memory are usually prone to costlier and more complex jumps and have their limits on single server setups).

 

These should, at least with few nodes, be easy to get running well (like no. 2 under "Initial steps" above, just more grunt)  and will open up Emby for more professional use. That with the new authentication plugin would make a new customer branch for Emby possible that could generate nice revenue that could be used for many things. It would probably substitute the need for complex custom solutions for load balancing that for a few reasons probably aren't that optimal.

Link to comment
Share on other sites

ddurdle

It is possible today to already do slave transcoding due to the fact emby uses ffmpeg.  I'm working on a proof-of-concept on an emby setup.  I'll share my findings after I've implementing.

  • Like 3
Link to comment
Share on other sites

Guest
This topic is now closed to further replies.
×
×
  • Create New...