Hardware for 4K HDR10 x265

[darrenkdean 29]

Have made the transition from H.264 to H.265.  At the same time, I am beginning to incorporate 4K HDR10 content into the Emby Library.  I am thinking of incorporating a dedicated GPU in tandem with Intel Quicksync to ensure there are no transcoding issues?  Thoughts/feedback?

Intel(R) Core(TM) i9-10900K CPU @ 3.70GHz, 20 cores
Intel UHD 630

Real memory

13.8 GiB used / 27.41 GiB cached / 46.87 GiB total

Operating system

Ubuntu Linux 20.04.3

Emby Server is hosted on NVME.  Transcoding folder hosted locally on NVME.  Network connection 10G SFP+ to Unifi SFP+ aggregation switch.  Media hosted on a separate dedicated file server running Microsoft Server 2022, single parity of spinning disks (no tiered storage or writeback cache), network connection 10G SFP+ to Unifi SFP+ aggregation switch.

At any given time, I may have up to 10 streams at one time.  Max I think I've ever seen is 15.  Have noticed on a rare occasion when 10+ streams, that streams are pausing every so often, even direct streaming.  Currently troubleshooting, & think it could potentially be a disk I/O issue on the 4K content.  Considering moving 4K content to a separate mirrored array to increase read speed & availability.

Has anyone made the transition to H.265 4K HDR10?  Any advice, bottlenecks, or lessons I can learn from?

I've had trouble finding documentation on how many 4K transcodes the UHD 630 can handle at any given time &/or how many H.265 streams it can handle....

Best-

Edited January 17, 2022 by darrenkdean

[darrenkdean 29]

Hardware Testing Update:

Performed a Transcoding vs. Direct Streaming, 4K HEVC, load test today.

Intel Quicksync had no issues with trandscoding 10x 4K HEVC streams to H.264, locally (all were separate files).  When switched to allow the 10x 4K HEVC to Direct Stream, we ran into pausing issues with all content, Direct & Transcoding Streams.  This leads us back to our initial conclusion that disk I/O on the file server may be the limiting factor with regards to 4K HEVC content when to many streams are direct steams.  Will re-attempt test using SSD's in mirror.

[RanmaCanada 354]

Quicksync will be more than enough, and cheaper than buying a video card at this point.  Yes 10-4k streams will seriously bottleneck any system that is using spinning rust.  It's possible if you move to a 12GB/s card you might get more throughput, or host them all on SSD's for big $$$.  The other thing I would recommend is gobs of more ram as you can set Emby up to transcode to ram instead of to the SSD (I believe).  For 4k HDR most people believe in the mantra "do not transcode 4k HDR", and as for finding documentation on quicksync encodes, there is very little.  There is a thread here in the hardware sub I believe where a few of us were posting our own results. 

[darrenkdean 29]

16 hours ago, RanmaCanada said:

Quicksync will be more than enough, and cheaper than buying a video card at this point.  Yes 10-4k streams will seriously bottleneck any system that is using spinning rust.  It's possible if you move to a 12GB/s card you might get more throughput, or host them all on SSD's for big $$$.  The other thing I would recommend is gobs of more ram as you can set Emby up to transcode to ram instead of to the SSD (I believe).  For 4k HDR most people believe in the mantra "do not transcode 4k HDR", and as for finding documentation on quicksync encodes, there is very little.  There is a thread here in the hardware sub I believe where a few of us were posting our own results. 

Cool.  Agreed that Quicksync will get the job done & that is in line with what we're seeing.  We do have a 12 GB/s card (LSI 9305-24i x8 lane).  About to blow away the storage array & go back to a tiered storage using mirrored SSD's in front of the Spinning Disks.  Quick question:  Any guidance for the spinning disk portion of the storage pool, mirror vs. single parity?  I'm not hurting by any stretch for storage space, just trying to head off any bottlenecks as best I can.  Right now, I'm thinking 2 SSD's (performance tier) with 8 Spinning disks in Raid 5/Single Parity.  Would end up with two identical storage pools.  Would mirrored be better for the spinning disks?

Best-

Darren

Edited January 18, 2022 by darrenkdean

[rbjtech 4312]

You need to monitor the system for where the bottleneck is - it is pointless throwing hardware at a problem unless you know what the actual issue is.

If you believe the issue is disk I/O - then do some reading up on RAID - ultimately you need to stripe the data across as many disks as possible.  Hint - RAID 5 is outdated and not advised for large storage arrays ..   Caching SSD's (tiered storage) are generally not going to help you with 100Gb+ 4K Remux's ..  At the end of the day -a 4K Remux from a bitrate perspective is 'huge' but from a disk I/O perspective - it's a walk in the park - 10-12 Mbytes/sec... which is nothing .. you just need to spead the load across disks and controllers and scale according to your needs.

The issue may be network related - NAS is actually fundamentally a very poor method for transferring data (lol) - convenient yes - but Direct Attached or SAN / Fibre Channel is far more efficient as it is designed to transfer data.  Again, monitor the packets with the likes of Wireshark - your 'pauses' could well be network related but you won't know unless you monitor it. 

Edited January 19, 2022 by rbjtech