Speeding up Emby Access

[crbdrbonline 2]

I have been using Windows Storage Spaces (Parity) with 7 4TB drives for movie storage and recording live TV in Emby since it came out years ago. Accessing movies through various local clients (Emby Theater PC, AppleTV4K, Roku, LG) is often painfully slow, and I have decided it is the parity pool that is killing me. It’s as if a few drives are going to sleep leaving some movies accessible while others are not. But no matter, I want to rebuild my Windows Emby server and move away from Storage Spaces. 
 

1. Is there really any benefit to pooling with something like DrivePool? I am seriously considering going back old school to single drives for simplicity sake. Back up each drive 1:1 and be done with it. Any shortcomings to doing this? Am I missing a big gotcha?
 

2. How do people move/copy massive amounts of data safely? I used to use SyncToy but I’m sure Emby users have better options. I need a GUI and I am willing to pay for it. An incremental backup feature would also be nice. 
 

3. Are there any other best practices to make sure Emby is quick to respond to the local clients? Using an SSD cache drive or...

[rbjtech 5284]

I went back to old school a couple of years ago and have never looked back.  For media library storage which is basically static day to day, then having redundancy is not only unnecessary but adds complexity making the entire thing more likely to fail.  With jbod you simply replace/upgrade the disk you need.  Job done.  The key is to use a unc share for each disk,  then map multiple shares as required for each library in emby.  (Don"t use drive letters).  Then you can move data around as and when you like and emby continues to work without any rebuilds.      I do use pooling for my 1:1 backup disks,  making use of multiple smaller drives to cleanly backup 1 larger drive.  Ie I backup an 8GB drive with two pooled 4GB disks.  No redundancy required as it's a backup anyway.     For 'live's data,  then yes you need a mirror of some sort but for media,  I've never personally seen a need to keep it mirrored as long as you have an OFFLINE and tested backup.

Re emby response time .. then yes put emby and imo all cache and local metadata on the ssd.  Unless you NEED to keep meta data with the media file then put it on the fastest i/o you have. Ssd cost per GB are cheap these days,  so this is no longer a problem.   It also does away with any complications on spinning up the disks unnecessarily.  The argument for ssd drive wearing is also now largely irrelevant as by the time the drive wears,   available capacity would have substantially  increased per dollar anyway.

Edited December 3, 2020 by rbjtech

[CBers 7450]

I have used DriveBender with Emby (MB2/MB3) for the last 9 years without any issues. 

 

[Sammy 790]

Drivepool is working fine for me two years in..

[crbdrbonline 2]

7 hours ago, CBers said:

I have used DriveBender with Emby (MB2/MB3) for the last 9 years without any issues. 

 

For use with Emby, what is actually the purpose that it serves for you personally? Emby will pull together movies across folders/drives on its own. You can create a “Library” in Windows to see all of your movies in one folder. And backups seem like they would just be easier with single unpooled drives. A lot of people pool their drives so I am just trying to understand what I will miss out on by using single drives that are backed up. 

Edited December 3, 2020 by crbdrbonline

[CBers 7450]

42 minutes ago, crbdrbonline said:

For use with Emby, what is actually the purpose that it serves for you personally?

I use DriveBender for more than just Emby. 

 

[PenkethBoy 2068]

why dont you try it out - DP and i assume DB have a free month trial

Emby benefits from DP/DB - but i suspect every user has his/her own reasons - and "Just" emby is not one of them

e.g. single multiple disks are a nightmare to manage and pooling solves a lot of this - including a single backup to manage

for me going back to single disks would be like going back to the 1990/2000's - but my minions are now long gone so i dont want the work!

 

[crbdrbonline 2]

46 minutes ago, CBers said:

I use DriveBender for more than just Emby. 

 

That’s why I was asking specifically about its use with Emby. I assumed people use it for more than that. Just wondered if there was something I was missing other than the fact that it creates one location for media. 

[sooty234 266]

I also use drivepool. The benefits will vary based on how you configure it. Duplication, read striping, SSD cache drives etc.

[sooty234 266]

And you can also create multiple pools, each with a different configuration. 

[CBers 7450]

1 hour ago, crbdrbonline said:

That’s why I was asking specifically about its use with Emby. I assumed people use it for more than that. Just wondered if there was something I was missing other than the fact that it creates one location for media. 

You give the pool a mountpoint (drive letter) then you create folders that you then use as normal. 

Not necessarily only one location for media. 

 

[Carlo 4561]

I use Drivepool.  It's just a hidden folder that all media under become part of the pool.  From outside of DP you can move things into or out off this directly quite quickly.  If you connect the drive to a computer not running DP software you can still access the drive just fine (but not the pool).

I do not use any of the advanced features like duplication as I only use it for combining drives.  So instead of having 60 mount points for my movies I can just mount /EmbyServer/Movies

Same for Plex or an FTP Server.

Drivepool doesn't care about the size of your discs either as there is no striping.  You can have 4, 6, 8, 10,12 TB all mixed together, no issue.

The only thing I wished it could do is bring network locations into the pool. I've got several small NASes like 8, 12 & 24 TB that would be nice to have "combined".

[PenkethBoy 2068]

OT

Carlo have you looked at the cloud drive option - to see if you could use that to pull the info from the nas's - as you can add a clouddrive pool to a normal pool

not tried but worth a look

 

[Carlo 4561]

I previously tried it (been a couple of years) and didn't like it.  I can't remember why however.

To be honest I don't remember how it even worked.  I might try this again with a test pool. Worst case is I'll remember why I didn't do it.

Thanks for reminding me about this feature.

[ebr 16184]

21 hours ago, crbdrbonline said:

Just wondered if there was something I was missing other than the fact that it creates one location for media. 

There could be other benefits but that one is huge in the case of administering several systems that need to point to that location and an ever-expanding or changing set of actual drives.  The pooling makes adding, removing, swapping drives a zero-admin task in all of those systems.

[dapper 13]

On 12/3/2020 at 12:43 AM, crbdrbonline said:

I have been using Windows Storage Spaces (Parity) with 7 4TB drives for movie storage and recording live TV in Emby since it came out years ago. Accessing movies through various local clients (Emby Theater PC, AppleTV4K, Roku, LG) is often painfully slow, and I have decided it is the parity pool that is killing me. It’s as if a few drives are going to sleep leaving some movies accessible while others are not. But no matter, I want to rebuild my Windows Emby server and move away from Storage Spaces. 
 

1. Is there really any benefit to pooling with something like DrivePool? I am seriously considering going back old school to single drives for simplicity sake. Back up each drive 1:1 and be done with it. Any shortcomings to doing this? Am I missing a big gotcha?
 

2. How do people move/copy massive amounts of data safely? I used to use SyncToy but I’m sure Emby users have better options. I need a GUI and I am willing to pay for it. An incremental backup feature would also be nice. 
 

3. Are there any other best practices to make sure Emby is quick to respond to the local clients? Using an SSD cache drive or...

It's weird that Parity is giving you problems for accessing the data -- it should be at least 3x faster than a single drive for reads.  (Writes on HDDs in Storage Spaces Parity are notoriously slow though)

I assume you store metadata / images in the media folders next to the media files?  This kills performance (really on any HDD setup) bc of all the access requests for tiny files spread across a bunch of disks competing in the queue with the large, sequential media files.

The most important thing for performance is to have the Program, Cache, Metadata, Transcoding, and Logs Folders all on SSDs

Also, obviously if your server runs other things, that can cause problems -- a very common performance drain is torrenting on the same HDDs as your media 

 

2.

I run Storage Spaces (Win Server 2019), and when I make a new Virtual Disk, I just use basic Windows File Explorer to "Move" from the old Virtual Disk to the new one

[crbdrbonline 2]

1 hour ago, dapper said:

It's weird that Parity is giving you problems for accessing the data -- it should be at least 3x faster than a single drive for reads.  (Writes on HDDs in Storage Spaces Parity are notoriously slow though)

I assume you store metadata / images in the media folders next to the media files?  This kills performance (really on any HDD setup) bc of all the access requests for tiny files spread across a bunch of disks competing in the queue with the large, sequential media files.

The most important thing for performance is to have the Program, Cache, Metadata, Transcoding, and Logs Folders all on SSDs

Also, obviously if your server runs other things, that can cause problems -- a very common performance drain is torrenting on the same HDDs as your media 

 

2.

I run Storage Spaces (Win Server 2019), and when I make a new Virtual Disk, I just use basic Windows File Explorer to "Move" from the old Virtual Disk to the new one

It is odd but not something I was going to dig into deeper since my intention was to go back to single discs (or DrivePool). My metadata is in the individual media folders (holdover from the Media Browser days). So I will look into getting that onto an SSD. Is there a work instruction you could link for that? I’ve seen it mentioned searching here (why I even asked), but have not see a workflow or anything yet. 

However, as another piece of information related to access problems, I can log into my server directly, open my large movie folder that holds all of my movies within the pool, open a banner image or fanart , and File Explorer will often hang and stop responding for a minute or more. It will eventually open after waiting

In fact, when attempting to play a movie through Emby on a Roku or AppleTV, I will often just get the spinning circle. If, while the client is just spinning (trying to play the movie), I go to the server directly, find the movie folder on File Explorer and open banner art or something, when the banner art finally opens from File Explorer on the server, the Roku will immediately stop spinning and start playing the movie. This is completely repeatable across various clients located in different parts of the house. 

Edited December 4, 2020 by crbdrbonline

[PenkethBoy 2068]

sounds like a HW fault (disk/controller) and/or SS is dying 

[dapper 13]

13 minutes ago, crbdrbonline said:

It is odd but not something I was going to dig into deeper since my intention was to go back to single discs (or DrivePool). My metadata is in the individual media folders (holdover from the Media Browser days). So I will look into getting that onto an SSD. Is there a work instruction you could link for that? I’ve seen it mentioned searching here (why I even asked), but have not see a workflow or anything yet. 

However, as another piece of information related to access problems, I can log into my server directly, open my large movie folder that holds all of my movies within the pool, open a banner image or fanart , and File Explorer will often hang and stop responding for a minute or more. It will eventually open after waiting

In fact, when attempting to play a movie through Emby on a Roku or AppleTV, I will often just get the spinning circle. If, while the client is just spinning (trying to play the movie), I go to the server directly, find the movie folder on File Explorer and open banner art or something, when the banner art finally opens from File Explorer on the server, the Roku will immediately stop spinning and start playing the movie. This is completely repeatable across various clients located in different parts of the house. 

You probably have a failing disk and/or an issue with Storage Spaces (from old slow HDDs) -- do you monitor SMART data with something like CrystalDiskInfo?  Any new reallocated sectors or other errors?

 

Open Powershell as an Administrator (Win Key+X then A) and run this command:

Get-PhysicalDisk | Get-StorageReliabilityCounter | Sort-Object -Property DeviceId  | ft DeviceId,*Laten*,Power*,*Error* 

 

"FlushLatencyMax", "ReadLatencyMax", and "WriteLatencyMax" should all be under 7000 (and really closer to 1000) if everything is working correctly

 

[Gilgamesh_48 1240]

1 hour ago, dapper said:

The most important thing for performance is to have the Program, Cache, Metadata, Transcoding, and Logs Folders all on SSDs

I do NOT agree with this at all. SSDs do improve performance but not really enough, in my experience, to warrant the cost. I have a big local drive to run Emby (not in my pool) and a rather large library that is stored in my pool. I also store all metadata and artwork etc. alongside my media files. My access is pretty fast and I almost never really have to wait for anything.

It is important that the Emby database NOT be stored in the pool but that causes Emby to crash so it is unlikely to cause performance problems.

Transcoding should also be stored outside the pool.

My setup is a large "C :" drive for boot and actually running Emby and a pool of nearly 72tb where all my media and metadata is stored. All of the drives are regular hard drives and the "C :" drive is installed in the chassis of the server while all the other drives are connected via USB. The setup works great to serve media to any of my clients including Rokus, Shields and Fire devices.

I believe the real "trick" to good performance on your server is to have the server do no other regular tasks except those needed to support its job of serving. A server should NOT be you daily use computer. 

It is a little more expensive to have a computer dedicated to being a server but I think overburdening a computer by trying to shoehorn to many applications into its duties is the primary cause of slow or erratic behavior.

BTW: I have tried all the major and a few of the minor pooling software and I have found the best and most reliable to be StableBit's Drivepool. It simply works and requires little work to keep it going. I have been running Drivepool for over six years and only once have I lost any data and that was due to my stupidity of trying to rearrange my drives on a shelf without powering down my system. My attempt to save time cost me time and data.

In computers in both hardware and software it is always good to remember the idiom: "Penny wise and pound foolish" as something to avoid.

Edited December 4, 2020 by Gilgamesh_48

[sooty234 266]

Each person will have varying experiences. Networking hardware, networking config, hard drives, bus, CPU etc. All of these things will factor. The access/seek time of an NVMe M.2 is significantly different to a 5400 rpm desktop HDD.

[Carlo 4561]

15 minutes ago, Gilgamesh_48 said:

I do NOT agree with this at all. SSDs do improve performance but not really enough, in my experience, to warrant the cost.

I sure do.  SSD make all the difference in the world on a larger system and cut way down the IO that normal drives have.  Besides SSD are cheap these days.

Putting your cache and transcoding folders on SSD can really speed things up.  If you use TV then recording to them and post processing after the fact really helps.

[dapper 13]

28 minutes ago, Gilgamesh_48 said:

I do NOT agree with this at all. SSDs do improve performance but not really enough, in my experience, to warrant the cost.

You can get a premium 500gb Samsung 970 Evo NVMe for $60 on Amazon right now (and that's overkill for most people) -- if you're running 70+TB of spinning disks, $100 of SSDs is practically negligible for the huge benefits

[crbdrbonline 2]

My boot drive is an M.2 drive. About how big of a drive do I need for Emby metadata? Assuming around 30TB of movies. 

[pwhodges 2012]

I have a bit under 10TB of media, and the 500GB SSD boot drive where Emby is installed and keeps the metadata is less than 20% full - and much of that is the OS, of course.

Paul