Completed Preemptively cache images

[rbjtech 5000]

16 minutes ago, nmkaufman said:

Unraid is set up to (optionally) utilize a SSD cache/landing-zone. New files hit the SSD landing-zone, and are periodically moved to mechanical storage.

1.) In step one, I copied the metadata files (in their original folder/sub-folder hierarchy) to a temporary location (my downloads folder), allowing me to review their contents before migrating them back to my media folders. I copied (rather than moved) the files, so they would be re-created on the SSD.

2.) I migrated the metadata back to their original media folders (but now they're on the SSD). I copied, again (rather than moved) simply so I could re-use my same script. Linux cp and mv commands use slightly different syntax, so I didn't want to add any additional complexity. I used the -l (lower case "L") flag, so the files would simply be hard-linked, rather than actually copied again, so it was just as fast as a move.

Unraid does allow you to address the physical disks, directly, so you could potentially considate the metadata directly by moving the files from /mnt/disk1/, /mnt/disk2/ etc. to /mnt/cache/.. but this would have to be done for each mechnical disk. My copying method just creates new copies of all the files on the SSD, and deletes the originals.

Believe me, if this all sounds daunting it's because it is.

I'm pretty new to linux, and running scripts and batch files outside a VM or Docker container still gives me anxiety.

I'm happy to report that this was successful, however. I'm now able to search and browse my library, and not even one of my mechanical disks spins up until I play a file.

Gray circle means sleeping. They turn green when they wake up:

I've been doing this for a while using Stablebit DrivePool as it has the capability out the box (with a little config) - it too uses a SSD 'landing zone' and using that in combination with File Placement rules (where the non-video files stay on the SSD) - all my non video-file activity (excl MKV etc) stay exclusively on the SSD even though they 'appear' to be on the HDD mount.    Emby cache is already on the nvme anyway and temp/transcoding files go to a RAM drive.

tbh I don't use it to power down the HDD's - I use it for performance reasons.

Edited December 14, 2023 by rbjtech

[Q-Droid 881]

15 minutes ago, nmkaufman said:

Unraid is set up to (optionally) utilize a SSD cache/landing-zone. New files hit the SSD landing-zone, and are periodically moved to mechanical storage.

1.) In step one, I copied the metadata files (in their original folder/sub-folder hierarchy) to a temporary location (my downloads folder), allowing me to review their contents before migrating them back to my media folders. I copied (rather than moved) the files, so they would be re-created on the SSD.

2.) I migrated the metadata back to their original media folders (but now they're on the SSD). I copied, again (rather than moved) simply so I could re-use my same script. Linux cp and mv commands use slightly different syntax, so I didn't want to add any additional complexity. I used the -l (lower case "L") flag, so the files would simply be hard-linked, rather than actually copied again, so it was just as fast as a move.

Unraid does allow you to address the physical disks, directly, so you could potentially considate the metadata directly by moving the files from /mnt/disk1/, /mnt/disk2/ etc. to /mnt/cache/.. but this would have to be done for each mechnical disk. My copying method just creates new copies of all the files on the SSD, and deletes the originals.

Believe me, if this all sounds daunting it's because it is.

I'm pretty new to linux, and running scripts and batch files outside a VM or Docker container still gives me anxiety.

I'm happy to report that this was successful, however. I'm now able to search and browse my library, and not even one of my mechanical disks spins up until I play a file.

 

Okay, got it. This works in Unraid as long as user shares are referenced which make use of the cache so the files end up in the cache/landing zone. Using -l  in this case is effectively a mv (rename) when followed up with delete as long as Unraid can interpret this correctly, ie: cache to cache copy (same fs). To be clear, this would be specific to Unraid and how it manages the cache disk/pools. Other storage options might be able to do something similar.

 

[nmkaufman 49]

1 hour ago, rbjtech said:

I've been doing this for a while using Stablebit DrivePool as it has the capability out the box (with a little config) - it too uses a SSD 'landing zone' and using that in combination with File Placement rules (where the non-video files stay on the SSD) - all my non video-file activity (excl MKV etc) stay exclusively on the SSD even though they 'appear' to be on the HDD mount.    Emby cache is already on the nvme anyway and temp/transcoding files go to a RAM drive.

tbh I don't use it to power down the HDD's - I use it for performance reasons.

I was a DrivePool (& formerly DriveBender) man for years. Great software. As my HDD's age, though, I started to feel anxious about failures.

I had an off-site backup, and  did incremental backups to an exernal HDD (even tried SnapRAID for a while,) but I love the real-time Parity you get with UnRAID.

There has certainly been a learning curve involved with switching to Unraid/Linux, but I think the added safety was worth the move.

Edited December 14, 2023 by nmkaufman

[Luke 40120]

This is coming in Emby Server 4.8:

 

[Fender1978 8]

On 1/24/2023 at 2:33 PM, rbjtech said:

Finally got around to doing this - and it's working perfectly.

In the drive pool 'File Placement' rules - just specify to place all non video data onto a fast nvme drive - ie  *.jpg / *.png / *.nfo / *.bif / *.srt etc

Drivepool does all the 'linking' - so you'll still 'see' all the above files on your media drives, but they are actually on the nvme drive - thus much faster to load.

Don't forget to backup the nvme drive though (if you don't back the 'pool' as an entity), as the 'real' files are now on there.  

Hi,

I really want to implement this but I have a few of questions.

1. What is the best way to backup the files on this nvme drive as a mirror would be ideal but drivepool wouldn't support this.

2. Is that nvme drive part of the pool?

3. Will that nvme drive accidentally act as a landing drive for all new files?

Thanks.

 

[Luke 40120]

1 hour ago, Fender1978 said:

Hi,

I really want to implement this but I have a few of questions.

1. What is the best way to backup the files on this nvme drive as a mirror would be ideal but drivepool wouldn't support this.

2. Is that nvme drive part of the pool?

3. Will that nvme drive accidentally act as a landing drive for all new files?

Thanks.

 

Just fyi the main feature discussed in this topic is already implemented in Emby server 4.8+. You just need to enable it in the settings for each library.

[rbjtech 5000]

16 hours ago, Luke said:

Just fyi the main feature discussed in this topic is already implemented in Emby server 4.8+. You just need to enable it in the settings for each library.

The 'solution' is only useful to those that store their media on a NAS (along with the metadata) - and even then, you cannot specify WHERE you want that cache to be held.

So in the case of wanting to use a fast NVME drive that is not your emby install drive - it's cannot be used - as it's fixed to the emby metadata drive (for some reason).

If the option allowed you to specify the drive - then I would probably use it myself to simplify the DrivePool configuration.

18 hours ago, Fender1978 said:

1. What is the best way to backup the files on this nvme drive as a mirror would be ideal but drivepool wouldn't support this.

Assuming you are backing up the media drive anyway - just make sure you specify to also backup the metadata - it will be pulled from the nvme drive.

18 hours ago, Fender1978 said:

2. Is that nvme drive part of the pool?

Yes - it has to be - but you specify in DrivePool that it's a 'SSD' drive and must be used as a cache.  Remember being DrivePool - you can add a used drive and it will just dynamically use what space you have left.  ie you don't need a 'blank' drive to add into an existing Pool.  This isn't RAID ..  

18 hours ago, Fender1978 said:

3. Will that nvme drive accidentally act as a landing drive for all new files?

Yes, it's designed that way - so you get ultra fast copying etc as it always gets copied to the NVME drive (including media).   DrivePool then moves it to the HDD's when the system is not under load or on the schedule you specify (balancing).    It greatly reduces 'upload and move file I/O time' for media.   The files you specify in the rules (the metadata), ultimately stay on there, hence being a faster overall HDD cache, but this applies to ALL files - not just those for emby.

Edited August 3, 2024 by rbjtech

[C.S. 76]

6 hours ago, rbjtech said:

Yes - it has to be - but you specify in DrivePool that it's a 'SSD' drive and must be used as a cache.

I'm not sure about this. I think it's better to add the nvme as a regular drive, then use file placement rules to put the image file types on there while putting video files over on the metals. Otherwise, if it's a cache drive, DP might try to move everything off the nvme and onto the metals. You can always use another ssd for cache if you want a cache.

I've been using DP for ten years at least, so maybe it's better now, but I remember having a lot of trouble trying to keep files on the cache drive. (Eventually I stopped using a cache all together and I haven't really missed it.)

[rbjtech 5000]

15 hours ago, C.S. said:

I'm not sure about this. I think it's better to add the nvme as a regular drive, then use file placement rules to put the image file types on there while putting video files over on the metals. Otherwise, if it's a cache drive, DP might try to move everything off the nvme and onto the metals. You can always use another ssd for cache if you want a cache.

I've been using DP for ten years at least, so maybe it's better now, but I remember having a lot of trouble trying to keep files on the cache drive. (Eventually I stopped using a cache all together and I haven't really missed it.)

You need to do both - have the SSD Optimizer balancer AND the File Placement rules - as explained originally.

Then the files stay on the SSD - thus effectively cached - but you also get the bonus of all files being written to the Pool (aka 'landing zone') are written to the fast NVME first.   Once balanced, then the media files are moved back to the HDD's - transparent to you/emby of course.

 

 

[C.S. 76]

Yeah what I'm saying is I've had trouble trying to make that work because DP seems to want to move everything off the cache drive no matter what boxes are checked (on the main balancing config page) under "File placement settings". But I admit I haven't tried it for a while, so maybe they fixed it?

[rbjtech 5000]

16 hours ago, C.S. said:

Yeah what I'm saying is I've had trouble trying to make that work because DP seems to want to move everything off the cache drive no matter what boxes are checked (on the main balancing config page) under "File placement settings". But I admit I haven't tried it for a while, so maybe they fixed it?

You possibly have the balacing rules overwriting the placement rules ?

Untick the above - and then the placement rules take priority over the balacing rules ..

You can also easily check its working - just look in the hidden PoolPart on the cache drive/SSD - you will see all the file placement files - but not the media file itself as that is on the HDD.

Edited August 5, 2024 by rbjtech

[C.S. 76]

Yup got all that. Just saying it wasn't working for me, but that was several years ago, so maybe I would have a different experience now.

[rbjtech 5000]

1 minute ago, C.S. said:

Yup got all that. Just saying it wasn't working for me, but that was several years ago, so maybe I would have a different experience now.

It works for me 100% - so yes, maybe it has been fixed but I've personally never seen it 'broken'...