Jump to content

Preemptively cache images


AdrianW
Go to solution Solved by Luke,

Recommended Posts

rbjtech
16 minutes ago, nmkaufman said:

Unraid is set up to (optionally) utilize a SSD cache/landing-zone. New files hit the SSD landing-zone, and are periodically moved to mechanical storage.

1.) In step one, I copied the metadata files (in their original folder/sub-folder hierarchy) to a temporary location (my downloads folder), allowing me to review their contents before migrating them back to my media folders. I copied (rather than moved) the files, so they would be re-created on the SSD.

2.) I migrated the metadata back to their original media folders (but now they're on the SSD). I copied, again (rather than moved) simply so I could re-use my same script. Linux cp and mv commands use slightly different syntax, so I didn't want to add any additional complexity. I used the -l (lower case "L") flag, so the files would simply be hard-linked, rather than actually copied again, so it was just as fast as a move.

Unraid does allow you to address the physical disks, directly, so you could potentially considate the metadata directly by moving the files from /mnt/disk1/, /mnt/disk2/ etc. to /mnt/cache/.. but this would have to be done for each mechnical disk. My copying method just creates new copies of all the files on the SSD, and deletes the originals.

Believe me, if this all sounds daunting it's because it is.

I'm pretty new to linux, and running scripts and batch files outside a VM or Docker container still gives me anxiety.

I'm happy to report that this was successful, however. I'm now able to search and browse my library, and not even one of my mechanical disks spins up until I play a file.

Gray circle means sleeping. They turn green when they wake up:

sleeping disks.png

I've been doing this for a while using Stablebit DrivePool as it has the capability out the box (with a little config) - it too uses a SSD 'landing zone' and using that in combination with File Placement rules (where the non-video files stay on the SSD) - all my non video-file activity (excl MKV etc) stay exclusively on the SSD even though they 'appear' to be on the HDD mount.    Emby cache is already on the nvme anyway and temp/transcoding files go to a RAM drive.

tbh I don't use it to power down the HDD's - I use it for performance reasons.

Edited by rbjtech
  • Like 1
Link to comment
Share on other sites

Q-Droid
15 minutes ago, nmkaufman said:

Unraid is set up to (optionally) utilize a SSD cache/landing-zone. New files hit the SSD landing-zone, and are periodically moved to mechanical storage.

1.) In step one, I copied the metadata files (in their original folder/sub-folder hierarchy) to a temporary location (my downloads folder), allowing me to review their contents before migrating them back to my media folders. I copied (rather than moved) the files, so they would be re-created on the SSD.

2.) I migrated the metadata back to their original media folders (but now they're on the SSD). I copied, again (rather than moved) simply so I could re-use my same script. Linux cp and mv commands use slightly different syntax, so I didn't want to add any additional complexity. I used the -l (lower case "L") flag, so the files would simply be hard-linked, rather than actually copied again, so it was just as fast as a move.

Unraid does allow you to address the physical disks, directly, so you could potentially considate the metadata directly by moving the files from /mnt/disk1/, /mnt/disk2/ etc. to /mnt/cache/.. but this would have to be done for each mechnical disk. My copying method just creates new copies of all the files on the SSD, and deletes the originals.

Believe me, if this all sounds daunting it's because it is.

I'm pretty new to linux, and running scripts and batch files outside a VM or Docker container still gives me anxiety.

I'm happy to report that this was successful, however. I'm now able to search and browse my library, and not even one of my mechanical disks spins up until I play a file.

 

Okay, got it. This works in Unraid as long as user shares are referenced which make use of the cache so the files end up in the cache/landing zone. Using -l  in this case is effectively a mv (rename) when followed up with delete as long as Unraid can interpret this correctly, ie: cache to cache copy (same fs). To be clear, this would be specific to Unraid and how it manages the cache disk/pools. Other storage options might be able to do something similar.

 

Link to comment
Share on other sites

nmkaufman
1 hour ago, rbjtech said:

I've been doing this for a while using Stablebit DrivePool as it has the capability out the box (with a little config) - it too uses a SSD 'landing zone' and using that in combination with File Placement rules (where the non-video files stay on the SSD) - all my non video-file activity (excl MKV etc) stay exclusively on the SSD even though they 'appear' to be on the HDD mount.    Emby cache is already on the nvme anyway and temp/transcoding files go to a RAM drive.

tbh I don't use it to power down the HDD's - I use it for performance reasons.

I was a DrivePool (& formerly DriveBender) man for years. Great software. As my HDD's age, though, I started to feel anxious about failures.

I had an off-site backup, and  did incremental backups to an exernal HDD (even tried SnapRAID for a while,) but I love the real-time Parity you get with UnRAID.

There has certainly been a learning curve involved with switching to Unraid/Linux, but I think the added safety was worth the move.

Edited by nmkaufman
  • Like 1
Link to comment
Share on other sites

  • 4 weeks later...
  • Solution

This is coming in Emby Server 4.8:

 

  • Like 4
Link to comment
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now
×
×
  • Create New...