Tiered storage

[pclausen 44]

I have a large collection of media living on 2 large RAID 60 arrays.  One is 145TB of usable storage with 24x 8TB drives.  The other one is 109TB of usable space with 24x 6TB drives.

Rather than keeping all those drives spinning all the time and the associated power cost, I have been looking at tiered storage.  In a perfect world, this is what I'd like:

Tier 1 - fast NVMe based with 12 2 TB NVMe drives.

Tier 2 - slow spinner storage with 24 14TB spinner drives.

The Emby metadata would always live on the Tier 1 storage and as new movies and TV shows come in, they would live on the Tier 1 storage for a few weeks until they have had a chance to be watched at least once.  Then they would roll off onto Tier 2.  Maybe a job would run once a week to move the old stuff to Tier 2?  And tier 2 would be spun down most of the time and only come online when the weekly job ran, or when I wanted to watch something that was on the tier 2 storage.

Are there any DYI solutions out there that supports this?  TrueNAS does not and I'm having a hard time getting an array to spin down on that solution.

Edited September 29, 2022 by pclausen

[Luke 42083]

@Baenwort may have some tips on this.

[Baenwort 118]

Yes, you are kinda fighting against TrueNAS when you want to do this.

It is possible to accomplish if you are up for the fight.

Setting up the transfer can be done via a Cron Job that moves the folder/file after it's date is greater than the time span you want.  A starting point would be reading these two:

https://forums.freebsd.org/threads/find-move-script.70094/ and https://serverfault.com/questions/505949/move-files-to-another-directory-which-are-older-than-a-date combined with the TrueNAS Cron area: https://www.truenas.com/docs/core/coretutorials/tasks/creatingcronjobs/

Make sure what ever time you set the Cron task to run on the TrueNAS side you have Emby perform a library scan a short time later. Unfortunately I don't have any way to trigger a library scan by the completion of the Cron Job so you'll have to estimate about how long you think the transfer will take.

 

The second part is harder. You're going to need to be pretty focused to remove disk access and checking events from the Tier 2 storage to get it to spin down. The best instructions I've used for this (I've since given in and leave my drives spinning since installing solar at home) are found at: https://www.truenas.com/community/resources/hdd-spindown-timer.122/ but if you want to see the history of this fight I would recommend https://www.truenas.com/community/threads/how-to-let-drives-spin-down.26314/page-7 

Emby can be set to run scheduled tasks all at the same time and since Emby keeps nearly all it's activity to the Jail drive except when you access data or perform scans this is certainly possible to do. Just make sure you don't store anything extra in the media folders (by default Emby stores nearly everything in the Jail but there are many options to store images and other items with the media files. Make sure you haven't done this).

However, you will be fighting this battle constantly. 

Edited October 1, 2022 by Baenwort

[pclausen 44]

Thank you for the great ideas and suggestions.  I'll definitely play with this when I get a chance!