Drive Pooling?

[casminkey 15]

I've been using Flexraid for many years and have been running in to some minor issues here and there. Upon investigating, I realized that the developer has apparently gone MIA back in 2019. So looks like I need to find a new solution to pool all of my hard drives in to one logical drive. What's everyone using these days?

I scoured the topics and found StableBit DrivePool which looks nice and trying that out for now. I don't care about the duplication options if a drive fails. I have an cloud backup service that everything is backed up to.

 

Thanks!

[Happy2Play 8282]

Been using StableBit DrivePool and Scanner for years without issue.  

You also have DriveBender also.

[ebr 14913]

I use DrivePool as well.

[TeamB 2353]

2 hours ago, Happy2Play said:

Been using StableBit DrivePool and Scanner for years without issue. 

+1 for DrivePool and Scanner

I also use SnapRaid.

I have a combination of media files and work files on my JBOD, backed up using RnapRaid, it has worked very well over the last 8 or so years, saved me from total drive loss, data corruption, and disk bit degradation (bad sectors popping up) a few times.

[rbjtech 4270]

+1 for DrivePool + Scanner here as well (+ CloudDrive, but I don't use that much)

I like the ability to actually use the raw NTFS drives without the software in a worst case scenerio as DrivePool effectively just sits in the middle of standard I/O Ops.

I also like the core utilities such as SSD Optimizer and File Placement that means you can put files on SSD's to improve performance - effectively storage tiering.

[Jammi3D 8]

I've been looking at StableBit for a while, I've got about 12 drives in my server varying from 6TB to 18TB and most are 90% full. (approx 120TB)

How would I go about setting up a drivepool for the drives with data on as I see all drives will need to be formatted.

[rbjtech 4270]

52 minutes ago, Jammi3D said:

How would I go about setting up a drivepool for the drives with data on as I see all drives will need to be formatted.

No need to format drives.

The beauty of Stablebit drive pool is it will work on existing drives - as essentially the 'pool' just sits in hidden NTFS folders.

However with the drives at 90% full - then you are not going to have a lot of wiggle room.

I would run this past Stablebit for best advice but in summary -

The official method would be to create a pool on the remaining 10% drive space - and then MOVE files into the pool - as you move the files, the pool will actually remain the same size despite you adding files to it because you are creating 'free' space on the moved file once it has been added to the pool.   If you are clever with your setup - then you can actually configure it to just 'move' the files on the same disk.   This 'may' be extremely fast - as it's not moving the data at all - it's just re-pointing the data in the pool.     Obviously if you COPY the files, then you are going to run out of disk space.

With all of this - ensure you have a working backup.  With 120TB, this is easier said than done I agree, but I strongly suggest a 'verify' operation before you begin a major file system re-organisation such as this (same with ANY solution, not just Stablebit).

[Jammi3D 8]

@rbjtechthank you for the info.
Thought I had read each disk has to be formatted before adding to the pool, I will probably upgrade the drives that are near full before moving files about so your method sounds easiest.
Tbf my server could do with some work as the motherboard is kinda on it's last legs. AM4 boards are silly money now though and my 3400G seems a bit under powered on some tasks now.

It will be 1 disk at a time for moving files but I'm not really bothered about losing any files on the drives that will be added to the pool as they are easily replaceable. Important stuff is backed up regular to multiple locations.

1 question on using NVME drives as cache drives, I have Emby to constantly scan for new content if using Cache drives will Emby see new files straight away or will it wait for StableBit to move the files?

[ebr 14913]

2 minutes ago, Jammi3D said:

Thought I had read each disk has to be formatted before adding to the pool

That's how unraid worked and is why I moved off of it to DrivePool.

[rbjtech 4270]

1 hour ago, Jammi3D said:

@rbjtechthank you for the info.
Thought I had read each disk has to be formatted before adding to the pool, I will probably upgrade the drives that are near full before moving files about so your method sounds easiest.
 

I have done this a few times now - you definitely do not have to format the drive - as long as the disk appears in the 'Add to Pool' list - then you can add it

1 hour ago, Jammi3D said:

1 question on using NVME drives as cache drives, I have Emby to constantly scan for new content if using Cache drives will Emby see new files straight away or will it wait for StableBit to move the files?

I have my 'pool' using an NVME drive also as a 'cache' (it's called File Placement in DrivePool) - and files then get moved to HDD as part of the 'balancing' schedule.  To emby, this makes no difference and thus it 'sees' the file instantly regardless of where they physically reside.

To note, I put the Emby cache on an NVME outside the pool anyway - but as per the above, this could be the same physical drive - just using a normal folder. 

Edited March 29, 2023 by rbjtech

[Gilgamesh_48 943]

41 minutes ago, Jammi3D said:

1 question on using NVME drives as cache drives, I have Emby to constantly scan for new content if using Cache drives will Emby see new files straight away or will it wait for StableBit to move the files?

The "drive" that DrivePool creates behaves just like any other drive and, for me at least, real time monitoring works just fine.

BTW: The way DrivePool creates its pool is that each physical drive gets a (hidden I think) directory called something like "PoolPart.159ed417-db72-4217-bbee-3a3b1a6d79f1"
Any file that is on the drive but not in that directory is not in the pool. And any file that is in that directory is on the pooled drive. That means that once you add the drive to the pool any file on that drive that you move directly into that directory is in the pool right away. That makes moving files very fast and easy.

I have a pool of about 70 tb across 12 drives and everything works well. I use "file duplication" for redundancy for my main movies and TV shows that I cannot easily recreate and, so far in over 8 years, I have lost 3 drives in the pool but I have never lost a single hard to recreate file. I do physically backup many files but, so far, I have not had to use the backup due to any failure in the pool. I did use it once but that was because I decided to do something stupid. If you keep in mind that "redundancy" is not a backup then the "file duplication" feature is very valuable and saves a lot of time.