Jump to content

Drive pooling


Guest asrequested

Recommended Posts

CBers

I'm also wondering how you backup a pool of drive, as if the data is constantly being moved over, the backup needs to mirror it. Wouldn't that be a bandwidth hog?

You don't "backup a pool of drive", you backup the data within the pool, which is mounted as a drive letter, which in my case, is my X:\ drive.

 

Everything is then accessed from my X:\ drive, such as my TV, Movies, music and photos.

 

I "backup" files from my other Office PC to the pool using robocopy.

 

I also use Macrium Reflect to image my OS discs to a folder in my pool.

 

My pool is small, at only 13Tb in size, but it's only about 60% full.

Link to comment
Share on other sites

CharleyVarrick

Depending on your backup process are you backuping up data or drives. 

Simple enough question, answer is not so simple.

 

My present backup is 2 similar computers with same number of drive/capacity. Presently using FreeFileSync as a 1-way file mirror from A to B.

As I am nearing the limits of the present setup, my future backup would be computer A files to cloud... And sometime soon after computer B to cloud.

Exactly how I am still weighing how. 

Link to comment
Share on other sites

CharleyVarrick

You don't "backup a pool of drive", you backup the data within the pool, which is mounted as a drive letter, which in my case, is my X:\ drive.

 

Everything is then accessed from my X:\ drive, such as my TV, Movies, music and photos.

I am not using the right syntax.

Take 2:

Here's what I mean: suppose I have a pool drive, 4 hdd in there with my movies, the drive pooling makes it appear as a huge hdd. Not only would I add/remove/modify some of the data within this pool on a daily basis, but the drive pooling also does its balancing act of data between 4 hdd on the source side. My question is: would the destination copy try to mimick, bit for bit, the balancing of data happening on the source side over on the destination backup side.

Edited by jlr19
Link to comment
Share on other sites

mediacowboy

Okay, If your asking if copying to the pool would it balance as the copy happens then the answer is yes for Drive Bender. If your asking if you are backing up from the pool the backup software would see it as one drive and it would be bit for bit. Does this help?

 

I can show examples as it balance while copying as I am in the process of renaming and copying everything over to a new server.

Link to comment
Share on other sites

Happy2Play

You software shouldn't see any of the balancing across the 4 drives.  It should only see the large virtual drive (pool) ie drive z.

 

Drive Z contains drives a, b, c, and d.

  • Like 1
Link to comment
Share on other sites

CBers

I am in the process of renaming and copying everything over to a new server.

Not copying due to a DriveBender issue though?

 

I moved my DriveBender disks from my old PC to my new server and DB auto-doscovered the discs and created the pool. This took all of 15 minutes, including the disc swap :)

Link to comment
Share on other sites

CharleyVarrick

You software shouldn't see any of the balancing across the 4 drives.  It should only see the large virtual drive (pool) ie drive z.

 

Drive Z contains drives a, b, c, and d.

I think even though my bad phrasing, you seem to understand what I meant.

I think the reason is the drive pooling software is only active at the source, and the backup copy doesn't care which clusters the data is written to.

 

Happy for prez!

Cheers

Edited by jlr19
Link to comment
Share on other sites

mediacowboy

Not copying due to a DriveBender issue though?

 

I moved my DriveBender disks from my old PC to my new server and DB auto-doscovered the discs and created the pool. This took all of 15 minutes, including the disc swap :)

Correct. I am copying everything over manually because I plan to keep drive bender on both systems and have the new server copy any chances over to old server as a in house backup and WAF satisfaction. The reason for the rename is the old file structure I had series name - s##e## - episode name. I am changing to show name.s##e##.episode title. Also a lot of my movies didn't have the year on them so now that folder structure is Movie name (year)\movie name (year).mp4.
  • Like 1
Link to comment
Share on other sites

CharleyVarrick

now that folder structure is Movie name (year)\;movie name (year).mp4.

What's that in your file name?

Link to comment
Share on other sites

mediacowboy

What's that in your file name?

Sorry should have typed that better.

 

Current file structure :

D:\Media\Movies\Movie Name\Movie Name.mp4

 

New file structure;

D:\Media\Movies\Movie Name (Year)\Movie Name (Year).mp4

 

Reason for the change is I have a lot of movies that have the same name but made in different years. Look at movies for Dracula or Journey to the Center Of The Earth. I want it in both locations for ease of identification if I fat finger something and move the movie into the wrong directory or something.

Link to comment
Share on other sites

Swynol

i think your making it more complicated than it needs to be.

 

Rather than backing up a HDD, i backup a folder within my pool. 

 

For example the Pictures folder is in my pool but the data is spread across 12 HDDs. the backup backs up the Pictures folder in my pool to an external HDD. The backup software doesnt care that the data is spread across 12 drives because the pool presents it as if its on one HDD. 

 

I think i know what your saying about "what happens if i lose 1 drive and how do i know what data is on it". honestly you dont know what data is on it as anything written to the pool is balanced across all the drives. This is where Pool/Folder Duplication comes in handy. I have 2x Duplication on all my folders. this means all data is written to at least 2x HDDs. if one fails then you dont see any difference as the data is still available. Just swap out the dead drive and all is good again. The issue come if you lose 2 drives at the same time. 

 

If i lost 2 drives i would have to restore from a backup. What i would do is replace the 2 HDDs, now data will be missing but we dont know what data. I just restore the backup to my Pool drive when it asks to overwrite data i say no that way only the lost data gets re-written. 

  • Like 1
Link to comment
Share on other sites

CharleyVarrick

Thanks, it's making its way, I have a much clearer picture now than a week ago.

I think I make it look complicated because I am considering making 2 changes at once to my setup, their consequences are overlapping, in my mind at least.

 

1) Introducing Drive pooling, because it is a lot of work to add a drive, or upgrade a drive capacity. Each time I need to manually re-balance my movie collection spread to multiple drives.

I really like the simpleness of popping an extra drive in the pool when needed.

2) Change to my backup strategy, from present 2 "identical" computers, Server "A", which sync a 1-way mirror to Server "B" daily.

I am testing out online backup to eventually go to Server "A" to Cloud, soon followed by Server "A" and "B" to Cloud.

 

About Pool/Folder Duplication: as I have (at the moment) 4 drives of movies (70% full)), I would need 8 drives to achieve data written to at least 2x drives (?) 

 

On a slightly related topic: I was reading in a computer magazine that Seagate is aiming to offer a 20Tb hard drive by 2020.

They presently are at the 10/12Tb level and should introduce a 14 then a 16 within the next year or two.

Edited by jlr19
Link to comment
Share on other sites

Swynol

Drive pool has made my life easier. just adding a new drive to the pool takes seconds. also replacing an existing drive is easy. 

 

Yes you will need either bigger drives or more of them to duplicate your current data. Over the years i have gone from 2 HDDs to 12 HDDs using a Norco 4224 case and SAS cards with backplanes. I also have 2 - 3 HDDs stored away which i use as spares incase of failure.

 

My current backup is, 2x folder duplication on everything.

Pictures and documents and music are backed up to an external drive once a week

Pictures are backed up to google photos cloud - uploaded instantly when added to my pictures folder

Documents backed up to Google Drive - uploaded instantly when added to my docs folder

Music backed up to Google Music - uploaded instantly when added to my music folder

Important videos backed up to external drive once a week.

Any other setup or programs files are backed up once a month to external drive. Most configs etc are also stored in cloud.

OS Drive is cloned around once a month incase of failure. 

I take a backup of my movies collection once every 6 - 12 months onto old hard drives which are stored in a fire save.

 

The external drives mentioned above (i have 2 of them) get rotated from being offsite and live backup every 2 - 4 weeks or more like when i remember. 

 

 

 

With the changes you want to do above, the simplest option is to get 2x drive pool licences, have one pool setup on Server A and an identical Pool setup on Server B. And then do the same backup as you do now. i.e. Pool A --> Pool B. 

If a HDD fails on Server A then the files will be missing from Pool A. You can then just copy the files back from Pool B to Pool A. 

 

Another option is to put all 8 HDDs into Server A and turn on Pool Duplication so all files are on at least 2x HDDs.   - then maybe over time add a few HDDs to server B and then do a backup of Pool A on Server A to Server B.

 

Seagate as well as HGST seem to be leading the way with the HDD size atm. their 10TB ironwolf and ultrastar archive drives are a decent jump over what others are offering and i can quite well imagine 20TB or bigger drives by 2020. 

 

2013 saw the first 6TB HDD, only a few years later in 2015 we saw the first 10TB HDD. that's almost double the size in 2-3 years. go back to 2010 which saw the first 3TB HDD. so again double the size in 3 years.

  • Like 2
Link to comment
Share on other sites

mediacowboy

 

 

Drive pool has made my life easier. just adding a new drive to the pool takes seconds. also replacing an existing drive is easy.

 

Yes you will need either bigger drives or more of them to duplicate your current data. Over the years i have gone from 2 HDDs to 12 HDDs using a Norco 4224 case and SAS cards with backplanes. I also have 2 - 3 HDDs stored away which i use as spares incase of failure.

 

My current backup is, 2x folder duplication on everything.

Pictures and documents and music are backed up to an external drive once a week

Pictures are backed up to google photos cloud - uploaded instantly when added to my pictures folder

Documents backed up to Google Drive - uploaded instantly when added to my docs folder

Music backed up to Google Music - uploaded instantly when added to my music folder

Important videos backed up to external drive once a week.

Any other setup or programs files are backed up once a month to external drive. Most configs etc are also stored in cloud.

OS Drive is cloned around once a month incase of failure.

I take a backup of my movies collection once every 6 - 12 months onto old hard drives which are stored in a fire save.

 

The external drives mentioned above (i have 2 of them) get rotated from being offsite and live backup every 2 - 4 weeks or more like when i remember.

 

 

 

With the changes you want to do above, the simplest option is to get 2x drive pool licences, have one pool setup on Server A and an identical Pool setup on Server B. And then do the same backup as you do now. i.e. Pool A --> Pool B.

If a HDD fails on Server A then the files will be missing from Pool A. You can then just copy the files back from Pool B to Pool A.

This is what I am planing once all the files have been renamed and copied over to my new server. I will then wipe the current server. Reinstall OS and everything since it become my gaming/everyday rig. Reinstall DB and have a one way sync to it. Doing this gives me a in house back up. As well as a server a friend has at his house for off site.

Link to comment
Share on other sites

CharleyVarrick

 

Seagate as well as HGST seem to be leading the way with the HDD size atm. their 10TB ironwolf and ultrastar archive drives are a decent jump over what others are offering and i can quite well imagine 20TB or bigger drives by 2020. 

 

2013 saw the first 6TB HDD, only a few years later in 2015 we saw the first 10TB HDD. that's almost double the size in 2-3 years. go back to 2010 which saw the first 3TB HDD. so again double the size in 3 years.

 

When I first met my to be wife in 2001, she had a 3k$ computer (how cool was that!) with a 20gb hard drive inside. That seemed like plenty back then.

First thing I did (well, maybe 2nd or 3rd... ;) ) was upgrade the OS from w98 2nd ed. to Windows XP. Ah, the promise of better things to come, looking at XP setup screens.

Edited by jlr19
Link to comment
Share on other sites

casminkey

Interesting. Does it let you put a pool in RAID? I would imagine that's a lot of processing.

 

I have been using Flexraid also for years and do this quite frequently. I have x number of data drives and 1 parity drive to use in the raid. When I'm low in free space I'll take the parity drive and turn it into a data drive to use as a disk pool. When I finally get around to getting another drive, I'll add that back as the parity drive for a raid. The only problem is that the pool/raid is offline while the parity drive is processed. I'll usually do this overnight and it's done in the morning.

Link to comment
Share on other sites

aptalca

To throw my hat in the ring, I use unraid that has a pool of 7 drives (1TB-3TB) plus two parity drives so it can tolerate two drive failures.

 

I also have a cache pool (btrfs mirrored) of 3 ssd drives. Most things get written to the cache pool first and then moved over to the array nightly so I get the ssd speed during the initial transfer

Link to comment
Share on other sites

Guest asrequested

Can I have a moment of silence, please? I lost a drive, today. It went pretty quickly. I don't think it suffered. R.I.P.

  • Like 1
Link to comment
Share on other sites

mediacowboy

Can I have a moment of silence, please? I lost a drive, today. It went pretty quickly. I don't think it suffered. R.I.P.

I am sorry for your loss. What brand was it and did you have it backed up?

Link to comment
Share on other sites

Guest asrequested

I am sorry for your loss. What brand was it and did you have it backed up?

 

'Twas a valiant drive. A seagate, 'twas. Sadly, but not regrettably, it's cargo was not precious and was not backed up. Ye' old DVR drive 'twas. It had a pool brother, so all was not lost. A replacement, I had. 

Link to comment
Share on other sites

PenkethBoy

i usually take mine out back and give them a good beating when the die - nice large sledge hammer does the trick usually  :P

  • Like 1
Link to comment
Share on other sites

Guest asrequested

i usually take mine out back and give them a good beating when the die - nice large sledge hammer does the trick usually :P

Working out some aggression, were you? Lol

Link to comment
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now
×
×
  • Create New...