Jump to content

OT: Disaster Recovery options


crashkelly

Recommended Posts

crashkelly

Hi,

 

So about a week ago I had a 2TB drive fail on me and all attempts to recover either the drive or the data were not successful. Thankfully out of all the drives I store media on it was probably the least important as it was 2D rips of a bunch of my 3D movies so they are recoverable, but it will still be a bit of work.

 

After giving up on the drive I did start seriously thinking about the condition of my other drives so I ran a check using Crystal Disk Info only to find that 3 of the 12 drives I have remaining have issues and Crystal Disk Info flags them with the big yellow "Caution" flag.

 

At that point I decided I have ignored any sort of disaster recovery plan for far too long.

 

My storage setup is comprised of 2 Mediasonic ProBox H82-SU3S2 8 Bay External Hard Drive Enclosures that are attached via USB3.0 to a Windows 8.1 rig. Currently I have 4 2TB drives (Green) and 8 3TB Drives and 1 6TB drive that is on the way to me as I write this. Of the 29TB of real storage that I have I would say I am at about 80% used, it is all movies and TV shows.

 

With that percent of used space I am really on the side of a solution that I can add non-empty disk to to save me trying to move data around for days, or more, to get free drives to keep adding to an expanding storage pool.

 

I did look at Windows Storage Spaces, but besides the fact it has to format your drives the user comments I was reading were about 50/50 on the bad and good.

 

I have been reading a bit on DriveBender and StableBit DrivePool.

 

A few years ago I had tried FlexRAID but found quite soon in that, while it was great for pooling, the "backup" processes that would occur overnight were not even getting close to complete before the morning. Did post that on the FlexRAID forums and it was attributed to the drives being in external USB enclosures.

 

I am not looking to replicate the media, mirroring, but rather something along the lines of using parity drives and at least right now, a Windows solution.

 

Would be appreciative of insight from the MB masses on what they are doing\using to protect against the unavoidable drive death.

 

Thanks much

CrashKelly

 

 

Link to comment
Share on other sites

I'm using the transparent product from FlexRaid. I used to run snapshot raid, but due to the size of the data being protected the maintenance for updates/validates/verifies was getting way out of control.

  • Like 2
Link to comment
Share on other sites

hamstercat

I'm using SnapRAID and it's working like a charm, never had to recover anything yet though and hoping it stays that way.  :P

 

It's an open-source project and while it was developed for Linux, I read it works fine on Windows. It's not really RAID, it's snapshot based, so the first sync is gonna be long, but after that it should be fine. It's file-based, so it only re-sync files that changed. It works with any drives without having to format them, you just have to sacrifice your biggest drives for parity.

 

I'd stay away from RAID with the number of hard drives you have. Plus, snapshot is quite nice for media as it tends to not change often, so real-time protection is usually not as important.

Edited by hamstercat
  • Like 1
Link to comment
Share on other sites

crashkelly

 

From: http://mediabrowser.tv/community/index.php?/topic/13694-ssd-vs-hdd-vs-hybrid-ssd-raid-need-to-upgrade-advice-storage-options/?hl=%2Bstartech+%2Bsata#entry151265 

 

I've since switched for media storage to an external 4 disk array with 6TB controlled by a LSI 8 drive controller so my RAID 5 write speed is 200/MBs,  

This solution gives me 18TB of usable space with parity on this external 4 disk shelf....and the RAID controller has capacity to add another 4 disk storage shelf for another 24TB (with 6TB disks) of space in the future.

 

 

Thanks for the info. Would love to go that direction and eventually will, or at least something similar. Already have a 42U server rack in the basement with all the network gear on it and am saving the pennies, okay nickels as we do not have pennies in Canada anymore, for a proper server unit and disk racks. Other problem is being in Canada, just the 4-bayenclosue from your link is $521 + shipping from amazon.ca

 

I'm using the transparent product from FlexRaid. I used to run snapshot raid, but due to the size of the data being protected the maintenance for updates/validates/verifies was getting way out of control.

 

Are you running internal or external drives?

 

Cheers and thanks again

CrashKelly

Link to comment
Share on other sites

crashkelly

I'm using SnapRAID and it's working like a charm, never had to recover anything yet though and hoping it stays that way.  :P

 

It's an open-source project and while it was developed for Linux, I read it works fine on Windows. It's not really RAID, it's snapshot based, so the first sync is gonna be long, but after that it should be fine. It's file-based, so it only re-sync files that changed. It works with any drives without having to format them, you just have to sacrifice your biggest drives for parity.

 

I'd stay away from RAID with the number of hard drives you have. Plus, snapshot is quite nice for media as it tends to not change often, so real-time protection is usually not as important.

 

That sounds interesting, I will check it out

 

Cheers

CrashKelly

 

EDIT: Just read up a bunch on SnapRAID and it looks interesting. If I went with this I would be sending my new 6TB drive back without ever opening it as I would not sacrifice that drive for a parity drive and would probably just snag two more 3TB drives and use one of those for parity

Edited by crashkelly
Link to comment
Share on other sites

I use RAID 10 on a rather large SAN array (3TB X 20 drives 18 active + 2 hot spare expandable to 45 drives) and backup to BackBlaze and my RV - BackBlaze has been nice for 95$ for two years I currently have about 8TB backed up to them.

  • Like 1
Link to comment
Share on other sites

crashkelly

I use RAID 10 on a rather large SAN array (3TB X 20 drives 18 active + 2 hot spare expandable to 45 drives) and backup to BackBlaze and my RV - BackBlaze has been nice for 95$ for two years I currently have about 8TB backed up to them.

 

Looking for the drool emoticon ...... Very nice setup

 

Looking for a solution that I can get into place, or at least started on, this weekend. Would love to move to a setup similar to yours, one day maybe after I build the CrashHT in my unfinished basement

 

Cheers

CrashKelly

Link to comment
Share on other sites

yeah it has taken me many many years to acquire the hardware in use today - I posted some pic's in the "show off your rig" thread page 3 - getting the wife's blessing was hard at first but after she got used to the setup she will not let me take it away so upgrading is pretty easy when needed :)

Edited by mbnwa
Link to comment
Share on other sites

crashkelly

yeah it has taken me many many years to acquire the hardware in use today - I posted some pic's in the "show off your rig" thread page 3 - getting the wife's blessing was hard at first but after she got used to the setup she will not let me take it away so upgrading is pretty easy when needed :)

 

Going to have to get your wife to talk to my wife :D

 

My wife uses MB more and more as the days go on. Getting the Samsung App right on her TV helped a lot as it is one remote to do everything and it is her TV so she is always logged and just uses.

 

Me getting the server rack was a big step and broke the ice

 

Took a look at your setup, and then looked again, and again, ......

 

Cheers

CrashKelly

Link to comment
Share on other sites

lol yeah when the wife found out we had 20+TB of DVR storage for shows + all our movies etc worked out well - I think the thing that sold her the most was when we are on vacation we can stream our Comcast Cable from the house to our remote devices and she can catch up on recorded tv, etc - It also helps with my kid when on trips with the movie server backed up to our RV she can stream movies directly to her iPad with out using cellular data

Edited by mbnwa
Link to comment
Share on other sites

Deathsquirrel

As always I say to hell with RAID, especially SW RAID, for home media backup.  Offline storage is where it's at IMO.  Buy backup drives, make a copy of existing media, dump in a safe deposit box or some other remote site.

 

Of course any solution will be a pain to start with when you already have 20+ tb of data.

Link to comment
Share on other sites

i use hardware raid with no more than 8 drives per raid-6 array. i do backups for the things i care about most, and for the rest i just do my best to maintain the health of the arrays using regular volume checks, things of that nature. if a drive reports a failure i get everything backed up before activating the hot spare, then let it rebuild. you'll generally be better off if you plan for failure and know how you're going to react when it happens.

Link to comment
Share on other sites

As always I say to hell with RAID, especially SW RAID, for home media backup.  Offline storage is where it's at IMO.  Buy backup drives, make a copy of existing media, dump in a safe deposit box or some other remote site.

 

Of course any solution will be a pain to start with when you already have 20+ tb of data.

 

I think a compromise is best for me and probably for many. If you have the time to periodically do manual backups onto separate drives then great, but most of us don't. So you backup what you care about, and then you can give the rest a pretty long lifespan with hardware raid, resiliency, and doing what the manual says.

Link to comment
Share on other sites

crashkelly

i use hardware raid with no more than 8 drives per raid-6 array. i do backups for the things i care about most, and for the rest i just do my best to maintain the health of the arrays using regular volume checks, things of that nature. if a drive reports a failure i get everything backed up before activating the hot spare, then let it rebuild. you'll generally be better off if you plan for failure and know how you're going to react when it happens.

 

Totally agree on the "plan for failure", at least now. I had always felt that I would just re-rip everything if disaster happened but now that disaster has happened I am not looking forward to the re-rip part. Hindsight is always 20/20

 

Cheers

CrashKelly

 

EDIT: I have always kept multiple backups of the important stuff, documents, photos, ...., but never the media and most of the reason for that was the cost of extra drives to backup a rapidly growing collection. A simple solution, in the interim, like SnapRAID might be a good start. That and keeping a better eyes on the general health of the harddrives

Edited by crashkelly
Link to comment
Share on other sites

Deathsquirrel

I think a compromise is best for me and probably for many. If you have the time to periodically do manual backups onto separate drives then great, but most of us don't. So you backup what you care about, and then you can give the rest a pretty long lifespan with hardware raid, resiliency, and doing what the manual says.

 

Honestly doesn't take much time.  New movies are copied to a second pc right after ripping and identification.  That way there are two copies from day 1.  When the drive I use for that fills I pull out a backup drive and move over everything to that.  When the backup drive fills I move it offsite and order a new drive of whatever size is ridiculously large at the time.

 

Get a sata to usb adapter and you can use internal drives to save a few bucks.

 

Even as a fully manual process it might be ten-twenty minutes a month.  Of course that's without doing any torrenting.  if that's your movie acquisition method your collection grows too fast for complete backup I expect.  I'm only growing by about 5 movies a week on average these days.  One catastrophic data loss and rerip was enough for me thanks!

Link to comment
Share on other sites

Right yea, for the first 10-20TB that's feasible. When your collection goes well beyond that, that's when i think people start picking and choosing a little more.

  • Like 2
Link to comment
Share on other sites

saitoh183

I have 17.3TB and i use a Stablebit Drivepool for pooling, Stablebit Scanner to keep an eye on drive health and scans and Flexraid (expert mode snapshot with update task only) for backup. Drivepool + Scanner is awesome because drivepool can automatically move data off dmg disc if Scanner detects something and instantly sends me a email. I have had to recover via Flexraid, and all has gone well. I also run Easeus ToDo Backup to backup the important stuff like pictures  and documents on too a external 2TB My Book Live

Link to comment
Share on other sites

crashkelly

Right yea, for the first 10-20TB that's feasible. When your collection goes well beyond that, that's when i think people start picking and choosing a little more.

 

Pretty much exactly the reason I was without a solution. Have to balance the $$ for extra drives with the time it will take to recovery from a loss. If I can drop $150 and at least be protected from a single drive failure that is, now, worth it to me, especially with 3 "Caution" drives that it is only a matter of a short time before at another one of them goes

 

Cheers

CrashKelly

Link to comment
Share on other sites

aspdend

I use UnRAID for my data storage. I took my old PC in a large Fractal Design Define XL case, added some drive cages and SATA controller cards. I bought a new 3Tb WD Red drive for the Parity then filled the machine with 15 of my old hard drives I had acquired and either discarded over the years when I bought bigger drives or was using for data storage. I also added an older 64Gb SSD as a cache drive. The HDD's range from 2Tb to 380Mb in size and I have 27Tb storage in total. This has been rock solid for me over the last year or two and I have no regrets about going the UnRAID route. It's easy to manage and I just leave it alone for most of the time, apart form adding any new media. I've had no drive failures, although one of my drives was starting to show issues so I bought a new 3Tb WD Red and replaced it.  

Link to comment
Share on other sites

Deathsquirrel

Well whatever you do, backup your TV show rips.  Identifying which episode is which on a disc is an unpleasant chore you won't want to repeat.  I had to once.  Spare yourself that at least.

  • Like 2
Link to comment
Share on other sites

Swynol

i use drive pool. all my drives are combined into one. it allows me to store my most important things across as many drives as i choose. 

 

for the important docs etc i have them copied across 3 drives and also backed up via google drive. I'm currently working on a backup solution for my 10TB collection, not sure what is feasible without costing too much and being expandable.  

Link to comment
Share on other sites

crashkelly

I use UnRAID for my data storage. I took my old PC in a large Fractal Design Define XL case, added some drive cages and SATA controller cards. I bought a new 3Tb WD Red drive for the Parity then filled the machine with 15 of my old hard drives I had acquired and either discarded over the years when I bought bigger drives or was using for data storage. I also added an older 64Gb SSD as a cache drive. The HDD's range from 2Tb to 380Mb in size and I have 27Tb storage in total. This has been rock solid for me over the last year or two and I have no regrets about going the UnRAID route. It's easy to manage and I just leave it alone for most of the time, apart form adding any new media. I've had no drive failures, although one of my drives was starting to show issues so I bought a new 3Tb WD Red and replaced it.  

 

Three years ago I would have for sure gone the unRAID route as I like what I have read about it. Unfortunately about 3 years ago it was time for a major tech purge and I got rid of everything computer related that I had collected over the years ..... I am still grieving that day

 

Cheers

CrashKelly

Link to comment
Share on other sites

crashkelly

Well whatever you do, backup your TV show rips.  Identifying which episode is which on a disc is an unpleasant chore you won't want to repeat.  I had to once.  Spare yourself that at least.

 

I hear you on that. A lot of the wife's shows are mini-series and are stored as S01E01 - Episode 1.mkv

 

Cheers

CRashKelly

Link to comment
Share on other sites

aspdend

Three years ago I would have for sure gone the unRAID route as I like what I have read about it. Unfortunately about 3 years ago it was time for a major tech purge and I got rid of everything computer related that I had collected over the years ..... I am still grieving that day

 

Cheers

CrashKelly

That's a shame - I knew there was a good reason to hold onto all my older tech!

Link to comment
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now
×
×
  • Create New...