Jump to content


Photo

OT: Disaster Recovery options

recovery drive

  • Please log in to reply
28 replies to this topic

#1 crashkelly OFFLINE  

crashkelly

    Advanced Member

  • Alpha Testers
  • 451 posts
  • Local time: 01:58 PM
  • LocationAirdrie, Alberta, Canada

Posted 29 January 2015 - 10:19 PM

Hi,

 

So about a week ago I had a 2TB drive fail on me and all attempts to recover either the drive or the data were not successful. Thankfully out of all the drives I store media on it was probably the least important as it was 2D rips of a bunch of my 3D movies so they are recoverable, but it will still be a bit of work.

 

After giving up on the drive I did start seriously thinking about the condition of my other drives so I ran a check using Crystal Disk Info only to find that 3 of the 12 drives I have remaining have issues and Crystal Disk Info flags them with the big yellow "Caution" flag.

 

At that point I decided I have ignored any sort of disaster recovery plan for far too long.

 

My storage setup is comprised of 2 Mediasonic ProBox H82-SU3S2 8 Bay External Hard Drive Enclosures that are attached via USB3.0 to a Windows 8.1 rig. Currently I have 4 2TB drives (Green) and 8 3TB Drives and 1 6TB drive that is on the way to me as I write this. Of the 29TB of real storage that I have I would say I am at about 80% used, it is all movies and TV shows.

 

With that percent of used space I am really on the side of a solution that I can add non-empty disk to to save me trying to move data around for days, or more, to get free drives to keep adding to an expanding storage pool.

 

I did look at Windows Storage Spaces, but besides the fact it has to format your drives the user comments I was reading were about 50/50 on the bad and good.

 

I have been reading a bit on DriveBender and StableBit DrivePool.

 

A few years ago I had tried FlexRAID but found quite soon in that, while it was great for pooling, the "backup" processes that would occur overnight were not even getting close to complete before the morning. Did post that on the FlexRAID forums and it was attributed to the drives being in external USB enclosures.

 

I am not looking to replicate the media, mirroring, but rather something along the lines of using parity drives and at least right now, a Windows solution.

 

Would be appreciative of insight from the MB masses on what they are doing\using to protect against the unavoidable drive death.

 

Thanks much

CrashKelly

 

 



#2 AgileHumor OFFLINE  

AgileHumor

    Advanced Member

  • Alpha Testers
  • 759 posts
  • Local time: 12:58 PM

Posted 29 January 2015 - 10:33 PM

delete me, pii cleanup


Edited by AgileHumor, 17 April 2015 - 01:57 AM.

  • crashkelly likes this

#3 Pais OFFLINE  

Pais

    Advanced Member

  • Members
  • 216 posts
  • Local time: 04:58 PM

Posted 29 January 2015 - 10:34 PM

I'm using the transparent product from FlexRaid. I used to run snapshot raid, but due to the size of the data being protected the maintenance for updates/validates/verifies was getting way out of control.


  • crashkelly and politby like this

#4 hamstercat OFFLINE  

hamstercat

    Advanced Member

  • Developers
  • 131 posts
  • Local time: 04:58 PM
  • LocationMontréal, Canada

Posted 29 January 2015 - 11:06 PM

I'm using SnapRAID and it's working like a charm, never had to recover anything yet though and hoping it stays that way.  :P

 

It's an open-source project and while it was developed for Linux, I read it works fine on Windows. It's not really RAID, it's snapshot based, so the first sync is gonna be long, but after that it should be fine. It's file-based, so it only re-sync files that changed. It works with any drives without having to format them, you just have to sacrifice your biggest drives for parity.

 

I'd stay away from RAID with the number of hard drives you have. Plus, snapshot is quite nice for media as it tends to not change often, so real-time protection is usually not as important.


Edited by hamstercat, 29 January 2015 - 11:06 PM.

  • crashkelly likes this

#5 crashkelly OFFLINE  

crashkelly

    Advanced Member

  • Alpha Testers
  • 451 posts
  • Local time: 01:58 PM
  • LocationAirdrie, Alberta, Canada

Posted 29 January 2015 - 11:08 PM

 

From: http://mediabrowser....ta#entry151265 

 

I've since switched for media storage to an external 4 disk array with 6TB controlled by a LSI 8 drive controller so my RAID 5 write speed is 200/MBs,  

This solution gives me 18TB of usable space with parity on this external 4 disk shelf....and the RAID controller has capacity to add another 4 disk storage shelf for another 24TB (with 6TB disks) of space in the future.

 

 

Thanks for the info. Would love to go that direction and eventually will, or at least something similar. Already have a 42U server rack in the basement with all the network gear on it and am saving the pennies, okay nickels as we do not have pennies in Canada anymore, for a proper server unit and disk racks. Other problem is being in Canada, just the 4-bayenclosue from your link is $521 + shipping from amazon.ca

 

I'm using the transparent product from FlexRaid. I used to run snapshot raid, but due to the size of the data being protected the maintenance for updates/validates/verifies was getting way out of control.

 

Are you running internal or external drives?

 

Cheers and thanks again

CrashKelly



#6 crashkelly OFFLINE  

crashkelly

    Advanced Member

  • Alpha Testers
  • 451 posts
  • Local time: 01:58 PM
  • LocationAirdrie, Alberta, Canada

Posted 29 January 2015 - 11:09 PM

I'm using SnapRAID and it's working like a charm, never had to recover anything yet though and hoping it stays that way.  :P

 

It's an open-source project and while it was developed for Linux, I read it works fine on Windows. It's not really RAID, it's snapshot based, so the first sync is gonna be long, but after that it should be fine. It's file-based, so it only re-sync files that changed. It works with any drives without having to format them, you just have to sacrifice your biggest drives for parity.

 

I'd stay away from RAID with the number of hard drives you have. Plus, snapshot is quite nice for media as it tends to not change often, so real-time protection is usually not as important.

 

That sounds interesting, I will check it out

 

Cheers

CrashKelly

 

EDIT: Just read up a bunch on SnapRAID and it looks interesting. If I went with this I would be sending my new 6TB drive back without ever opening it as I would not sacrifice that drive for a parity drive and would probably just snag two more 3TB drives and use one of those for parity


Edited by crashkelly, 29 January 2015 - 11:36 PM.


#7 mbnwa OFFLINE  

mbnwa

    Advanced Member

  • Alpha Testers
  • 478 posts
  • Local time: 01:58 PM

Posted 29 January 2015 - 11:34 PM

I use RAID 10 on a rather large SAN array (3TB X 20 drives 18 active + 2 hot spare expandable to 45 drives) and backup to BackBlaze and my RV - BackBlaze has been nice for 95$ for two years I currently have about 8TB backed up to them.


  • crashkelly likes this

#8 crashkelly OFFLINE  

crashkelly

    Advanced Member

  • Alpha Testers
  • 451 posts
  • Local time: 01:58 PM
  • LocationAirdrie, Alberta, Canada

Posted 29 January 2015 - 11:57 PM

I use RAID 10 on a rather large SAN array (3TB X 20 drives 18 active + 2 hot spare expandable to 45 drives) and backup to BackBlaze and my RV - BackBlaze has been nice for 95$ for two years I currently have about 8TB backed up to them.

 

Looking for the drool emoticon ...... Very nice setup

 

Looking for a solution that I can get into place, or at least started on, this weekend. Would love to move to a setup similar to yours, one day maybe after I build the CrashHT in my unfinished basement

 

Cheers

CrashKelly



#9 mbnwa OFFLINE  

mbnwa

    Advanced Member

  • Alpha Testers
  • 478 posts
  • Local time: 01:58 PM

Posted 30 January 2015 - 12:00 AM

yeah it has taken me many many years to acquire the hardware in use today - I posted some pic's in the "show off your rig" thread page 3 - getting the wife's blessing was hard at first but after she got used to the setup she will not let me take it away so upgrading is pretty easy when needed :)


Edited by mbnwa, 30 January 2015 - 12:01 AM.


#10 crashkelly OFFLINE  

crashkelly

    Advanced Member

  • Alpha Testers
  • 451 posts
  • Local time: 01:58 PM
  • LocationAirdrie, Alberta, Canada

Posted 30 January 2015 - 12:11 AM

yeah it has taken me many many years to acquire the hardware in use today - I posted some pic's in the "show off your rig" thread page 3 - getting the wife's blessing was hard at first but after she got used to the setup she will not let me take it away so upgrading is pretty easy when needed :)

 

Going to have to get your wife to talk to my wife :D

 

My wife uses MB more and more as the days go on. Getting the Samsung App right on her TV helped a lot as it is one remote to do everything and it is her TV so she is always logged and just uses.

 

Me getting the server rack was a big step and broke the ice

 

Took a look at your setup, and then looked again, and again, ......

 

Cheers

CrashKelly



#11 mbnwa OFFLINE  

mbnwa

    Advanced Member

  • Alpha Testers
  • 478 posts
  • Local time: 01:58 PM

Posted 30 January 2015 - 12:14 AM

lol yeah when the wife found out we had 20+TB of DVR storage for shows + all our movies etc worked out well - I think the thing that sold her the most was when we are on vacation we can stream our Comcast Cable from the house to our remote devices and she can catch up on recorded tv, etc - It also helps with my kid when on trips with the movie server backed up to our RV she can stream movies directly to her iPad with out using cellular data


Edited by mbnwa, 30 January 2015 - 12:14 AM.


#12 Deathsquirrel OFFLINE  

Deathsquirrel

    Advanced Member

  • Members
  • 1994 posts
  • Local time: 01:58 PM

Posted 30 January 2015 - 12:24 AM

As always I say to hell with RAID, especially SW RAID, for home media backup.  Offline storage is where it's at IMO.  Buy backup drives, make a copy of existing media, dump in a safe deposit box or some other remote site.

 

Of course any solution will be a pain to start with when you already have 20+ tb of data.



#13 Luke OFFLINE  

Luke

    System Architect

  • Administrators
  • 135735 posts
  • Local time: 04:58 PM

Posted 30 January 2015 - 12:26 AM

i use hardware raid with no more than 8 drives per raid-6 array. i do backups for the things i care about most, and for the rest i just do my best to maintain the health of the arrays using regular volume checks, things of that nature. if a drive reports a failure i get everything backed up before activating the hot spare, then let it rebuild. you'll generally be better off if you plan for failure and know how you're going to react when it happens.



#14 Luke OFFLINE  

Luke

    System Architect

  • Administrators
  • 135735 posts
  • Local time: 04:58 PM

Posted 30 January 2015 - 12:29 AM

As always I say to hell with RAID, especially SW RAID, for home media backup.  Offline storage is where it's at IMO.  Buy backup drives, make a copy of existing media, dump in a safe deposit box or some other remote site.

 

Of course any solution will be a pain to start with when you already have 20+ tb of data.

 

I think a compromise is best for me and probably for many. If you have the time to periodically do manual backups onto separate drives then great, but most of us don't. So you backup what you care about, and then you can give the rest a pretty long lifespan with hardware raid, resiliency, and doing what the manual says.



#15 crashkelly OFFLINE  

crashkelly

    Advanced Member

  • Alpha Testers
  • 451 posts
  • Local time: 01:58 PM
  • LocationAirdrie, Alberta, Canada

Posted 30 January 2015 - 12:31 AM

i use hardware raid with no more than 8 drives per raid-6 array. i do backups for the things i care about most, and for the rest i just do my best to maintain the health of the arrays using regular volume checks, things of that nature. if a drive reports a failure i get everything backed up before activating the hot spare, then let it rebuild. you'll generally be better off if you plan for failure and know how you're going to react when it happens.

 

Totally agree on the "plan for failure", at least now. I had always felt that I would just re-rip everything if disaster happened but now that disaster has happened I am not looking forward to the re-rip part. Hindsight is always 20/20

 

Cheers

CrashKelly

 

EDIT: I have always kept multiple backups of the important stuff, documents, photos, ...., but never the media and most of the reason for that was the cost of extra drives to backup a rapidly growing collection. A simple solution, in the interim, like SnapRAID might be a good start. That and keeping a better eyes on the general health of the harddrives


Edited by crashkelly, 30 January 2015 - 12:36 AM.


#16 Deathsquirrel OFFLINE  

Deathsquirrel

    Advanced Member

  • Members
  • 1994 posts
  • Local time: 01:58 PM

Posted 30 January 2015 - 12:40 AM

I think a compromise is best for me and probably for many. If you have the time to periodically do manual backups onto separate drives then great, but most of us don't. So you backup what you care about, and then you can give the rest a pretty long lifespan with hardware raid, resiliency, and doing what the manual says.

 

Honestly doesn't take much time.  New movies are copied to a second pc right after ripping and identification.  That way there are two copies from day 1.  When the drive I use for that fills I pull out a backup drive and move over everything to that.  When the backup drive fills I move it offsite and order a new drive of whatever size is ridiculously large at the time.

 

Get a sata to usb adapter and you can use internal drives to save a few bucks.

 

Even as a fully manual process it might be ten-twenty minutes a month.  Of course that's without doing any torrenting.  if that's your movie acquisition method your collection grows too fast for complete backup I expect.  I'm only growing by about 5 movies a week on average these days.  One catastrophic data loss and rerip was enough for me thanks!



#17 Luke OFFLINE  

Luke

    System Architect

  • Administrators
  • 135735 posts
  • Local time: 04:58 PM

Posted 30 January 2015 - 12:46 AM

Right yea, for the first 10-20TB that's feasible. When your collection goes well beyond that, that's when i think people start picking and choosing a little more.


  • crashkelly and mbnwa like this

#18 saitoh183 OFFLINE  

saitoh183

    Advanced Member

  • Members
  • 997 posts
  • Local time: 04:58 PM
  • LocationCanada, St-Eustache, QC

Posted 30 January 2015 - 01:19 AM

I have 17.3TB and i use a Stablebit Drivepool for pooling, Stablebit Scanner to keep an eye on drive health and scans and Flexraid (expert mode snapshot with update task only) for backup. Drivepool + Scanner is awesome because drivepool can automatically move data off dmg disc if Scanner detects something and instantly sends me a email. I have had to recover via Flexraid, and all has gone well. I also run Easeus ToDo Backup to backup the important stuff like pictures  and documents on too a external 2TB My Book Live



#19 crashkelly OFFLINE  

crashkelly

    Advanced Member

  • Alpha Testers
  • 451 posts
  • Local time: 01:58 PM
  • LocationAirdrie, Alberta, Canada

Posted 30 January 2015 - 01:19 AM

Right yea, for the first 10-20TB that's feasible. When your collection goes well beyond that, that's when i think people start picking and choosing a little more.

 

Pretty much exactly the reason I was without a solution. Have to balance the $$ for extra drives with the time it will take to recovery from a loss. If I can drop $150 and at least be protected from a single drive failure that is, now, worth it to me, especially with 3 "Caution" drives that it is only a matter of a short time before at another one of them goes

 

Cheers

CrashKelly



#20 aspdend OFFLINE  

aspdend

    Advanced Member

  • Alpha Testers
  • 540 posts
  • Local time: 09:58 PM
  • LocationWest Sussex

Posted 30 January 2015 - 05:11 AM

I use UnRAID for my data storage. I took my old PC in a large Fractal Design Define XL case, added some drive cages and SATA controller cards. I bought a new 3Tb WD Red drive for the Parity then filled the machine with 15 of my old hard drives I had acquired and either discarded over the years when I bought bigger drives or was using for data storage. I also added an older 64Gb SSD as a cache drive. The HDD's range from 2Tb to 380Mb in size and I have 27Tb storage in total. This has been rock solid for me over the last year or two and I have no regrets about going the UnRAID route. It's easy to manage and I just leave it alone for most of the time, apart form adding any new media. I've had no drive failures, although one of my drives was starting to show issues so I bought a new 3Tb WD Red and replaced it.  







Also tagged with one or more of these keywords: recovery, drive

0 user(s) are reading this topic

0 members, 0 guests, 0 anonymous users