2 drives failed in Raid 5

[JuJuJurassic 49]

Talk about unlucky...

Raid 5 array, well it's TruNas's equivalent, 1 drive failed, no problem, under warranty, replacement disk arrives, plug it in, a few clicks and resilvering starts, then it tells me one of the other disks is failing in the array. I check it out, and another drive is dying. Hopefully it'll resilver before the other drive fails, but.....

I've had a drive fail before, all the data is backed up, but the meta data takes days to rebuild. I have the data stored in /var/lib/emby/metadata, It looks like I may have to restore from backup, but if I restore the data, it's in a smb share, will the meta data be safe? Previously I've lost it, 

How can I preserve it this time?

Thanks

 

[Luke 42077]

Hi, by backup do you mean a backup created by Emby server?

[JuJuJurassic 49]

bith,  a emby backup and a copy of the media files into another server, 

[Luke 42077]

OK, I don't see why you couldn't restore it. Did you try it?

[rbjtech 5282]

imo RAID5 or any form of a parity disk is very out of date now, especially with multi-TB disks.   The rebuild process is a huge red alert where your entire array is at serious risk and also putting significant load on it while rebuilding, leading to possibly causing more failures = end of array.

For static media content, I've gone done the route of no redundancy at all - but I have a 1:1 backup on offline HDD (or across multiple HDD's).   Should a disk fail, I simply swap in the backup disk (into the pool - it's flexible enough to do that) and I'm up and running.   The replacement disk becomes the new backup once I've copied it.    No rebuilding, no parity, no extra risk.     I tend to use previously upgraded HDD's (capacity/density) as the backup HDD's.  The bonus of course is you also have an offline backup, which is NOT the same as RAID.

For constantly changing storage - then yes, some for of duplication is essential - but for static media, my argument is why do you need to be able to reproduce it on the fly ?

[Neminem 1516]

41 minutes ago, rbjtech said:

imo RAID5 or any form of a parity disk is very out of date now, especially with multi-TB disks.   The rebuild process is a huge red alert where your entire array is at serious risk and also putting significant load on it while rebuilding, leading to possibly causing more failures = end of array.

I would tent to say your are right.

But It's always been a cost / benefit.

10 Years ago I went down the road you describe, but cost was to high for me.

So I later opted for unRaid, with 90 day parity check ups.

And 1 thing that has always been I my minde is the "bad batch".

Newer buy the same drive model from the same vender.

Newer use the same drive mode / batch nr. for parity drive as your data drives.

Always check the batch nr. when buying new drives, if you have it, don't repeat it.

[rbjtech 5282]

@LessajI've no issue with the disagreement but tell me your thoughts on why.   I'm open to alternatives if they provide a better outcome.

[Lessaj 467]

5 minutes ago, rbjtech said:

@LessajI've no issue with the disagreement but tell me your thoughts on why.   I'm open to alternatives if they provide a better outcome.

I strongly believe in using arrays, and I agree that having to rebuild a drive in an array is quite an intense operation if you get unlucky and have another drive failure where the array wasn't designed to handle that scenario, you're effectively re-reading all of the data and recalculating parity and writing out to a single drive (which is the bottleneck in terms of speed) but I run monthly scrubs which is doing almost the same thing. Whether you go 1 2 or 3 disks of parity is dependent on a few factors, for large multi-TB arrays it should be at least 2 drives of parity with checksums to deal with bitrot. My main issue with using single drives is checks and balances - how do you know that the data being written was written correctly? How do you know it was read correctly? Especially if you're not using ECC RAM in the system, bits can and absolutely do flip in memory randomly. If you're using something like DrivePool where you have the option to just slot a drive back in, sure you can do that, but I don't believe in keeping data on a single drive without any parity information or ways to detect corruption. Also that means you have hardware lying around unused that you may need to manually maintain in terms of making sure it has all the same information a few months later. Having a backup is absolutely important and is an extra cost. I put my extra cost into a second array instead, which is always available and has a similar maintenance strategy.

At the end of the day the implementation you go with will depend on how much you care about the availability and accuracy of the data. I used MD raid for a while and had to deal with scenarios of needing to repair the data on the XFS file system and later on went with a ZFS solution for availability and data accuracy reasons - especially when it comes to very large arrays with a lot of data. The majority of people probably don't need to do this and probably don't even care.

This video covers most of the topic much more eloquently than I could write out. There's also a really good followup video too.

 

[rbjtech 5282]

18 hours ago, Lessaj said:

 My main issue with using single drives is checks and balances - how do you know that the data being written was written correctly? How do you know it was read correctly? Especially if you're not using ECC RAM in the system, bits can and absolutely do flip in memory randomly. If you're using something like DrivePool where you have the option to just slot a drive back in, sure you can do that, but I don't believe in keeping data on a single drive without any parity information or ways to detect corruption.

First of all, thanks for taking the time to post - some interesting info here - some way above my head as I'm a network/security guy, not a storage guy

I think this goes back to the point you made on the availability and accuracy of the data and a point I stated in my explanation on why I do it this way.

"For static media content.."

For anything that you can reproduce bit-for-bit from another source - then the option to recover this yourself, at your cost (time/effort), is 100% your choice.   If a video file (99% of my storage pool capacity) drops a bit or even hundreds of bits, generally, error correction in the codec will simply ignore/mask it and carry on.  Yes of course there will be a time when corruption kills the file but that is rare and unlikely to be caused by I/O.  On the flip side, I agree 100% on if you lose a bit of a dynamic data/bin file then you are in trouble.   This is why I do run drivepool across 3 platters for the important files - but none of my emby system, video nor metadata use this pool for the above reasons.

18 hours ago, Lessaj said:

Also that means you have hardware lying around unused that you may need to manually maintain in terms of making sure it has all the same information a few months later. Having a backup is absolutely important and is an extra cost. I put my extra cost into a second array instead, which is always available and has a similar maintenance strategy.

Again, key word here is static - I have a 'point in time' cold standby system that i can bring operational on the network in 5 minutes while it boots.  At the time of the backup/sync (as that's what it is) it was 100% verified from the backup software.  Can I guarantee it's 100% bit for bit - no, but as above, it doesn't really matter.   It also puts to good use all my upgraded 'old' (but still serviceable) HDD's - so my backup/recovery  system actually runs more than twice as many HDD's as my primary system - because the HDD's are at least half the capacity of the new 'live' ones.   But as it's offline, it doesn't matter.

For Enterprise class storage - I'm 100% with you, but for home Emby storage - my solution works well for me as I have traded recovery time/effort for less spinning platters and storage complexity on my live system.

Edited February 5 by rbjtech

[yaksplat 66]

I haven't used raid since 3TB drives came out. This is why i use robocopy and mirror every drive instead.  My rebuild is just a copy from the backup.

I also have an offsite backup, just in case. 

[Lessaj 467]

4 hours ago, rbjtech said:

First of all, thanks for taking the time to post - some interesting info here - some way above my head as I'm a network/security guy, not a storage guy

For Enterprise class storage - I'm 100% with you, but for home Emby storage - my solution works well for me as I have traded recovery time/effort for less spinning platters and storage complexity on my live system.

My IT professional career is in the enterprise space so I've tried to adopt a lot of the enterprise methodologies where it's reasonable to do so with my homelab environment. If I could run my own SAN I totally would! I don't have a dedicated cluster for testing for example, I do everything in my "production" cluster, but I can spin up a test VM in that infrastructure if needed. My career and hobbies collided so my skills and knowledge go all the way up and down the stack, from network ingress all the way down to how data is stored on platters/NAND and consideration of factors like EMI, transport errors, even incorrect CPU calculations. While my actual role revolves around application support/development I have been pulled in to support issues in all areas of the infrastructure and it's allowed me to have a very broad base to work from. I may not have experience with every technology or product, but I can generally pick up anything really quickly. Again, overkill for most people to treat my environment/services like an enterprise. I'm not aiming for 5 9's of uptime, that's something like 5 minutes out of the whole year, but if I can 2 or 3 9's of uptime I think that's really cool for a homelab.

4 hours ago, rbjtech said:

I think this goes back to the point you made on the availability and accuracy of the data and a point I stated in my explanation on why I do it this way.

"For static media content.."

For anything that you can reproduce bit-for-bit from another source - then the option to recover this yourself, at your cost (time/effort), is 100% your choice.   If a video file (99% of my storage pool capacity) drops a bit or even hundreds of bits, generally, error correction in the codec will simply ignore/mask it and carry on.  Yes of course there will be a time when corruption kills the file but that is rare and unlikely to be caused by I/O.  On the flip side, I agree 100% on if you lose a bit of a dynamic data/bin file then you are in trouble.   This is why I do run drivepool across 3 platters for the important files - but none of my emby system, video nor metadata use this pool for the above reasons.

Again, key word here is static - I have a 'point in time' cold standby system that i can bring operational on the network in 5 minutes while it boots.  At the time of the backup/sync (as that's what it is) it was 100% verified from the backup software.  Can I guarantee it's 100% bit for bit - no, but as above, it doesn't really matter.   It also puts to good use all my upgraded 'old' (but still serviceable) HDD's - so my backup/recovery  system actually runs more than twice as many HDD's as my primary system - because the HDD's are at least half the capacity of the new 'live' ones.   But as it's offline, it doesn't matter.

For Enterprise class storage - I'm 100% with you, but for home Emby storage - my solution works well for me as I have traded recovery time/effort for less spinning platters and storage complexity on my live system.

You're likely correct that more than likely the player/ffmpeg would be able to handle small errors if they do happen in the data, unless you get really unlucky with what becomes corrupted. I do agree that it's all able to be recreated if something were to happen - and there is time/effort involved in doing that - my philosophy is around what is the most approachable option. It could take me weeks to recreate things without a solid backup or some kind of list of what I had or spot checking everything to find something wrong, whereas building an array the right way the first time that has redundancy might be a few thousand dollars up front, but what is my time and peace of mind worth? If all I have to do is swap a drive and walk away (which I can do while the system is online with hot plug, and still have full access to all the data) and just check on it occasionally until it's done then I'm happy with that design.

I'm also thinking about the future of the data and it's why I want to prevent bitrot from happening if I can because some of it cannot be replaced, old photos and videos from devices or drives I don't have anymore. It would suck if it's (silently) corrupted on the main array, because that means the backup is corrupted too. A lot of the media I'm collecting now I don't have time to watch, but later in life when I have more free time I'll have an opportunity to do that and I want my data from 20+ years ago to be safe and corruption free. I've carried a lot of it forward from years gone by, and I hope to continue to do that.

Another point that I didn't specifically touch on was performance, you get a lot more speed out of the drives working in tandem rather than being limited to the throughput of 1-2 drives which can significantly speed up operations. It's not really needed most of the time and again most people probably won't care, but performance is something that's important to me. My array handles more than just housing media but it's still mostly "write once, read many". I like having one place for everything, logically separated by datasets/folders which can have their own policies if wanted. You could specify this dataset has compression, or deduplication, or different record sizes because the files are all small/large. There's lots of flexibility within a single array.

With regards to your "point in time" this is what makes ZFS snapshots so powerful, and it's what I utilize on the arrays as part of the backup strategy. If I end up overwriting a file by accident I can just look in the hidden snapshot directory and recover it, or if I want to see what folders I had at that point in time I can do that going back across multiple snapshots. I've utilized the snapshots in the past at a file level but also at an array level, I was building a new array for my media and instead of replacing one drive at a time until the new space became available (which is something you can do as well, and I have done that before) I just took a snapshot and used zfs send | zfs recv to send the dataset up to that snapshot which is the bulk of the data, then at the end I just create another one to send any differential data, stop the arrays and swap the drives over (again completely online with hot plugging) and start the array back up and I'm ready to go with minimal downtime of any applications/services that use the array. Technically my backup array is only needed for a few hours each day as part of daily differential for my media array, hypervisors, and firewall, but it runs in a VM in my cluster so it's always available.

With the recent improvements to ZFS I finally have the ability to add drives to the VDEV with raidz expand, I just can't change the parity level - I would need to add an entirely new VDEV that has the parity layout I want and then I believe as long as it has the space you can remove the other VDEV and it will move all the data over, otherwise you have to make a new array. But that's where the zfs send | zfs recv to do a block level copy of the data to the new array comes in, so it copies at max speed the entire time because it's not copying files it's copying chunks of block level data, all those millions of little files that slow down a transfer are a complete non-issue - and I can even do that over the network with SSH if I want, it doesn't have to be on the same system. I've done something similar before with LVM and pvmove but never specifically tried removing a VDEV with ZFS, I've only removed drives from a VDEV where space allowed and then readded.

I don't fundamentally disagree with your approach, it certainly works for you and lots of other people and it's probably unlikely that anything catastrophically bad will happen. It's a pretty longstanding product with a large user base and is a very simple way for Windows users to have access to a pretty solid array (storage spaces is garbage), but I hope I've highlighted some of the risks of relying on a single drive over long periods of time - whether that be a drive used for data, or a drive used for parity - and that it helped peak a curiosity to learn more about what is really happening down on the hardware level.

[rbjtech 5282]

10 hours ago, Lessaj said:

I don't fundamentally disagree with your approach, it certainly works for you and lots of other people and it's probably unlikely that anything catastrophically bad will happen. It's a pretty longstanding product with a large user base and is a very simple way for Windows users to have access to a pretty solid array (storage spaces is garbage), but I hope I've highlighted some of the risks of relying on a single drive over long periods of time - whether that be a drive used for data, or a drive used for parity - and that it helped peak a curiosity to learn more about what is really happening down on the hardware level.

Absolutely - some great info there.  I'm also in the Enterprise space, I've previously worked with EMC Symetrix SAN's and NetApp filers but then moved into Networks & Security many years ago.  I expect many will consider my home network OTT which, as you have done for storage, is built using Enterprise level methodologies (dmz's, perimeter zones, reverse proxies, IPS's blah blah).     I will certainly follow up on checking my long term backups again as un-reproducable things like digital photo's & home video I've never really given much thought to ensuring they are recoverable - assuming the media is reliable.  I do have multiple copies - but I have no checksum to suggest all the multiple copies are 100%.