Would smaller, but more, media libraries bring better performance?

[JeremyFr79 228]

So here's the problem with external drives.  The vast majority of OEM ones say Seagate/WD etc are not intended for "everyday" storage, They're almost always manufactured with backup duty in mind i.e. they use something of the likes of a "green" drive typically 5400 RPM configured to CONSTANTLY spin down etc.  Now there are some out there like say the "MyCloud" stuff that aren't this way, but the vast majority of externals are.  Sadly people will go into a store see an external 4TB drive for say 100bux and an internal for 150 and they go with the cheapest option, not realizing that the 50 dollar difference is that for a slower drive with less cache, etc that will never in it's wildest dreams perform anywhere near its internal counterpart.  It's like comparing an I3 to an I7 processor, at the end of the day they may be manufactured under the same process but you get what you pay for in loss of features/cores etc.

Edited May 16, 2016 by JeremyFr79

[Ponyo 22]

I disagree. For most people disk performance is completely irrelevant. They use external disks because it's easy. The disk doesn't have to be spinning all the time because most people just use it to save larger files or photos on it, not files they are frequently accessing.

 

If somebody ends up with 15 external drives that is bad planning IMO. You'd think after filling three or so drives up with media you'd realize you are going to save stuff up and not delete anything.

 

@theusedversion: How many TB are those drives? If you collected them over time chances are one 4 or 6TB drive could replace 5 of your current drives.

 

If all those drives are 3.5" drives chances are that they have normal SATA connectors and you could take them out of the casing and put them in a NAS.

Have you ever considered building your own NAS? 4 bay NAS' are so expensive that you could easily build your own system capable of holding many more drives for the same amount of money.

 

6 port mainboard, low end cpu, memory: ~150

PSU: ~80

Decent full tower case: ~80?

2x 6TB internal disk: ~500 (in case you want to get rid of some of your external disks )

 

8 port sata controller are ~100 for the cheaper models in case you need to add more disks in the future.

 

Little over 300 for a basic setup and still less than a 1000 for something that will hold a lot of your current drives plus a fair bit of additional storage.

[dcook 266]

You can get a 5bay QNAP NAS for $550

 

http://www.amazon.com/QNAP-TS-563-2G-x86-based-2-0GHz-10G-ready/dp/B00ZQ05TGE/ref=sr_1_8?ie=UTF8&qid=1463488041&sr=8-8&keywords=QNAP

 

 

And you already have drives (as long as they are 3.5")

Edited May 17, 2016 by dcook

[JeremyFr79 228]

I disagree. For most people disk performance is completely irrelevant. They use external disks because it's easy. The disk doesn't have to be spinning all the time because most people just use it to save larger files or photos on it, not files they are frequently accessing.

 

If somebody ends up with 15 external drives that is bad planning IMO. You'd think after filling three or so drives up with media you'd realize you are going to save stuff up and not delete anything.

 

@theusedversion: How many TB are those drives? If you collected them over time chances are one 4 or 6TB drive could replace 5 of your current drives.

 

If all those drives are 3.5" drives chances are that they have normal SATA connectors and you could take them out of the casing and put them in a NAS.

Have you ever considered building your own NAS? 4 bay NAS' are so expensive that you could easily build your own system capable of holding many more drives for the same amount of money.

 

6 port mainboard, low end cpu, memory: ~150

PSU: ~80

Decent full tower case: ~80?

2x 6TB internal disk: ~500 (in case you want to get rid of some of your external disks )

 

8 port sata controller are ~100 for the cheaper models in case you need to add more disks in the future.

 

Little over 300 for a basic setup and still less than a 1000 for something that will hold a lot of your current drives plus a fair bit of additional storage.

The type's of drives that they normally use in OEM externals will burn out quickly in a NAS setup, they don't have correct firmware for NAS/RAID and can lead to devastating data loss very quickly if used that way.  Not all hard drives are created equal.  In a NAS/RAID you want drives that aren't constantly trying to spin up/down, that will intelligently bypass a bad block thus not breaking your entire RAID due to a timeout because a consumer drive just sits there trying to read a bad block over and over and over etc, not to mention a far higher MTBF rating as well.  There's a reason why real NAS's are expensive it's because they use high quality parts among many other things.  Sure you can build a "NAS" but if you're going to do so on a "budget" then you might as well not do it at all.

[Ponyo 22]

If you want to run a RAID setup, yes. But in this case we have 15 external disks connecting over USB. I'd say the first priority would be to get all those disk on one case, provide some decent cooling, and a sata bus for some additional bandwidth to solve his performance issues.

 

If he wants to he can always upgrade to disks more suitable for storage/RAID configurations from there.

[Deathsquirrel 741]

If you want to run a RAID setup, yes. But in this case we have 15 external disks connecting over USB. I'd say the first priority would be to get all those disk on one case, provide some decent cooling, and a sata bus for some additional bandwidth to solve his performance issues.

 

That's the configuration I'm using.  For most users it's more than adequate.

[Guest asrequested]

For the amount of space she'll require, it's probably prudent to build a RAID 5 with 4 or 6TB drives. That's a lot of data to reallocate.

[JeremyFr79 228]

For the amount of space she'll require, it's probably prudent to build a RAID 5 with 4 or 6TB drives. That's a lot of data to reallocate.

If you care about your data you don't want to use RAID5 anymore especially with drives that size.  It will take so long to rebuild the array that you will mostly likely lose another drive during the rebuild and your entire data set along with it.  Something like that you'd want to use RAID10 or if you're set on RAID5 have a complete backup somewhere else or run a 2nd mirrored array.

[Guest asrequested]

You prefer mirroring over striping? I didn't suggest 10 because you need twice the physical space. I was thinking 5, to use less drives, have decent read speeds and still have good redundancy.

[JeremyFr79 228]

You prefer mirroring over striping? I didn't suggest 10 because you need twice the physical space. I was thinking 5, to use less drives, have decent read speeds and still have good redundancy.

See this reddit post......https://www.reddit.com/r/sysadmin/comments/ydi6i/dell_raid_5_is_no_longer_recommended_for_any/

 

Using RAID5 on an array with member disks larger than 1TB each is in no way recommended, as I stated above the biggest issue is that with disks above 1tb each the rebuilds when you have a disk failure take so long to perform that your chances of losing a 2nd drive and in turn the entire array are extremely high.  Remember rebuilds especially in a parity RAID are insanely taxing on your drives you are reading/writing every bit from beginning to end of the spindle during a rebuild it's activity that makes normal array use pale in comparison in terms of IOPs and the physical work your drives go through.  

 

So again if you care about your data don't use RAID5, if you understand the risk that you will most likely lose all your data on the array after an initial disk failure but you want the benefits that RAID5/50 provide then by all means use it.  But don't use it because you think it's in some way protecting your data because in very real terms it most likely wont.  There really is no perfect RAID that gives you 100% protection of your data.  This is why you should always maintain multiple copies in multiple locations of data you actually care about.

 

For me I have all my important data mirrored on 2 separate arrays and that data is real time mirrored on 2 separate off site backup providers both provide file versioning as well (which paid off dividends recently when most of my wife's data became corrupted somehow).  The rest of my data like movies, tv, music is located on mutliple arrays but if I lose those data sets I'm not too worried about it.  I have learned the hard way what can happen when you rely on one solution for data integrity.

[Guest asrequested]

I'm pickin' up whatcha puttin' down 

[swhitmore 781]

if that's what you want, then what you should do is open a feature request topic asking that we store a cloned copy of all images underneath the server's program data folder. then we can use the cached versions while your drives stay sleeping until a full library scan or playback.

 

I like the sound of this, and wonder if it could help my own performance issues. Is it something that would likely happen if enough people request it?

Edited May 20, 2016 by swhitmore

[NWGiles 10]

Hello

 

i have several raided NAS making up my library, not much smaller than the OP in size, however my drives are set to spin down when not in use, as at least half the day they aren't used. I do find the very first use is a few seconds slower due to the waking of the drives but I wouldn't class it as a performance issue.

Normal use is pretty instant, metadata/images etc are on the ball. 

 

I would suggest moving away from USB drives to something a little more resistant. 

[rhodges 39]

See this reddit post......https://www.reddit.com/r/sysadmin/comments/ydi6i/dell_raid_5_is_no_longer_recommended_for_any/

 

Using RAID5 on an array with member disks larger than 1TB each is in no way recommended, as I stated above the biggest issue is that with disks above 1tb each the rebuilds when you have a disk failure take so long to perform that your chances of losing a 2nd drive and in turn the entire array are extremely high.  Remember rebuilds especially in a parity RAID are insanely taxing on your drives you are reading/writing every bit from beginning to end of the spindle during a rebuild it's activity that makes normal array use pale in comparison in terms of IOPs and the physical work your drives go through.  

 

So again if you care about your data don't use RAID5, if you understand the risk that you will most likely lose all your data on the array after an initial disk failure but you want the benefits that RAID5/50 provide then by all means use it.  But don't use it because you think it's in some way protecting your data because in very real terms it most likely wont.  There really is no perfect RAID that gives you 100% protection of your data.  This is why you should always maintain multiple copies in multiple locations of data you actually care about.

 

For me I have all my important data mirrored on 2 separate arrays and that data is real time mirrored on 2 separate off site backup providers both provide file versioning as well (which paid off dividends recently when most of my wife's data became corrupted somehow).  The rest of my data like movies, tv, music is located on mutliple arrays but if I lose those data sets I'm not too worried about it.  I have learned the hard way what can happen when you rely on one solution for data integrity.

RAID6 - 2 times the parity. You lose the capacity of 2 drives, but, whatever. If you lose 1 drive, then during a rebuild if you lose another drive, you are fine.

 

I have run RAID 6 first with 500GB, then 1TB, 2TB and now 4TB drives. It is a hardware raid card and it is a little old. I am would like to get a newer one for faster rebuilds.  I have about 36TB of usable diskspace and it takes 20 or 26 hours to check the array, and a few days to rebuild if needed.

 

Don't buy all your disks at once. Buy some from different vendors, etc, etc.

Edited May 20, 2016 by rhodges

[JeremyFr79 228]

RAID6 - 2 times the parity. You lose the capacity of 2 drives, but, whatever. If you lose 1 drive, then during a rebuild if you lose another drive, you are fine.

 

I have run RAID 6 first with 500GB, then 1TB, 2TB and now 4TB drives. It is a hardware raid card and it is a little old. I am would like to get a newer one for faster rebuilds.  I have about 36TB of usable diskspace and it takes 20 or 26 hours to check the array, and a few days to rebuild if needed.

 

Don't buy all your disks at once. Buy some from different vendors, etc, etc.

RAID 6 doesn't offer much more protection than RAID5, yes using different drives could help with losing the array during a rebuild but introduce their own issues.  with RAIDs it is always best to have all the same drives with all the same firmware.  Mixing and Matching different drives with different performance characteristics etc again can cause it's own issue.  You can of course use the same drives but with different manufacture dates but that only marginally lowers your chances of losing a complete data set.  I'm not saying you can't get through a rebuild "unscaved" I'm just trying to make it very clear that your chances of doing so are very low.  You also run the very real risk of losing more disk's in the near future after an initial disk loss and rebuild essentially snowballing on you.