Jump to content

Pooling drives with storage spaces


1971camaroguy

Recommended Posts

CBers

You can just share out the F: drive or any sub-folder in Windows Explorer, like you can with any other drive.

 

That way, you should be able to see it across the network.

Link to comment
Share on other sites

1971camaroguy

You can just share out the F: drive or any sub-folder in Windows Explorer, like you can with any other drive.

 

That way, you should be able to see it across the network.

Thanks yeah that did it, it hit me a sec ago and I changed the settings and came on here to tell what I did lol

 

Thanks for the heads up..that was a rookie mistake, I should have done that right off the bat lol

  • Like 1
Link to comment
Share on other sites

Fratopolis

I personally also lost data a little less than two years ago with storages spaces. I'm sure it's come a long way but decided to move to a Enterprise class raid card. They are too cheap to pass up and take chances on things like this again.

 

I say why chance it when 40-50 bucks will get what you need.

  • Like 1
Link to comment
Share on other sites

1971camaroguy

I have a handle on drivebender right now, but man is this storage pool slooooow to copy files now. I went ahead and designated one of my larger drives as a Landing spot for the files. But a 40 gb copy from another drive on the same computer is taking 25-30 minutes.

 

It seems like the software is slowing things down a bit. I have 16gb ram on one computer with an 3.2ghz i5 processor and a server grade motherboard with a Xeon CPU- E3-1220 3.10 ghz processor and 24 gb of ram and both are using about 50% of the available resources.

 

And it seems like simply making folders and renaming them within the pool is acting glitchy and slow. I get a "not responding" prompt for a split second until it catches up.

 

I like the idea of large storage, but I miss how fluid it was with my old set-up. Maybe this is the norm for storage pools and I have gotten spoiled.

 

I'm going to toss an ssd in the mix sometime before the weekend and make that the landing spot and see if that changes anything

Edited by 1971camaroguy
Link to comment
Share on other sites

CBers

@@1971camaroguy I have a 20Tb (4 drives) and a 8Tb (4 drives) pool setup on the one server serving different applications.

 

File enumeration used to be slow but is much faster with the latest version.

 

Do you have your disks set up to go to sleep? If they do, then that may explain the slowness.

 

If not, something else is going on.

 

You could join the DriveBender forums and/or raise a Support Ticket.

  • Like 1
Link to comment
Share on other sites

1971camaroguy

I don't have them going to sleep that I am aware of..but I till def double check that. Thanks - I'll join the drivebender forums and search around there before asking some questions

 

The only thing that "could" be causing an issue is I am using goodsynch and trying to back up my two computers, most importantly the boxed set files...I have about 1.5 - 2 tb left to copy. It's taking a while and that could be what's hosing up my pool.

Edited by 1971camaroguy
Link to comment
Share on other sites

CBers

@@1971camaroguy Just tested copying 176Gb data between my 2 pools and it took 25 minutes.

 

You may find a few familiar face or two over on the forums :)

 

If you do raise a Support Ticket, let me know the reference and I'll speak to the dev about looking at it.

Link to comment
Share on other sites

1971camaroguy

Thanks, I have been browsing around over there and went ahead and bought two licenses

 

What I found out so far is my main computer is running pretty good

 

My backup computer is getting I/O errors on a couple of drives and doing some weird stuff which would explain some of my frustration. I haven't pulled the drives yet to check them. But on that computer I am using two 4 port non raid sata cards to be able to max out my drives. I am thinking one of the cards could be bad, or my power supply is failing in the computer.

 

I have since removed drive bender and put my windows 7 image back on that computer and still having issues with a couple of drives so that could be it. But I didn't have this issue until I upgraded to win 10 and installed drive bender...this set-up had been running for a few years with windows 7 and the drives seperate with thier own driver letters....just weird

 

 

But on my main computer it's running pretty decent with the storage pool, I am still on the fence if I want to pool all of my drives (including the other drives over the network) and do backup with drivebender across all of them, or just keep the drives on the main computer pooled....and backing up to individual drives on the 2nd computer with goodsynch. 

 

It would be more simple to do it that way, I don't really cared if the drives on my backup computer are pooled just as long as they are just replicating copies off my main pool.

Edited by 1971camaroguy
  • Like 1
Link to comment
Share on other sites

It seems you might have settled on DriveBender.

 

I haven't tried the latest but did run the previous version for a while.  I've also used RAID 0,5,6,10 (hardware and software), Storage Spaces and a few different Linus/Unix products but keep going back to StableBit DrivePool for Plex/Emby use.

 

To me it's fast and just works.  It runs with system drivers vs user mode drivers like DriveBender (at least previous version) so it fully supports all windows commands including symlinks which I use.

 

I had previous high hopes for Storage Pools but it's not really designed for our type of use.  You can't add new drives like with DriveBender/DrivePool or pull drives and attach to another computer for recovery or anything like that.

 

Something I'll throw out to you.  You might want to consider looking into the freeware SnapRAID product instead of completely duplicating everything to another system.  With SnapRAID you can use the 2nd machine drives as PARITY drives to all the drives in your pool.  It's kind of like a backup in a sense if you have a couple of parity drives but you won't need to duplicate the storage 1:1.  For example if you had 100 drives in your pool, you could use 7 Parity drives and be pretty sure that normal failure wouldn't take you down.  You could loose any 7 drives and still be able to rebuild your media with SnapRAID in this fashion.

 

The good thing about both DriveBender and DrivePool without any type of backup whatsoever is that if a drive fails you only loose data on that one drive and it doesn't take out your whole pool.

 

Best of luck to you but I'd suggest giving DrivePool a test run after you get your drives fixed up.

  • Like 1
Link to comment
Share on other sites

Gilgamesh_48
...

 

The good thing about both DriveBender and DrivePool without any type of backup whatsoever is that if a drive fails you only loose data on that one drive and it doesn't take out your whole pool.

 

Best of luck to you but I'd suggest giving DrivePool a test run after you get your drives fixed up.

 

You don't even lose any data on a single drive failure with DrivePool if you use the duplication feature. You can turn on duplication by folder or by file. i use folder duplication for all my video folders that are at all hard to recreate. Since I started using DrivePool several years ago I have had a couple of my very old drives fail and I have lost no data.

 

But, as you know, using any of the pooling softwares or any of the raid systems does NOT mean you have your data backed up. It just means you have redundancy and protection against single (sometimes more) drive failure. Backup means a second safe copy often stored well away from the primary copy.

 

I do not bother with backups for most of my movie and TV library simply because recreating those are just a matter of reripping DVDs or rerecording the shows BUT there are some that I do backup since they are no longer available and I have some that I recorded from very old VHS tape and the tape is no longer playable.

 

Too many times I have seen people that thought they had a backup when what they had was redundancy and a disaster happened and they lost everything.

Link to comment
Share on other sites

Yes of course.

I had a friend with a hundred plus TB of data on his system that was duplicated to another computer and NAS.

 

He had a fire.  You can guess what happened.  All gone!

 

I had about 95% of the same content so he was fortunate.as he was able to bring his new server over and copy my media months later when he got back on his feet.

 

Just goes to show if your backup isn't offsite it's not a real backup. :)

 

I wouldn't use 1:1 duplication in pool software, but with SnapRAID and a few parity drives I'm prepared for drive loss but like my friend, not a catastrophic event.  To be honest my media would be the last thing I'd be worried about in that type of situation.  I do however keep all my original media in a different location. Same with my tapes from video cameras so I could redo all my family personal media.  I do also have a small NAS of about 50 TB in my garage (50 yards from my house) that I use for backup of computer desktop drives and also my family pics/videos and business/personal info.  So I do backup "offsite" things that are important to me besides my movies/shows.

 

I've also had everything backed up in the cloud on several different vendor but everyone of them got shut down or changed policies. I was a fan of Amazon Drive, but it is no more unlimited.

 

What I like about SnapRAID is that it isn't real-time parity but only updates the parity when you run it.  That may sound like a draw-back but to me it's really not as our media doesn't really change, we just add to it.  So if you ever accidentally delete folder or do some other bonehead admin thing you can easily recover/restore your media from parity if you haven't ran the parity update.  I seem to do a bonehead maneuver like this about once a year.  Either an accidental folder wipe or some script program I work on or use that someone else wrote, doesn't do exactly what I thought it should do.  Parity drives are great for this type of "backup".

Edited by cayars
Link to comment
Share on other sites

Anyone use Flexraid still? I do, though many have shied away because of the developers MIA activity and the licensing. Anyway, you can the drivepooling and raid array with Flexraid as well.

 

Mark

Link to comment
Share on other sites

  • 2 weeks later...
mlcarson

Does Emby work with Snapraid's native drivepooling?   It's read-only but that shouldn't matter for existing collection of media that isn't changing.   Looks like it's just done with symbolic links. 

Link to comment
Share on other sites

Does Emby work with Snapraid's native drivepooling?   It's read-only but that shouldn't matter for existing collection of media that isn't changing.   Looks like it's just done with symbolic links. 

Don't go that route.  You want your media folders writable so you can store your nfos and artwork there. :)

  • Like 1
Link to comment
Share on other sites

  • 2 months later...
charlesk

Don't go that route.  You want your media folders writable so you can store your nfos and artwork there. :)

 

@@cayars So you can't use Snapraid with Emby?

Edited by charlesk
Link to comment
Share on other sites

  • 2 weeks later...

SnapRAID has an INFERIOR Pooling feature.  That was specifically what I was referring to.

 

Use any of the other drive pooling software already mentioned.  THEN use SnapRAID for it's parity features (not pooling) along side the other pooling software.

Link to comment
Share on other sites

dapper

Currently I have five 8 tb hard drives with various folders set-up on each on with emby mapped to folders on each of the drives.

 

I also have a similar computer set up with the same size and number of hard drives doing a 1:1 mirror with a program I use called goodsynch to prevent data loss.

 

Has anyone had any good luck with storage spaces on windows 10?  Right now I my media broken down by genre and type spread out across the drives. Some drives have more storage than others and I am basically wasting space. 

 

It would be a better allocation of resources to combine all of the drives into one storage pool, and would be easier to work with too. I figure I could make one big E: drive one each computer and let goodsynch manage backups.

 

I'm just really paranoid about something going wrong lol....but if I am able to do this it will make life easier with managing my media. If anything I could pool the main computer with emby and leave the backup on the other computer as individual drives. 

 

 

I use Storage Spaces on Windows Server 2019 [have previously used on Win 10 and Win Server 2016] and I find it extremely powerful with excellent performance.  It's not hugely intuitive because it really requires setting it up with PowerShell to have everything working optimally for a specific build/use-case.

 

After a bunch of testing of basically every possible variation, I've found that it is almost always better to go for the least complex Storage Spaces setup possible.  I'm confident that my current HDD Pool can survive and completely self-heal within ~4 hours after losing any 1 disk without any intervention.  I can survive 2-disk failure without data loss as long as it's not in that ~4-hour rebuild (I'm willing to take that risk to not have to deal with the hassles of 3-way mirroring for now ... I may upgrade to 3-way mirroring in the future). 

 

In general:

  • Things to use [for maximum flexibility/performance/ease-of-use]: 
    • Mirror Spaces; Thin Provisioning; One Storage Pool per Disk Type; PowerShell
  • Things to avoid [for minimum headaches/hassles]: 
    • Parity Spaces; Tiered Spaces; Fixed Provisioning; Write Cache; Mixed SSD/HDD Pools; Storage Spaces GUI

 

 

And for my setup, I have 3 Virtual Disks (aka Storage Spaces) across 2 Storage Pools:

 

  1. Bulk Storage on HDD Storage Pool (5x 8TB HDDs)

    [this has all Media libraries]

     

    - 2-columns 2-copy (RAID-10 equivalent); thin-provisioned; no write-back cache; 8TB pool space unallocated to permit complete rebuilds; formatted in ReFS; Integrity Streams On for all data

     

     

  2. High-Performance Programs/Personal Data on SSD Storage Pool (4x 120GB SSDs)

    [this has Emby + Emby Metadata + daily config backups + similar] 

     

    - 1-column 2-copy (RAID-1 equivalent); thin-provisioned; no write-back cache; formatted in ReFS; Integrity Streams Off

     

     

  3. High-Performance Cache Data on SSD Storage Pool (4x 120GB SSDs)

    [this has EmbyCache + transcoding cache + active downloads + similar]

     

    - 4-column 1-copy (RAID-0 equivalent); thin-provisioned; no write-back cache; formatted in NTFS

 

The HDD Pool gets consistent sequential reads/writes between the SSD Pool at 450-550 MB/s and obviously saturates my gigabit network.

Link to comment
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now
×
×
  • Create New...