Jump to content

Media Storage Solutions


bnwbass

Recommended Posts

bnwbass

Hello all,

 

Asking the Emby team and community for some advice and best practices regarding data storage of all things media.

I am at the point where I have many HDD installed in two-three machines via network shares etc. Organizing, merging and long term storage is now issue that I need to address.

 

I have supn up a new win-10 pro machine with a solid CPU and memory planned for new Emby server environment. I have build many Emby environments before so my question is focused more on how others build their systems knowing that managing large data files and associated directories is inherent to building these system. I am starting with installing (4) 8TB disks to accomodate my entire collection of media files and data. This is mixed content ( home photos, AVI, Mp4, ISO, VOB etc.) so when I move files from 10-15 HDD over to the (4) 8TB disks I will organize later.

However, at this point I am considering "drive pool" by stablebit or possib MS storage space and then using something like snapRAID as backup solution.

 

From the Emby standpoint, what is best practice and/or some products that should be included when building the most robust media server possible when it comes to disk storage configurations?

Thx,

Bw

 

@@cayars

Link to comment
Share on other sites

Forget about Storage spaces.  Go with Drivepool.  DP works really well in our type of environment.

 

What you might want to consider is possibly getting a new case that holds a dozen or so drives or get a cabinet just for drives.

I use these guys https://www.startech.com/support/S358BU33ERM

S358BU33ERM.E.jpg

That box runs roughly around $300 US.  It holds 8 hot-swappable 2.5 or 3.5" drives.  It has it's own power supply and connects to your Emby Server via eSATA, USB 3.0 or UASP.

 

To the computer connected to the box each drive appears separately as if it was installed in the computer case.  So this is NOT a NAS box but a dedicated box to hold and power HDDs with eSata or USB connection to the computer.  It's a nice easy way to add drives with proper cooling to your system.

 

So I use those boxes with Stablebit DriveSpace after trying many different RAID and NAS boxes.   This is just dirt simple and works.

  • Like 1
Link to comment
Share on other sites

bnwbass

Thanks cayars for the reply, I built one machine with flexraid and even though no real issues, there is more sensible options over raid today.

I am going with Stablebit.  What about a crash plan/ DR plan, are you willing to share backup strategies that you have used over the years for large media data files?

 

Thjx

Bw

Link to comment
Share on other sites

Gilgamesh_48

Forget about Storage spaces.  Go with Drivepool.  DP works really well in our type of environment.

 

What you might want to consider is possibly getting a new case that holds a dozen or so drives or get a cabinet just for drives.

I use these guys https://www.startech.com/support/S358BU33ERM

 

That box runs roughly around $300 US.  It holds 8 hot-swappable 2.5 or 3.5" drives.  It has it's own power supply and connects to your Emby Server via eSATA, USB 3.0 or UASP.

 

To the computer connected to the box each drive appears separately as if it was installed in the computer case.  So this is NOT a NAS box but a dedicated box to hold and power HDDs with eSata or USB connection to the computer.  It's a nice easy way to add drives with proper cooling to your system.

 

So I use those boxes with Stablebit DriveSpace after trying many different RAID and NAS boxes.   This is just dirt simple and works.

I agree completely about StableBits DrivePool. It simply is the best software, as far as doing what it is supposed to do, that I have ever owned.

 

I do not even bother about an enclosure for multiple drives. I simply use external USB drives and a couple of USB expanders and I have, currently, 13 drives varying in size from 3tb to 10tb connected via USB and it all runs very smoothly. I use an old rack that I salvaged from a convenience store that was use for display and everything including a UPS sits in that rack. The wires with all the USB cables and power cables are a bit of a mess but my server lives out of sight in my spare bedroom/workshop/office so appearance is of little import. If you decide to use external drives like I do remember that external drives are not well cooled so you might want to, as I do, have a small house fan blowing across the whole mess. I have never had a drive fail from heat by doing that even when the temp in the room gets into the 80s. (Fahrenheit) eight of my thirteen drives have been in continuous use for three years and four of those for almost five and two are coming up on six years soon.

 

DrivePool simply makes everything so easy that you almost forget it is running.

 

As far as figuring how much space you need it really is pretty simple. Use whatever method you think give you the best estiment and triple that and add a fudge factor. There are too general statements to keep in mind:

1. There is not such thing as too much storage space.

2. Data grows to fill available space.

 

I remember the first computer I ever bought. A TRS-80 with an external audio cassette for storage. I thought I had enough storage. Less than a year later I added 2 128kb single density 5 1/4 floppy drives and I thought that is all the storage I will ever need. The a couple of years later I bought an IBM PC with a 10mb hard drive and I thought what will I ever do with all that space. I later I added a 30mb drive thought the same. Then through many many computers and drives I went to a total of 2gb then 30gb then 1tb then 10tb then 20tb then through 30, 40, 50 and now I am approaching 90tb in total drives attached to three or sometimes four computers. I have never again though I was not going to fill what I have, it was just a question of when. I am trying to figure when I will need to add my next 10tb.

 

BTW: In many ways I miss that old TRS80. :D:P:rolleyes:

Link to comment
Share on other sites

bnwbass

Thx for comments, I will read up on DrivePool, but my understanding is the no-reformat required to configure HDD.

Link to comment
Share on other sites

clarkss12

I agree completely about StableBits DrivePool. It simply is the best software, as far as doing what it is supposed to do, that I have ever owned.

 

I do not even bother about an enclosure for multiple drives. I simply use external USB drives and a couple of USB expanders and I have, currently, 13 drives varying in size from 3tb to 10tb connected via USB and it all runs very smoothly. I use an old rack that I salvaged from a convenience store that was use for display and everything including a UPS sits in that rack. The wires with all the USB cables and power cables are a bit of a mess but my server lives out of sight in my spare bedroom/workshop/office so appearance is of little import. If you decide to use external drives like I do remember that external drives are not well cooled so you might want to, as I do, have a small house fan blowing across the whole mess. I have never had a drive fail from heat by doing that even when the temp in the room gets into the 80s. (Fahrenheit) eight of my thirteen drives have been in continuous use for three years and four of those for almost five and two are coming up on six years soon.

 

DrivePool simply makes everything so easy that you almost forget it is running.

 

As far as figuring how much space you need it really is pretty simple. Use whatever method you think give you the best estiment and triple that and add a fudge factor. There are too general statements to keep in mind:

1. There is not such thing as too much storage space.

2. Data grows to fill available space.

 

I remember the first computer I ever bought. A TRS-80 with an external audio cassette for storage. I thought I had enough storage. Less than a year later I added 2 128kb single density 5 1/4 floppy drives and I thought that is all the storage I will ever need. The a couple of years later I bought an IBM PC with a 10mb hard drive and I thought what will I ever do with all that space. I later I added a 30mb drive thought the same. Then through many many computers and drives I went to a total of 2gb then 30gb then 1tb then 10tb then 20tb then through 30, 40, 50 and now I am approaching 90tb in total drives attached to three or sometimes four computers. I have never again though I was not going to fill what I have, it was just a question of when. I am trying to figure when I will need to add my next 10tb.

 

BTW: In many ways I miss that old TRS80. :D:P:rolleyes:

You must be almost as old as I am. It was commonly called a trash 80.(:
Link to comment
Share on other sites

Gilgamesh_48

You must be almost as old as I am. It was commonly called a trash 80.(:

Yet it was just about the most powerful computer you could buy for home use. In simple speed of processing it put its nearest competitor, the Commodore 64, to shame. Yes, there were more games for the competition but I never bought computers for games. It had a pretty good word processor and a built in Basic interpreter and an assembly compiler that both were very good.

 

I wish I would have had the forethought to keep that old TRS80. I saw one the other day on Ebay sell for a little less that $50,000. Some things from that time period have become very valuable. I did keep one thing from my childhood and I may just leave it for my granddaughters. I have a 78RPM copy of "You ain't nothin' but a hound dog" better known as simply "Hound Dog" recorded by Elvis in 1956. It is in near mint condition as 45 rpm records became common and I usually played the 45 instead of the 78 as I grew up.

Link to comment
Share on other sites

You must be almost as old as I am. It was commonly called a trash 80.(:

I had a TRS 80 Model 1 and that was my 3rd computer. :)

Link to comment
Share on other sites

maegibbons

So what were your first two?

 

Krs

 

Mark

 

A 'like' is always appreciated!

Link to comment
Share on other sites

maegibbons

Commodore PET for me too with Square Keyboard and Blue border screen Serial Number 5 in the UK.

Apple ][ after that.

 

Krs

 

Mark

 

A 'like' is always appreciated!

Link to comment
Share on other sites

clarkss12

I had a TRS 80 Model 1 and that was my 3rd computer. :)

Wow!!  You are OLD>........ :)

  • Like 1
Link to comment
Share on other sites

pwhodges

The first computer I used, at university in 1966, was an English Electrric Leo Marconi KDF9 .  The one I used in my first (temporary) job in 1968 was also a KDF9 - in fact the very one in this video, and my job was to rewrite for greater efficiency (I achieved a 20-times speed improvement) the graphical output system  talked about in the video.

 

Paul

Link to comment
Share on other sites

mholin

Back to the original topic:

I recently moved my system from am Shuttle XPC with I7 3.5 GHz and attached 4x USB Box with 4x 8TB Seagate consumer drives with no RAID

 

to a Fujitsu TX200 S7 Server, which is able to host 8x 3.5 drives. 

I built this myself with 8x 6TB Seagate Exos Enterprise drives. The drives are attached to a 10 years old LSI 9260 8i Controller.

 

I set up one large Raid6 over all 8 hdds. This gives a total of around 30 TB with security, that 2 of the 8 HDDs may fail, and all data is still there.

On the 30 TB host drive I set up 4x virtual VHDX as my older single drive letters.

 

The VHDX drives can be written at about 200 MB/s from SSD. 

 

mirgation of my existing emby library was free of hassle, almost all pictures and metadata was same as before.

Copying of 24 TB of cause took some days.

 

I guess this is now quite stable. 

 

bye Michael

Link to comment
Share on other sites

Sorahl

for myself, i use https://smile.amazon.com/dp/B078YQHWYW/ref=sr_1_19?dchild=1&keywords=HDD+enclosure&qid=1588694741&sr=8-19

this is the second gen, i have two working Gen 1 (one dead gen 1) and 1 Gen 2. I need to get approval from the CFO to purchase a couple more Gen2 so i can move the other 5 drives in my mediacenter pc out of it, leaving just the Boot drive, and i'm thinking about replacing that boot drive with a 2 tb usb stick when i rebuilt this mediacenter. 

Link to comment
Share on other sites

braigo64

I use Unraid https://unraid.net/ to manage my media server.  There is a free version and a one-time cost pro version.  It's quite simple, but can be very powerful as the community has created a wealth of plugins including running docker containers of whatever you want really.  But for me all I have is a large tower case, a motherboard with plenty of sata ports, a cpu, ram, and psu. The Unraid "OS" just sits on a usb stick. I've been using Unraid for over a decade now.  In fact I've been using the same motherboard and cpu combo the whole time too. So it doesn't need powerful hardware to run. If you plan on doing more with it than just hosting disks then certainly budget to mid hardware these days should suffice. 

 

If a disk goes bad or I've run out of total space, I just shutdown, swap out the oldest drive with a new drive, power up and let it do it's thing. No data loss, no problems.  I started out back then with 500GB and 1TB drives. As you can see in the screenshot I'm up to 4 and 6 TB drives. And a 24TB total server size.  And you can see my old single core AMD Sempron CPU and 2GB of RAM.

 

5eb1a077e325b_unraid_1.png

 

5eb19e9e3cbb7_unraid_2.png

Link to comment
Share on other sites

BAlGaInTl

I use Unraid https://unraid.net/ to manage my media server.  There is a free version and a one-time cost pro version.  It's quite simple, but can be very powerful as the community has created a wealth of plugins including running docker containers of whatever you want really.  But for me all I have is a large tower case, a motherboard with plenty of sata ports, a cpu, ram, and psu. The Unraid "OS" just sits on a usb stick. I've been using Unraid for over a decade now.  In fact I've been using the same motherboard and cpu combo the whole time too. So it doesn't need powerful hardware to run. If you plan on doing more with it than just hosting disks then certainly budget to mid hardware these days should suffice. 

 

If a disk goes bad or I've run out of total space, I just shutdown, swap out the oldest drive with a new drive, power up and let it do it's thing. No data loss, no problems.  I started out back then with 500GB and 1TB drives. As you can see in the screenshot I'm up to 4 and 6 TB drives. And a 24TB total server size.  And you can see my old single core AMD Sempron CPU and 2GB of RAM.

 

 

 

 

 

 

I'm also an Unraid user. ETA - It's also an easy to use drive pooling solution.  

 

I have some shares that I consider critical and back them up off site and on the cloud.

 

For my media, I don't really backup.  I have redundancy, and even if multiple disks fail, I wouldn't lose everything.  None of the media is important enough to me to spend a lot of money on a complete backup solution.  I own most of it and could re-create it if I absolutely had to.  It would suck... but I could do it.

Edited by BAlGaInTl
Link to comment
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now
×
×
  • Create New...