How do YOU backup/recover your Emby System ?

[rbjtech 5284]

Why not setup DrivePool on the destination system as well?  This is what I do.

 

Actually my storage needs for backup are in the hundreds of TBs so I backup to 3 different destination systems made up of "smaller" NASes or servers.

I've got one backup for movies, another for TV Shows and the 3rd for everything else (sports, home video, etc).  I did this to make use of existing hardware I already had and it works just fine for my needs.

 

Yep - that's the plan based on some great experience and thoughts on this thread -  2 'pools' - one per NAS and then manually replicate the two. 

 

 

wow thats quite impressive. you literally couldnt be any more wrong!!

 

no media within emby is important or private. 90% of it could be recovered in a few weeks. if you read my other post you would see that i say i backup photos and music (that which is important) locally and keep a copy off site.

 

please tell me where i can get 120Tb of local storage for £70 a year? 

 

Agreed - and again, with thoughts in this thread, I didn't realise cloud storage was so cheap.  BUT being a cynic, I'm not sure if these cloud storage providers can sustain this model at this cost, remember Amazon had unlimited storage, as did Microsoft - both pulled the plug after about a year as it was not a cost effective business model.

 

For average Joe - 1-2 Tb max, it works well - for us Media types then clearly not.  120Tb for £70 per year is a fraction of the disk cost, yet alone powering and maintaining it. ! (and remember THEY need to have a backup of it too - so 240Tb for them ...   )..

 

Maybe use it while it lasts .. haha

Edited March 12, 2019 by rbjtech

[Carlo 4561]

@@cayars has a library that makes mine, which I think of as pretty big, look quite small. Any advice he gives for library management is probably quite good and you cannot go wring by following it.

 There are a lot of people here with rather large libraries and we all do things a bit differently (cause we can).  Listen to all the advice you get from everyone and then use that knowledge in your own environment.

 

im backing up 120tb to the cloud [emoji3]

 I've backed up to the cloud as well but no longer do this for media at large.  I still backup pictures and other small files this way as I can share with other family members.

 

I would never ever use the "cloud" for anything important or private. There are way too many ways thing can go horribly wrong with files that are out there for anybody to find and use. Also companies maintaining the cloud systems are not trustworthy and are not reliable in the long run.

 

Local storage is cheap and MUCH more secure and reliable.

However, I've been burnt far to many times to rely on them as my sole backup. I've had more then a few hundred TB of data on each:

Bitcasa

Microsoft

Amazon

Google

and have lost it all on each of them. The first 3 did away with their "unlimited" storage (can't imagine why) LOL

 

The Google account got messed up by the software I was using (Stablebit CloudDrive) and then again by some scripts other use.  I then had to switch from FIOS with 400 Mb upload to Xfinity with only 10 Mb upload so cloud storage is no longer viable even for playing around with for me.  I only play with it for Emby syncing now.

 

Just remember, any backup is better than no backup.

Having a backup in house may not be a backup at all in some cases (ie basement flood if all equipment is there).

If you backup in house try to locate the backup equipment if at all possible in a different part of the house (see above). 

[JLJ 1]

Why not setup DrivePool on the destination system as well?  This is what I do.

 

Actually my storage needs for backup are in the hundreds of TBs so I backup to 3 different destination systems made up of "smaller" NASes or servers.

I've got one backup for movies, another for TV Shows and the 3rd for everything else (sports, home video, etc).  I did this to make use of existing hardware I already had and it works just fine for my needs.

 

Yep - that's the plan based on some great experience and thoughts on this thread -  2 'pools' - one per NAS and then manually replicate the two. 

 

 

 

Agreed - and again, with thoughts in this thread, I didn't realise cloud storage was so cheap.  BUT being a cynic, I'm not sure if these cloud storage providers can sustain this model at this cost, remember Amazon had unlimited storage, as did Microsoft - both pulled the plug after about a year as it was not a cost effective business model.

 

For average Joe - 1-2 Tb max, it works well - for us Media types then clearly not.  120Tb for £70 per year is a fraction of the disk cost, yet alone powering and maintaining it. ! (and remember THEY need to have a backup of it too - so 240Tb for them ...   )..

 

Maybe use it while it lasts .. haha

@@rbjtech This sounds like a winner here, I normally just backup everything via 1 10TB external hdd plus another stored offsite.And I change them out weekly.

I have about 6TB's total data as of now.But this is growing rapidly(Movies/TV Shows/Home content).So building another server would eliminate exchanging hdd's weekly.

I Have always been skeptical of cloud backup but... I was looking at BackBlaze($60/YR) and Carbonite($72/YR) for their unlimitied storage plans.Any thoughts?!

I think this combo would complete my backup scenerio. Great Info here.

Edited March 12, 2019 by JLJ

[awdspyder 11]

Much like others here, I use a combination of SAN/NAS with RAID for primary storage and JBOD with Stablebit for backup.  I suspect mine's somewhat different than most in the implementation, however, in that mine's all home-grown. 

 

Primary storage:

• SuperMicro 2U CSE-216 (24 x 2.5" + 2 x 2.5" rear SAS/SATA) Linux NAS/SAN 
• SuperMicro 2U CSE-216 (24 x 2.5" SAS/SATA) JBOD disk enclosure via SFF-8088
• SuperMicro X9SCM-F
• Intel Core i3-2130
• 8 GB ECC RAM
• LSI SAS 9211-4i in IT mode
• LSI SAS 9200-8e in IT mode
• 2 x 60GB SSD for OS in RAID-1
• Linux MD RAID
• 8 x Samsung Enterprise SSD block storage 2 x RAID-10 2+2 (VMs), 4 x ST4000LM024 4TB in RAID-10 block storage RAID-10 2+2 (VMs), 16 x ST4000LM024 4TB in 4 x RAID-10 2+2 (CIFS), 4 x ST5000LM000 5TB in RAID-10 2+2 (CIFS)
   
• NAS is served up via Samba/CIFS with mergerFS overlayed, so ~40TB mergerFS pool.  No LVM striping across RAID groups (Though I wish I had the guts to  ), so performance is limited to a single RAID group (~230 MB/s R/W)
• SAN is block storage via LIO using Qlogic QLE2464 4GB FC with direct fibre connections to 2 HP DL380 clustered Hyper-V servers
• Emby runs in a Windows virtual machine on the Hyper-V cluster

This configuration recently saw 685 days uptime (on UPS of course) until an extended power outage forced me to power it down gracefully (Grrr...).  I've also been able to expand my block SSD/spindle storage for VMs as well as the mergerFS pool for media from 10TB to 50TB with zero downtime.  It could just as easily support ZFS (with more RAM), hardware RAID, SnapRAID, etc.

 

 

 

Backup storage:

 

• SuperMicro 3U CSE-836 (16 x 3.5") Linux SAN
• SuperMicro X9SCM-F
• Intel Core i3-2130
• 4 GB ECC RAM
• LSI SAS 9211-8i in IT mode
• 2 x 60GB SSD for OS in RAID-1
• 5 x 4TB WD Blue, 2 x 6TB WD Blue
   
• SAN block storage via LIO using QLogic QLE2464 4GB FC with direct fibre connections to 2 HP DL380 clustered Hyper-V servers
• Backup server runs in a Windows virtual machine on the Hyper-V cluster with StableBit DrivePool/CloudDrive
• Backup SAN/VM power up each night via script to run custom backup scripts for Hyper-V, shares, media, etc.  Otherwise, servers are powered down.

As others have mentioned the beauty of DrivePool is that it accomplishes several goals, that I, like you had:

1. No striping to prevent more than a single disk's worth of data loss.
2. Employs disk pooling, so "drive" appears as a single large NTFS-based drive for backup targets.
3. Data is easily moved/shuffled/recovered at the disk level via normal NTFS tools
4. I know exactly what backup drives contain which files
5. Drives can be removed and data viewed in any NTFS-capable system.
6. Truly important data (documents, pics, etc.) can be replicated to 2 or 3 separate disks in the pool so a single drive loss is not so impactful.

It's worth noting that I could have achieved the same result with a 2nd box running mergerFS/CIFS, but I like having disparate things to play with, and I wanted a Windows box to run PowerShell scripts, etc.  Bottom line is you need a second server to replicate the data.  Keep in mind that the scripts need to be conservative to reduce human error - I have mine only sync new data, but do not MIRROR the arrays.  This prevents an accidental deletion from being replicated to the backup device.  I then have other scripts that run to remove obsoleted files, once the list has been manually vetted.   

Edited March 12, 2019 by awdspyder

[Carlo 4561]

It's worth noting that I could have achieved the same result with a 2nd box running mergerFS/CIFS, but I like having disparate things to play with, and I wanted a Windows box to run PowerShell scripts, etc.  Bottom line is you need a second server to replicate the data.  Keep in mind that the scripts need to be conservative to reduce human error - I have mine only sync new data, but do not MIRROR the arrays.  This prevents an accidental deletion from being replicated to the backup device.  I then have other scripts that run to remove obsoleted files, once the list has been manually vetted.   

 

 

I'll add something here to this conversation.  These "backups" don't need to be performed every night.  Maybe once a week or even once a month is fine for many system.  At worse case you would only loose data from the last time you backed up.

 

You also don't need to spend a lot of money (other than the discs themselves).  As an example you could add a "box" like this: https://www.newegg.com/Product/Product.aspx?Item=9SIA6ZP8TZ9101&Description=8-bay%20enclosure&cm_re=8-bay_enclosure-_-17-707-367-_-Product

which gives you hotswap drives.  Available in 2 to 8 bay and range from roughly $100 to $300 in price.

 

I like these "dumb" boxes because they are purely JBOD and can be accessed via external SATA or via USB.  You could plug this box into your server or plug it into an old laptop or even your home router and have 2 to 8 "external" drives available for backup.  May not be the fastest setup in the world but will work very reliably and get the job done.  If you plug something like this into any old computer you could run drivepool on it and mount one big drive for your server to backup to.

 

Fill it up and you can pull all 2 to 8 drives and keep them as a set (labeled) and even move the drives off site.  Put in 1 or more new drives and start a new pool for more backup ability.

 

You could have 50 drives in your main server but only ever need the 2 to 8 drives (whatever) mounted in this box to cover your new media additions.  So don't think you must have a "duplicate" system running in real time as you don't.  You just need a way to have storage available to backup new stuff. AND to be able to easily recover data if needed, hence why I like working with sets of 8 drives for bigger systems if going this route.

 

For smaller systems a 2 to 4 bay chassis may be all you ever need even without ever swapping the drives.

 

I know guys who have a stack of 4 TB external WD drives they use one drive at a time to fill then take offline/offsite.  What ever works to allow you to backup is good and doesn't need to be over complicated like some of the system we run.

[rbjtech 5284]

Thanks for all your input guys - it's been a really useful thread for me (and hopefully many others) to confirm how other people do this.  The bit I was missing was the drive 'pool' and I'm now going to use Stablebit for that, the rest (multi server, manual but automated replication (backup, not mirror) etc) I already had and that appears to be the best solution utilising all your old disks and infrastructure, so just costs you a bit of time to setup.

 

[scb99 245]

Interesting thread for me too. I have about 70TB of data so not so much compared to some of you. My Emby system(s) are my lowest tier backup. My master copies are kept offline on a shelf and are exactly the already mentioned 4TB WD drives. I have various data collection sources, I go through a routine to manually copy these to the appropriate master copy WD (e.g. "Orange5" … don't laugh...). Then I validate each of these to the first tier backup which is simply an array, with libraries corresponding to the master disks. Then the masters go back on the shelf safely offline for another month - I do this cycle about monthly. Then these libraries automatically sync to the Emby NAS's, which are in a different location. I am lucky enough to afford fast connections in both places. The online Emby arrays are now PR4100 with parity disk but no spare. So 30TB per box, 3 boxes with 90TB of which I am up to about 70TB used. Very simple but works and I feel safe, 3 copies of everything across two locations, and the master copy is offline and untouchable.

 

Just to add it's probably about half a day's work once a month to update the masters and sync them to the first line backup.

Edited March 13, 2019 by scb99

[awdspyder 11]

Just another data point: 

 

There's an Emby user on here that is a friend and was a coworker of mine who probably has one of the larger collections here (in the 100's of TB).  He also has a very fast uploads, so he and a good friend of his (who's ~100 miles away) simply sync their libraries together.  Of course, both have sizable investments in hardware, are committed to the platform, etc.  So this works as a backup of sorts since their libraries are nearly identical at any given moment.

 

I have a good buddy who would probably like to go this route with me, but unfortunately he can't get his wife on board with the hardware requirements. 

[qazwsx10 7]

 There are a lot of people here with rather large libraries and we all do things a bit differently (cause we can).  Listen to all the advice you get from everyone and then use that knowledge in your own environment.

 

 I've backed up to the cloud as well but no longer do this for media at large.  I still backup pictures and other small files this way as I can share with other family members.

 

However, I've been burnt far to many times to rely on them as my sole backup. I've had more then a few hundred TB of data on each:

Bitcasa

Microsoft

Amazon

Google

and have lost it all on each of them. The first 3 did away with their "unlimited" storage (can't imagine why) LOL

 

The Google account got messed up by the software I was using (Stablebit CloudDrive) and then again by some scripts other use.  I then had to switch from FIOS with 400 Mb upload to Xfinity with only 10 Mb upload so cloud storage is no longer viable even for playing around with for me.  I only play with it for Emby syncing now.

 

Just remember, any backup is better than no backup.

Having a backup in house may not be a backup at all in some cases (ie basement flood if all equipment is there).

If you backup in house try to locate the backup equipment if at all possible in a different part of the house (see above). 

I don't mean to resurrect this thread, but I currently have a physical duplicate backup of all of my media, but since it's starting to grow, I may not try to continue to do this. I do have a Google account with unlimited storage, but I am slightly concerned about them flagging any of my content as pirated (even though it's not) and deleting my account...

[awdspyder 11]

I don't mean to resurrect this thread, but I currently have a physical duplicate backup of all of my media, but since it's starting to grow, I may not try to continue to do this. I do have a Google account with unlimited storage, but I am slightly concerned about them flagging any of my content as pirated (even though it's not) and deleting my account...

 

Knowing what I know about Google/Amazon/Microsoft, etc., I would say without a shadow of a doubt that they would all be cognizant of exactly what you have stored there.  Whether or not they care enough to do anything about it is another matter.  If it were me, I'd take a hard look at something like Backblaze or even the Backblaze B2 environment.  I personally know Plex users that have piles of data out there for very nominal costs. Perhaps it's just wishful thinking, but I still feel like they'd be less likely to snoop than the big guys.  Of course, even Backblaze would also give up the goods on you in a heartbeat if they were subpoenaed.

 

Given that your data is legit, you shouldn't have much to worry about legally either way, but I don't think that Backblaze will just knee-jerk delete your data.

Edited September 13, 2019 by awdspyder

[Deathsquirrel 745]

One thing to consider with online backups is your ISP's data caps, if any.  Most of the major internet providers in the US only allow you to download a TB per month.  As a Comcast customer with about 70TB on my NAS recovery would cost me a lot and/or take forever.

Edited December 9, 2019 by Deathsquirrel

[Gilgamesh_48 1240]

One thing to consider with online backups is your ISP's data caps, if any.  Most of the major internet providers in the US only allow you to download a TB per month.  As a Comcast customer with about 70TB on my NAS recovery would cost me a lot and/or take forever.

Actually I have noticed that "most" ISPs in the US do not have any data cap and many hat do are dropping them.

 

Comcast is the exception rather than the rule. AT&T is another with caps but Charter, mostly, and most small ISPs do not have them also even those that do have run afoul of State and  local laws and have had to drop them in many areas.

 

BUT, I still think using online backup for anything important is usually foolish as you have no control and there is really nothing to stop the ISPs and/or data warehouses from snooping into your data except some, probably virtually unenforceable, "agreement" that has the real legal power of the "Finders Keepers" saying. Local storage has become so cheap that you can buy enough space to store almost anything cheaper than one or two years worth of online storage and you have quick access to it and do not have to download it and wait for the download to finish. Even if the online storage were cheaper it still would make no sense in that it would be a case of "Penny wise, pound foolish."

[scb99 245]

A LOT of sense in the above post.