How do YOU backup/recover your Emby System ?

[rbjtech 5005]

[lightsout 156]

What do you mean by "Emby becoming ‘unavailable’"

 

Sounds like your talking about backing up your media library and not Emby itself?

 

I use snapraid which with my setup can recover one failed drive. Not the best but I ran with nothing for years so...

[rbjtech 5005]

I'm talking about having a totally separate COPY (in time, but not real-time) of the media used to provide content to Emby.

 

I can spin up a copy of Emby in seconds and restore the config - that's not the issue here, the issue is the Media sitting behind Emby of which I have probably spent 100's of hours (maybe even 1000's..   ) ripping, downloading, organising etc

 

A very small % is irreplaceable (home video/photo's are backed up multiple times in multiple formats in multiples places ..) but I'm referring to the Blu-Ray rips and more recently the 4K rips which would take MONTHS to re-rip.

[mastrmind11 722]

I mean, the only way is pure redundency, which costs $ but the only way to be sure.

 

FYI, your understanding of ZFS is incorrect, and is likely exactly what you are looking for.  For eg, I have 8 3TB drives (mirrored 2x4 mirror) in a single ZFS pool, which shows up as a single mount point.  Yes, I lose half my storage capacity using a mirrored approach, but this means I can lose 4 drives before I lose anything.  Yes, I know, raid/mirror/whatever isn't a true backup strategy, but the odds of losing 4 drives simultaneously means I have a house fire and I'll have more on my mind than my media collection.  TL;DR zfs is a truly amazing filesystem tbh.

 

If we agree on my original statement, then backups (incremental or otherwise) is a simple rsync command.  Here's a good post outlining what you might be looking for https://serverfault.com/questions/842531/how-to-perform-incremental-continuous-backups-of-zfs-pool

[rbjtech 5005]

Thanks for the link - going to read that now - I looked at ZFS in some detail so perhaps I need to go back and look at it again.

 

But as I said, disk failure is not of concern here - with respect, you could lose the contents of your entire mirrored ZFS pool in seconds with a rogue command but all 8 of your disks are perfectly healthy.

 

Perhaps 2 x rsync'd RAID 0 ZFS pools is what I am after but rsync of the equivalent JBOD's appears to be safer .. 

Edited March 9, 2019 by rbjtech

[videopalace 28]

I use "AOMEI Backupper Pro". And I do the real time sync with that program.

[JLJ 1]

Have you looked into Stablebit Disk Pool tech ? It's what I use, although I don't have nearly as much data as you at the present time.

You could keep multiple copies of data on different disk or pools to minimise data lose in the event of disk failure etc... In addition to your backup solutions.

 

https://stablebit.com/

Edited March 10, 2019 by JLJ

[Deathsquirrel 744]

I save my metadata with my media.  When I add a movie or series I have emby scan it, make any metadata or image changes I want, and thenback it up to a set of external drives.  The drives are kept disconnected most of the time.  I connect the latest unfilled drive as needed, usually about once a week, to backup new movies I've purchased.

 

It's not an automated process but it's an easy one.  I'm only buying a few movies/shows each week.

 

You're correct that restoration of my 60TB of stuff will take quite a while, but should it be needed I doubt the biggest concern I'll have will be the duration of unattended file transfer needed for restoration.

[rbjtech 5005]

Have you looked into Stablebit Disk Pool tech ? It's what I use, although I don't have nearly as much data as you at the present time.

You could keep multiple copies of data on different disk or pools to minimise data lose in the event of disk failure etc... In addition to your backup solutions.

 

https://stablebit.com/

 

If the pool replication works over a network, then this could be exactly what I'm looking for.  I can get a single pooled drive but effectively a RAID 1/Mirror over physically separate drive pools machines .. 

 

Thanks - will look into this 

 

*update* - looking very promising - you can create a 'clouddrive' in the cloud but this can also be a remote share - perfect - all pooled, replicated (on a schedule) and readable by any NTFS filesystem    

Edited March 11, 2019 by rbjtech

[JLJ 1]

If the pool replication works over a network, then this could be exactly what I'm looking for.  I can get a single pooled drive but effectively a RAID 1/Mirror over physically separate drive pools machines .. 

 

Thanks - will look into this 

There is a setting in which you can set up Folder or Pool file duplication under the Manage Pool tab.

I would assume that it can be done over the network but never tested it out myself. Hope this works out for ya. 

[BAlGaInTl 280]

I'm talking about having a totally separate COPY (in time, but not real-time) of the media used to provide content to Emby.

 

I can spin up a copy of Emby in seconds and restore the config - that's not the issue here, the issue is the Media sitting behind Emby of which I have probably spent 100's of hours (maybe even 1000's..   ) ripping, downloading, organising etc

 

A very small % is irreplaceable (home video/photo's are backed up multiple times in multiple formats in multiples places ..) but I'm referring to the Blu-Ray rips and more recently the 4K rips which would take MONTHS to re-rip.

My media is like yours. Most of it is not critical, and although it would take a long time re-create it, it can be done.

 

I use SnapRaid + MergerFS. It's great for big files that don't change a lot. You can have multiple disk parity.

 

For my photos and home videos, the stuff that isn't replaceable, I use a separate share that gets rsynced to a different server. I also occasionally make copies of that share and take it offsite for storage.

 

So with SnapRaid... even if I lost a disk, I wouldn't lose my whole collection. I would only have to rebuild what was on the failed disks.

[legallink 187]

I back up using backblaze. Have had to restore a few files here and there and it’s been pretty speedy.

[rbjtech 5005]

I back up using backblaze. Have had to restore a few files here and there and it’s been pretty speedy.

 

Thanks, had a look at this but there has got to be some smallprint somewhere with 'fair use' as while they advertise 'unlimited', there has to be a limit.  It appears they make their money on the download charges as it's not free to download everything you have uploaded haha.. but it's another cloud option for my irreplaceable files. (< 1Tb's worth)

[Gilgamesh_48 1132]

I use StableBit's DrivePool with duplication on for all my Media files and I copy the database into a duplicated folder in that pool on a regular basis. It is a very bad idea to run Emby so that any dynamic files are part of the pool. I keep a copy of files that I cannot recreate in an off-site backup so as to protect those files from loss. I also, except for the database, do not backup any of the Emby created folder structure.

 

I do NOT backup most of my media (Recorded TV shows being an exception) as I have the original source (or a copy I made from old VHS or BetaMax tapes) stored away from my house in a safe location so, should the there be a disaster of some sort, I can recreate most of my library directly from the original source. If there is a disaster that destroys my file server system my media library is probably pretty low on the "worry" list so the fact that it will take some time to re-rip most of my library is pretty unimportant.

 

It should be noted that I do have a quite large, but not as large as some, library consisting of over 32000 TV episodes, over 3000 movies, 100s of pictures and 100s of audiobooks. I do not use Emby for music at all.

 

If anyone wonders how I can have original disks/tapes for such a large library it really is pretty simple: I inherited much of the total volume for an uncle many years ago and I already had a good size library. I have also added to that library ever since.

 

One more thing about StableBit's DrivePool: It is not really a backup solution but it is a really good solution for creating strong redundancy. Since I have been using it (5 years or so) I have only had one disk fail (A 4 tb one) and I simply ordered a new one and plugged in into the pool. I lost no data at all. DrivePool can be configured (I think it is the default) where it assures that files are always duplicated on different drives.

 

My system is not perfect but, for me, it is close enough to provide me with security that is good enough.

[legallink 187]

Thanks, had a look at this but there has got to be some smallprint somewhere with 'fair use' as while they advertise 'unlimited', there has to be a limit.  It appears they make their money on the download charges as it's not free to download everything you have uploaded haha.. but it's another cloud option for my irreplaceable files. (< 1Tb's worth)

So, I've never been charged anything for downloading a file.  I have roughly 30TB's backed up, so if I am outside of fair use, they haven't said anything.  As for restoring, you have the option of having them ship you a drive or downloading and if you ship the drive(s) back to them, they refund you the $$ for free.  It seems pretty straight forward to me but maybe I'm missing something.

[rbjtech 5005]

So, I've never been charged anything for downloading a file.  I have roughly 30TB's backed up, so if I am outside of fair use, they haven't said anything.  As for restoring, you have the option of having them ship you a drive or downloading and if you ship the drive(s) back to them, they refund you the $$ for free.  It seems pretty straight forward to me but maybe I'm missing something.

 

Hmm good to know - I got the costs from their website after looking at your post - maybe it's a new policy thing ?

 

30Tb in the cloud for $5 a month - now that's what I call a bargain ! 

[rbjtech 5005]

I use StableBit's DrivePool with duplication on for all my Media files and I copy the database into a duplicated folder in that pool on a regular basis. It is a very bad idea to run Emby so that any dynamic files are part of the pool. I keep a copy of files that I cannot recreate in an off-site backup so as to protect those files from loss. I also, except for the database, do not backup any of the Emby created folder structure.

 

I do NOT backup most of my media (Recorded TV shows being an exception) as I have the original source (or a copy I made from old VHS or BetaMax tapes) stored away from my house in a safe location so, should the there be a disaster of some sort, I can recreate most of my library directly from the original source. If there is a disaster that destroys my file server system my media library is probably pretty low on the "worry" list so the fact that it will take some time to re-rip most of my library is pretty unimportant.

 

It should be noted that I do have a quite large, but not as large as some, library consisting of over 32000 TV episodes, over 3000 movies, 100s of pictures and 100s of audiobooks. I do not use Emby for music at all.

 

If anyone wonders how I can have original disks/tapes for such a large library it really is pretty simple: I inherited much of the total volume for an uncle many years ago and I already had a good size library. I have also added to that library ever since.

 

One more thing about StableBit's DrivePool: It is not really a backup solution but it is a really good solution for creating strong redundancy. Since I have been using it (5 years or so) I have only had one disk fail (A 4 tb one) and I simply ordered a new one and plugged in into the pool. I lost no data at all. DrivePool can be configured (I think it is the default) where it assures that files are always duplicated on different drives.

 

My system is not perfect but, for me, it is close enough to provide me with security that is good enough.

 

Thanks for your thoughts.  Having looked at Stablebit's website from a previous post recommending it too, I'm going to look to combine a Stablebit Drivepool with Stablebit Clouddrive (via NAS) to see if I can satisfy both the requirements of having a single pool (but importantly NOT RAID 0), and via Cloudrive replicate it (manual or scheduled only) to another single pool via Ethernet on a duplicate server.

[Carlo 4552]

You could do that but I wouldn't.

That will replicate all changes (good and bad).

 

What if you accidentally deleted your Movies root folder as an example?

 

What I would do instead is setup another drive pool on another machine and create a network mount point to it.  Then you can use just about any utility that can "sync" or copy files from one drive to another.

 

I like the simplicity of XCOPY that uses commands to watch the backup attribute.  It will copy (one way only) files from my server to it's backup locations and turn off the attribute after backing up the new or modified files.

 

This works really well as long as you aren't moving files on the server.  Example you wouldn't want to have files in Movies2 then move them to Movies1 libraries as the backup script would essentially "backup" the files to both backup locations.  Some common sense is needed.

 

I for example don't backup my DVR folders as I know they will get commercials cut and converted to a better format.

 

The key to this approach is that deletes or missing drives will NEVER automatically propagate to the backup location.

[Spaceboy 2565]

I’m backing my entire library up to the cloud using rclone. It’s going well albeit a little slow.

 

I have backups of os locally using veeam

 

I have a backup of important things, periodic os backups, photos and music in my storage locker

[rbjtech 5005]

You could do that but I wouldn't.

That will replicate all changes (good and bad).

 

What if you accidentally deleted your Movies root folder as an example?

 

What I would do instead is setup another drive pool on another machine and create a network mount point to it.  Then you can use just about any utility that can "sync" or copy files from one drive to another.

 

I like the simplicity of XCOPY that uses commands to watch the backup attribute.  It will copy (one way only) files from my server to it's backup locations and turn off the attribute after backing up the new or modified files.

 

This works really well as long as you aren't moving files on the server.  Example you wouldn't want to have files in Movies2 then move them to Movies1 libraries as the backup script would essentially "backup" the files to both backup locations.  Some common sense is needed.

 

I for example don't backup my DVR folders as I know they will get commercials cut and converted to a better format.

 

The key to this approach is that deletes or missing drives will NEVER automatically propagate to the backup location.

 

Thanks Cayers - I agree hence the "( manual or scheduled only) " bit in my response   - as I then control the replication.  I currently use SynBack Pro to do exactly what you suggest (using the archived attribute bit) and it works great but the bit I'm missing is the single pool - and as Stablebit solves this for me but maintains the JBOD flexibility, then I think I have the perfect solution (for my use anyway ..)   

 

 

 

 

 

 

[Carlo 4552]

Why not setup DrivePool on the destination system as well?  This is what I do.

 

Actually my storage needs for backup are in the hundreds of TBs so I backup to 3 different destination systems made up of "smaller" NASes or servers.

I've got one backup for movies, another for TV Shows and the 3rd for everything else (sports, home video, etc).  I did this to make use of existing hardware I already had and it works just fine for my needs.

[Gilgamesh_48 1132]

Why not setup DrivePool on the destination system as well?  This is what I do.

 

Actually my storage needs for backup are in the hundreds of TBs so I backup to 3 different destination systems made up of "smaller" NASes or servers.

I've got one backup for movies, another for TV Shows and the 3rd for everything else (sports, home video, etc).  I did this to make use of existing hardware I already had and it works just fine for my needs.

 

@@cayars has a library that makes mine, which I think of as pretty big, look quite small. Any advice he gives for library management is probably quite good and you cannot go wring by following it.

 

Personally I use only local systems, as it appears he does, for my backups and I move the physical media off site on a rotating basis for safety.

 

One more advantage of DrivePool is that all files in the pool are Windows readable files so is is possible to read and restore files even in the absence of the DrivePool software being installed. It is a little cumbersome to do so but it is not at all hard to do.

[Spaceboy 2565]

@@cayars has a library that makes mine, which I think of as pretty big, look quite small. Any advice he gives for library management is probably quite good and you cannot go wring by following it.

 

Personally I use only local systems, as it appears he does, for my backups and I move the physical media off site on a rotating basis for safety.

 

One more advantage of DrivePool is that all files in the pool are Windows readable files so is is possible to read and restore files even in the absence of the DrivePool software being installed. It is a little cumbersome to do so but it is not at all hard to do.

im backing up 120tb to the cloud [emoji3]

[Gilgamesh_48 1132]

im backing up 120tb to the cloud [emoji3]

 

I would never ever use the "cloud" for anything important or private. There are way too many ways thing can go horribly wrong with files that are out there for anybody to find and use. Also companies maintaining the cloud systems are not trustworthy and are not reliable in the long run.

 

Local storage is cheap and MUCH more secure and reliable.

Edited March 12, 2019 by Gilgamesh_48

[Spaceboy 2565]

I would never ever use the "cloud" for anything important or private. There are way too many ways thing can go horribly wrong with files that are out there for anybody to find and use. Also companies maintaining the cloud systems are not trustworthy and are not reliable in the long run.

 

Local storage is cheap and MUCH more secure and reliable.

wow thats quite impressive. you literally couldnt be any more wrong!!

 

no media within emby is important or private. 90% of it could be recovered in a few weeks. if you read my other post you would see that i say i backup photos and music (that which is important) locally and keep a copy off site.

 

please tell me where i can get 120Tb of local storage for £70 a year?