Recovering from my NIGHTMARE situation! (help ENCOURAGED)

[Garbonzo17 26]

Ok I'll try to keep this simple. I screwed up.

I basically deleted my TV Shows folder from a drivepool by mistake. I did it from the windows machine where it is physically hosted so recycle bin got back about 1/3 of it (size threshold not large enough). I then did some creative file recovery (recuva and the like) and drive shuffling and got back all I could (only files that were 100% intact).  Though some of this was able to preserve directory structures, much of it did not.  Just a big folder of files that Auto-Organize is still chewing on at the moment. 

SO. Looking at the logs I am seeing ALOT of missing episodes. Is there a simple or best way of getting a complete list?  I can't recall if Embystat had anything that would be helpful, but I need to re-install that as well.

On the brighter side, this pushed me to FINALLY migrate my emby server to the machine I wanted it on and re-install (my database was a little twerked after years of updates and migrations since x2.x) and encrypt my connection (thanks @mythofechelon)... so not all bad???..

My TV Shows were pretty neatly curated from a slow consistent build over the last 15 years. Hundreds and hundreds of DVD collections ripped... probably 5-times that recorded from my tuner.. and probably equal from other sources...  Oh, I was able to salvage the Directory structure (empty) somehow. That made NO sense, but I'll take it because there are 1016 show titles (I thought I had 1100 and change) and I can probably use the missing episodes to show me what I am missing completely.

So again, like I mentioned in the title. Help is very much welcomed! I am looking for insights into recovering and restoring what was lost. Not from the drives. that ship has sailed and I didn't have enough hard drives to keep them untouched.  I am talking the hard way... actually determining whats missing and then re-rip/re-capture/re-acquire... It's been like 2-months of denial and streaming Netflix/HBOmax/Showtime. but my bandwidth caps make that really unsustainable. But yeah, I realize its gonna blow .

Looking forward to and creative insight.

Thanks in advance.

-G

[Luke 37132]

@CBers don't you use a drivepool?

[CBers 6772]

3 hours ago, Garbonzo17 said:

SO. Looking at the logs I am seeing ALOT of missing episodes. Is there a simple or best way of getting a complete list?  I can't recall if Embystat had anything that would be helpful, but I need to re-install that as well.

So again, like I mentioned in the title. Help is very much welcomed! I am looking for insights into recovering and restoring what was lost. Not from the drives. that ship has sailed and I didn't have enough hard drives to keep them untouched.  I am talking the hard way... actually determining whats missing and then re-rip/re-capture/re-acquire... It's been like 2-months of denial and streaming Netflix/HBOmax/Showtime. but my bandwidth caps make that really unsustainable. But yeah, I realize its gonna blow .

Looking forward to and creative insight.

Thanks in advance.

-G

You could install Sonarr and import your TV shows. It would then show what is missing. 

50 minutes ago, Luke said:

@CBers don't you use a drivepool?

No. 

[Gilgamesh_48 944]

56 minutes ago, Luke said:

@CBers don't you use a drivepool?

I am one that uses Drivepool and it seems that Drivepool did exactly what it was supposed to do. When you delete something from the pool Drivepool really deletes it even if you have duplication turned on once a file or directory is deleted it is really deleted. Drivepool provides redundancy BUT it is not a backup solution at all. It does nothing to protect from mistakes. The solution to recovering files deleted as @Garbonzo17 did does not lie within the abilities of Drivepool. The duplication feature protects from the loss of or damage to a drive within the pool but it does not protect from deletions issued from the operating system.

One thing is that the Drivepool software should be turned off as soon as possible after such a deletion because Drivepool is always moving files around and, if it is on, the deleted files will get overwritten.

My solution is to run rotating backups outside Drivepool. You can use additional hard drives or you can store backups in the cloud if you trust the cloud, which I do not.

I might add that I only backup media the I no longer have the originals of or that would be a lot of trouble to recreate like some movies I have on Betamax tapes.

Unfortunately Drivepool is just not designed as a backup solution.

EDIT: @CBers' idea of using Sonarr is a good one. when installed it can scan TV shows and report what is missing.

I believe @cayars also uses Drivepool. Maybe he has some other ideas.

Edited August 10, 2020 by Gilgamesh_48
addition

[Carlo 4330]

45 minutes ago, Gilgamesh_48 said:

I believe @cayars also uses Drivepool. Maybe he has some other ideas.

I don't allow DrivePool to shuffle the media between drives because of issues like this.

I also don't use it's duplication features either but instead use SnapRAID for CRC data duplication to help rebuild data when a mistake happens like this.

Considering what happened I think the OP is doing all that can be done to recover as much as possible.

[Gilgamesh_48 944]

16 minutes ago, cayars said:

I don't allow DrivePool to shuffle the media between drives because of issues like this.

I also don't use it's duplication features either but instead use SnapRAID for CRC data duplication to help rebuild data when a mistake happens like this.

Considering what happened I think the OP is doing all that can be done to recover as much as possible.

In my case I decided that I was MUCH more likely to lose a drive than I was to accidentally delete something. Drivepool, with duplication, seamlessly protects from a single drive failure so that is why I use duplication.

In the 50 some odd years since I started using computers, or they started using me, I believe I have accidentally deleted something maybe twice. I make plenty of mistakes but that is one that I seem wired to avoid.

I seems that we accomplish virtually the same thing by using our tools differently. 

As you do not complexly trust Drivepool for redundancy I do not trust any form of "raid" for backup protection. I simply duplicate my pool onto drives that I store in  a fireproof safe and bring out weekly or so to make backups. I use "Goodsync" to perform the backups. I feel about as secure as is possible in the world of computers.  I have two redundant copies of my data in the pool and another, updated weekly, in a fireproof save and another on drives stored in save deposit box that I update every six months or so. 

With all that I have never had to use the backups I keep off site and only once have I needed the backups I keep in my safe. That one time was when a file became corrupted for an unknown reason. I still keep the backups fresh, just in case, but I am a belt and suspenders kind of guy.

[Carlo 4330]

The thing I don't like about duplication is that it cuts your media storage in half. 100 TB is only useful for storing 50 TB.  With CRC/checksums used by programs like SnapRAID you can have 1 drive protect against 20 drives so you only "loose" 1/20th the amount.  If you want to protect against a 2 drive failure use 2 drives for checksums and only give up 1/10th the space, etc..

Since it's not like hardware RAID if you accidentally format a drive, wipe a master folder or anything contributed to "user error" can you rebuild your data.  With SnapRAID you tell it when to create it's checksum either manually or on a schedule.  

No right or wrong way to use these types of tools, just preferences in how we each like to use them to fit our needs an environments.

[CBers 6772]

1 hour ago, Gilgamesh_48 said:

Drivepool, with duplication, seamlessly protects from a single drive failure so that is why I use duplication.

Same here, but DriveBender. 

 

[Gilgamesh_48 944]

15 minutes ago, CBers said:

Same here, but DriveBender. 

When I first started using DrivePool I looked at DriveBender as well by the implications were, at that time, that DriveBender was nearing or at its end of support and development. Has that changed?

Actually a quick search seems to show that I either misunderstood what I read way back then or Drivebender was in some kind of flux state. I am not at all upset with my choice but if I were choosing at this time I would have to look closer at Drivebender. One thing I notice is that Drivebender has a very slick web page while StableBit's site for DrivePool is a bit more primitive looking. It is easier to read and digest the DrivePool site and that is actually a positive to me. I tend to distrust companies with "slick" web pages. It just feels that the "slickness" is there to mask something from tight scrutiny.

That is at least a small part of why I trust Emby more than Plex. Plex's web site is just too slick and it is a lot like a politician's speech in that it says little in a lot of words and pictures.

 

[Ronstang 205]

I find this thread interesting as I am looking for a back up solution myself.  I currently just have all my files on second hard drives in the closet I only put into docking stations to add data so these drives are essentially brand new because they have almost no hours on them.....and all my system drives power down after a few minutes of non use so most of my drives just sit unless I watch a movie or record one.

I like the idea of SnapRAID but I don't fully understand how it protects you from data loss if a drive fails without redundant data storage.  IS there an easy to understand resource I could read to learn and understand this technology Cayars?  

[Gilgamesh_48 944]

32 minutes ago, Ronstang said:

I like the idea of SnapRAID but I don't fully understand how it protects you from data loss if a drive fails without redundant data storage.  IS there an easy to understand resource I could read to learn and understand this technology Cayars?  

Do not worry about not understanding how backups like SnapRaid work. I have a PHD in applied math and I 100% understand the mathematics of how Raid and CRC data storage works and how it allows data to be stored in and recovered from a MUCH smaller space that the actual size of the data I still find that it is more like magic than science. I may understand it but I cannot really explain it in any manner that would be seen as "simple" because it is not simple.

Words do not handle mathematics well at all and data compression is one of the many areas where words simply cannot do anything more than describe what happens they cannot explain it.

You might want to look at the following thread as it contains some pretty good explanations of what happens: SourceForge discussion of SnapRaid and how it does what it does and quite well at that.

[Ronstang 205]

Thank you Gilbamesh_48.  I understand math pretty well too but my area of specialty is mechanics.  I know enough about computers to get almost anything working, even if I don't always understand exactly how it works.  The problem is I like to know how stuff works when I can and this sounds just like you say "magic"....but I am smart enough to trust those more knowledgeable then me.  I assume that since all computer data is stored in a binary state that you can have a file that is compressed text with the exact layout of the "1s" and "0"  by using markers that save repeating patterns or something such that you can reconstruct the file from a "blueprint".....or am I way off base?

[CBers 6772]

8 hours ago, Gilgamesh_48 said:

When I first started using DrivePool I looked at DriveBender as well by the implications were, at that time, that DriveBender was nearing or at its end of support and development. Has that changed?

Yes, we've had quite a few updates since that statement. 

The developers seem fully committed to continue to develop the product. 

 

[rbjtech 4291]

I've used the majority of the drive pooling software and while they are all great in concept - the simple matter of fact is if they go wrong, they go wrong big style and data recovery becomes impossible, a nightmare or takes forever with the data quantities we are generally talking about with TB's worth of video files.

Taking a step back - ask yourself the question what changes on your media collection day/day ?

Sure - you may have TV recordings going on - but other than that, your collection is either STATIC or you ADD to it.

So for the STATIC files - why do you need to keep a resilient version online ?

Copy (backup!) the STATIC file to your old/previously upgraded HDD's sitting on the shelf - just use JBOD - making them totally portable on any system.

You now have an OFFLINE version of the files, take them off site if you like - and in the event of needing them, they are individually available.  They are not susceptible to being manually deleted, pool software crashes/corruption, ransomware and the like - and will work perfectly in any system immediately.  

If you are smart, then keeping this backup aligned with your 'live' system means that a 'live' hard drive failure simply means swapping in your 'backup' disk and you are up and running again.  Of course, you need to re-create a new 'backup' but that is only for the disk you lost - not the entire array.

This 'concept' obviously only works for Static data - but I find it works perfectly for my 50Tb worth of online data with 50Tb(on old HDD's) sitting offline AND acting as a backup.

 

 

 

 

Edited August 11, 2020 by rbjtech

[Gilgamesh_48 944]

1 hour ago, rbjtech said:

I've used the majority of the drive pooling software and while they are all great in concept - the simple matter of fact is if they go wrong, they go wrong big style and data recovery becomes impossible, a nightmare or takes forever with the data quantities we are generally talking about with TB's worth of video files.

Taking a step back - ask yourself the question what changes on your media collection day/day ?

Sure - you may have TV recordings going on - but other than that, your collection is either STATIC or you ADD to it.

So for the STATIC files - why do you need to keep a resilient version online ?

Copy (backup!) the STATIC file to your old/previously upgraded HDD's sitting on the shelf - just use JBOD - making them totally portable on any system.

You now have an OFFLINE version of the files, take them off site if you like - and in the event of needing them, they are individually available.  They are not susceptible to being manually deleted, pool software crashes/corruption, ransomware and the like - and will work perfectly in any system immediately.  

If you are smart, then keeping this backup aligned with your 'live' system means that a 'live' hard drive failure simply means swapping in your 'backup' disk and you are up and running again.  Of course, you need to re-create a new 'backup' but that is only for the disk you lost - not the entire array.

This 'concept' obviously only works for Static data - but I find it works perfectly for my 50Tb worth of online data with 50Tb(on old HDD's) sitting offline AND acting as a backup.

The reason, in my case, for using duplication on my libraries is NOT really data protection . Rather it is for easy recovery in the event of a single drive failure.

As an example consider the case of a drive failure and what must happen to continue watching media that was stored on that drive:

The following assumes I want to replace the failed drive.

1. I have a good backup of all my files but no local duplication. The drive fails I must attach the backup drive or drives and run whatever restore process is required and then wait until the restore process finishes and that would take hours to complete and can't even start until I acquire a new drive to replace the failed one.

2. I have DrivePool and duplication is on. The files are, at worst, available as soon as DivePool realizes there has been a disk failure and that is a max of about  2-3 minutes. I then remove the failed drive and get anther to replace it and attach it to the computer and then attach it to the pool and DrivePool takes care of the rest. All the while my viewing is never really interrupted. After putting a new drive in the pool DrivePool will handle moving the files around in the background and making any duplicates that were lost.

That works that way as long as only one drive fails but I have never seen or heard of two drives failing at the same time. Should that happen I am no worse off than I would be without DrivePool. I just restore from a separate backup like I would have to do without DrivePool and, because of duplication, I will only actually have lost about half the files as any actually lost files would need to both the file and the duplicate stored on both drives. DrivePool prevents both a file and its duplicate from being stored on the same drive.

I tested by unplugging two of my 13 drives and I found that only 50-100 files would actually be lost should both drives fail at the same time. I did that about 5 times using different pairs of drives and the results were always in the same range.

It is not a matter of "change" but rather it is a matter of probable data loss and ease of recovery from said loss. I keep a backup, just in case. But I have never had to use it in the years sine I started using DrivePool. 

[rbjtech 4291]

..and for that 'easy' recovery (if and when it happens), you have double the number of disks spinning and being powered 24x7.    50% are being 'wasted' 99.99% of the time. 

With my solution - probable data loss = 0%,  ease of recovery= quicker than any array rebuild (my data is already on the disk..).  The only downside is my users will not be able to watch a percentage of my overall collection while I go and swap the disks around.  For my home usage - more than a fair compromise.

 

 

[Gilgamesh_48 944]

3 minutes ago, rbjtech said:

..and for that 'easy' recovery (if and when it happens), you have double the number of disks spinning and being powered 24x7.    50% are being 'wasted' 99.99% of the time. 

With my solution - probable data loss = 0%,  ease of recovery= quicker than any array rebuild (my data is already on the disk..).  The only downside is my users will not be able to watch a percentage of my overall collection while I go and swap the disks around.  For my home usage - more than a fair compromise.

Modern hard drives are designed to spin 24/7. In fact there is quite a bit of data that shows that a large numbers of spinup/spindown cycles shorten drive life. It is my strong beliefe that servers and the drives attached thereto should never sleep. The only problem that happens from constant running of a server is heat and that can be handled easily by having a simple box fan blowing across the server and the drives all the time.

The most likely time a hard drive will fail is when it is starting or stopping just like a light bulb is most likely to fail when it is turned off or on. The physical stresses are greatest during a short period during startup and an even shorter period during spindown.

Your way may seem easier to you but it would not be easier for me. I like the way my system is set up.

BTW: I do not have any "users" at all except me. I do not know anyone I like well enough to do the work and cost of acquiring media for them and maintaining my system. 

[pwhodges 1534]

But keeping a backup drive in a cupboard is not likely to cause large numbers of start/stop cycles.  OTOH, if it is left for several years there is a possibility of it failing to start because the mechanics have seized up - I've seen that, but not for over a decade now.

Paul

[rbjtech 4291]

Sure - i wasn't saying your setup is wrong at all - if it works for you great.  All I am suggesting to everyone is to think about how important and critical their data really is vs spending $1000's of pounds on redundant disks to cater for what, in your own words, is a very small possibility of a disk failing.  FAR more likely is human error, software crash/corruption, malicious attack - ie the OP's original scenerio.

You are lucky enough to have offline backups, but if you did not, your duplicated disks would have not helped at all ....

[CBers 6772]

I think this is going a bit off-topic now. 

@Garbonzo17 Are you happy with the responses, or do you need anything else?

 

[Shidapu 14]

Ouch, thats a real bummer situation you got yourself in. Recovering all those files, and replacing them will take a really long time.

I've been doing my own projects with Topaz Video Enhance AI , and that thing can take up to 16 hours per movie to upscale a movie to 1080p.
I'd be breathless loosing all that time and effort, so just remember to make regular backups in the future, might even try Cloud backup solutions? One good tip is the Business Subscription on Google Drive, you will need a domain for that, and the cost is insanely low for "Unlimited" storage.

If your stuck with a bandwidth cap on the ISP, i'd look for another ISP.

[Garbonzo17 26]

21 hours ago, CBers said:

You could install Sonarr and import your TV shows. It would then show what is missing. 

No. 

I hadn't looked into Sonarr in a LONG time.. I will dig into that tonight. Thanks for that suggestion.

[Garbonzo17 26]

20 hours ago, cayars said:

I don't allow DrivePool to shuffle the media between drives because of issues like this.

I also don't use it's duplication features either but instead use SnapRAID for CRC data duplication to help rebuild data when a mistake happens like this.

Considering what happened I think the OP is doing all that can be done to recover as much as possible.

I think I am doing all I can as well.  I immediately broke the pool (after restoring the recycle bin) and began recovering the stuff that been removed from the FAT tables but were intact.  My problem is that I just don't have enough empty drive  space to copy things to without then having to erase a previous disc to make room.. its frustrating, and if a had a few hundred bucks to throw at it right now, I could've made this ALOT simpler.

But SnapRAID looks interesting. I like the idea of kinda stand alone parity.. is that the idea? I bookmarked it and will get into it asap.

Thanks.

[Carlo 4330]

Yes, that's the idea.  Something to keep in mind.  The parity drives needs to be at least as big as your largest filled drive.  You can split parity across multiple drives but then you're adding another point of failure so it's best to make sure to use your largest disk for parity.

[Garbonzo17 26]

12 minutes ago, cayars said:

Yes, that's the idea.  Something to keep in mind.  The parity drives needs to be at least as big as your largest filled drive.  You can split parity across multiple drives but then you're adding another point of failure so it's best to make sure to use your largest disk for parity.

Thats good to know. I need to look into how it works on a tech level and how it would play with drivepool (with duplicating turned off).

I currently have 3x8tb and 2x5tb in the pool, but can easily pull an 8 (or buy an external for cheap enough, like 119 most of the time at costco).

I gotta think that any rebuilding would be terrrrrriblty slow, especially since this is all on a not to modern pc... since its really ONLY doing emby/drivepool and anything related. but I have another 5tb so in theory I could pull one of the 5's and swap in the spare to test... either way it's worth looking into. So thanks again.

-G