Recovering from my NIGHTMARE situation! (help ENCOURAGED)

[Garbonzo17 26]

4 hours ago, cayars said:

I've seen that and caused it to happen on purpose testing.  You can easily do this on purpose by removing data and replacing it OR moving things between drives and not updating the parity after making changes.  When you keep the parity up to date you shouldn't have problems.  That why I turn off any type of drive balancing so once files are written they are there for good.

That actually makes sense.

My biggest reason for using drive pool was I had more TV than would fit on my largest drives (8tb) and didn't like having some TV on an 8tb and some on a 5tb (each with a mirror) and then further movies on a 5 with only MOST mirrored to a 3tb.  But  I am thinking rather than having my 3x8tb and 2x5tb in this pool I should probably pool a single 8 & 5 together, and mirror it manually to another 8 & 5 together.... and use that for TV.... then use the spare 8tb for SnapRAID.  (and get another drive to mirror movies (though for the time being they'll fit on the same pools as the TV.

This way, drivepool is keeping my content "appearing" together, and by manually syncing drivepool isn't shuffling things around.

But I still have alot of corrupted video files that I am having to "discover" and decide if its worth the effort to re-acquire.

-G

 

[Carlo 4330]

Instead of keeping the mirror I think I'd rather have 2 drives for SnapRAID.  That would protect all your drives and it would take a loss of 3 drives before you lose anything.  Secret to DrivePool and SnapRAID use IMHO is to turn off all the fancy features in DrivePool and just let it do the combining but nothing else.  That way once a file is written it doesn't move.

But yes, if you felt better have both a mirror and parity that would be the way to do.

[Gilgamesh_48 944]

I guess my data and ease of recovery are more important to me than disk space or the cost of drives.

I do not like the idea of waiting for a long recovery process if I have a drive failure and I do not like the idea of not being able to read any file on my backup without any "special" software and I do not trust any system, like parity disks, where I do not have immediate access to any file on any drive in my backup I also do not trust "cloud" storage.

As I said before I run DrivePool and have duplication turned on for my primary live system. I have a second set of drives that are also pooled )but do not have duplication) and I run a good backup software that does not encrypt the drive or files in any way. (I actually have two sets of those backup drives) I keep one live and run incremental backups nightly. The other set is kept in a fireproof safe. I rotate the two backups about monthly.

So if I have a major failure, like a fire or lightening strike the destroys everything in my media center the worst case is that I loose about one months worth of stuff that was new in that month.

If I have a failure that also destroys the drives in my fireproof safe then my problems are MUCH greater than just a little media. I used to store one set of backups off site in a safety deposit box but I decided that was just too much trouble for what small extra protection it gave,

Hard drives get cheaper all the time and the value of my time keeps going up (Partially because it just does like everything else and a larger part because my health prevents large efforts to be expended.)

My values are different from others but the actual operation of my system is simple and, should I have a major failure it would take little effort to get back into operation.

I should also say that in a normal failure (one drive goes bad) my viewing is never interrupted and I just have to get a new drive to replace the busted one and add the new drive to the pool.

Everyone has different ways of attacking the "backup" problem but, to me, within reason, I think cost of drives should be one of the last things considered.

It is said that "Time is money" but I believe "My time is much more valuable than money."

[feerlessleadr 155]

On 8/17/2020 at 5:44 PM, Garbonzo17 said:

Well that is disconcerting. I was really hoping it would get me so security in addition to the file mirroring.

But I am also saving the cash to finally build another array that will be a mirror and then put it at a relatives house across town. That way I can mirror it onsite and drive it over there and then have them sync nightly or something.  I have about half as much data as you ~17tb.... but I also have a 1tb monthly cap and no real alternative ISP available. So uploading to any cloud provider would be an issue. (I was originally looking at something like backblaze)

I don't mean my post to be a common occurrence with snapraid, since if it was the case for many, the github/website/etc. would be blowing up with angry users. 

In the end though, I was pretty diligent in making sure that my parity was up to date (should have used 2 parity drives though), I tested recovering small amounts of files, but in the end, I couldn't get an actual restore w/ major data loss to complete successfully. 

That's why I decided to move on to something else. 

[chef 3748]

In order to help out here,

I have almost finished a quick plugin which will calculate the missing items in collections such as TV Shows and list them in an easy to read chart on the plugin configuration page.

I have it currently logging to the console, I will just complete the list in html and put a copy of it here on this thread, in hope that it helps sort out what might be missing from peoples collections when data is lost.

 

It will look prettier then this when it is complete

 

 

 

 

[Carlo 4330]

Looking good chef.

[Garbonzo17 26]

3 hours ago, chef said:

In order to help out here,

I have almost finished a quick plugin which will calculate the missing items in collections such as TV Shows and list them in an easy to read chart on the plugin configuration page.

Interesting.

I have been using (and appreciating how far it has come since I last looked at it) Sonarr to identify strictly missing episodes.  My new (and actually probably worse) issue is the fact and some percentage (less than 10% I believe) are corrupted even though I only recovered files that my restore software deemed 100% recoverable.  So now outside of going into every single show/season folder and clicking to play them (and it either working or giving a "failed to render" in my player) I can't know until me or another family member attempts to play a file if it is bad or not.

I got some suggestions using ffprobe.exe but that was a little beyond my current comfort zone, though I have done some more reading and testing, I still need some work to be able to script it into something relatively automated.

Anyway, thanks for what you have done, and I am looking forward to seeing what becomes of it.

Edited August 22, 2020 by Garbonzo17

[chef 3748]

Strange, while I was writing code for this, I had a hard drive failure and I lost about 500 gigs.

That is ironic. One thing I have noticed, is that if you open emby and search the libraries without a media scan, the missing items will appear without a folder image. I would imagine this is only if you have the images saved into folders. But I has helped me figure out what I am missing.

[Garbonzo17 26]

On 20/08/2020 at 09:51, feerlessleadr said:

I don't mean my post to be a common occurrence with snapraid, since if it was the case for many, the github/website/etc. would be blowing up with angry users. 

Agreed. And it's not as much of an issue now. I think if I was gonna have to buy another 8tb to do parity, that I am just gonna lean in and get 2 and pool them into that second array that once mirrored I will move to my uncles place in a nearby town 45mins away and have a one-way mirror nightly or every 48hrs... that's one of 3 family members accessing my server that won't need to connect over the internet anymore either, he will be able to use it locally at his place... so all around it should be a win... I will keep my 22tb-ish pooled/duplicated locally at my place on my 30tb pool. And that same 22tb (so I guess only 11tb not duplicated) on the 16tb array at the uncles place.

So I guess I am learning I lost about 2tb of files overall... before this loss on my part I had about 16tb of actual video (tv 13ish/movies 3+) I didn't loose any of the movies, just the TV that I improperly synced... so now I have TV 11ish and the same 3+ in movies.... So all in, it could have been worse... but also better if I can't determine what I am missing and what I have that is now corrupted/unusable.

Thanks to everyone for their help in this situation. This community is always top notch. I haven't posted much in the years since I began using the software (since it was still M.B.) mostly because anytime I have had a question simply searching the forum generally answered my questions.  Anyway, there are names that have been responding to this post that I have seen it countless other replies that have helped me on other topics in the past... @CBers, @Gilgamesh_48, @cayars and now the famous @chef... all people that I have recieved help from indirectly by there responses to other peoples questions... and all I feel I should have taken the time to thank in those instances rather than just reading/accepting the fix and moving on.

So, to anyone in the community that is active in helping people with their issues, thank you for taking the interest and the time to do so... And anyone else who my current F-up, and the suggestions contained within, might have benefited... Take a second to thank anyone who's insights pushed you in a direction that hadn't occurred to you previously.

-G

[Garbonzo17 26]

16 hours ago, chef said:

One thing I have noticed, is that if you open emby and search the libraries without a media scan, the missing items will appear without a folder image. I would imagine this is only if you have the images saved into folders. But I has helped me figure out what I am missing.

I thought that missing images might be helpful (but not as helpful as if the "show missing episodes" flag could let me export to a text file)

So in my case Sonarr is proving more accurate.

The attached images show that while Sonarr and Emby (not folder video) are the same for some (The 4400. The Act), Sonarr clearly shows (and is correct) that some shows Emby has images for are still missing files (The Affair, 9-1-1).

-G

Edited August 22, 2020 by Garbonzo17
forgot to attach images

[chef 3748]

What Am I Missing Plugin 

Will List missing BaseItems (Episodes) in your TV collections, which (if you had them) would make the collection complete.

Does it work?

Yes.

Does it work perfectly?

No.

1. Currently the plugin will list all the item missing in tv show collections, this includes items which have not aired yet in upcoming seasons. This could be easily remedied, if needs be

2.There is a chance that this will only work if you have "Show missing episodes in season" turned on. However this hasn't been tested.

(I just have it turned on, and I think that it is how the Emby API is returning items that don't exists in my library...), however it might work even if this option is turned off.

 

I wrote this entirely in javascript which is a bit different then I would have coded it before (in c#). Maybe useful LOL.

WhatAmIMissing.zip

[Garbonzo17 26]

5 minutes ago, chef said:

Does it work perfectly?

No.

1. Currently the plugin will list all the item missing in tv show collections, this includes items which have not aired yet in upcoming seasons. This could be easily remedied, if needs be

Thanks. and I will check it out later (I cant restart emby server right now, 2 people are watching movies/shows, everybody catches up over the weekends).;

But you certainly don't need to put yourself out doing any work on this on by behalf... Are you pulling the info from Emby's list of missing episodes and just displaying it in a table or doing something else?  I am just curious because this seems to be what I was looking for in the OP.  But once I was reminded of Sonarr and went down the Jackett rabbit hole I don't know that I need it anymore.

My biggest issue right now is the % of corrupt files I have, but they don't outwardly appear bad... only when clicking them do I get a "failed to render" message... I need to figure an automatic way do to this and give me a list so I can move them elsewhere to double check (and then compare to some duplicate files that I have, which will obviously be a manual process).

-G

[chef 3748]

Yeah, 

It looks at each entire season, then filters out what you have, which leaves only what you don't have

Puts that data in a table.

[Garbonzo17 26]

8 minutes ago, chef said:

It looks at each entire season, then filters out what you have, which leaves only what you don't have

Ok, I just didn't know if it was comparing it to something online to get the episode counts so it knows what is missing (imdb. tvdb, or whatever). I'll dig into it tonight once everyone logs out.

thanks again.

 

[Garbonzo17 26]

On 17/08/2020 at 16:46, PenkethBoy said:

you missed of the 1 after the %

Sorry. Its been a few days since I had time to get into this issue anymore.

I did it again and the other day and got the same text file output. I am not sure what I am missing, but I just did it again on new files (I already re-ripped the other show) and on this one and a few other all I get is { } for the "bad" file and get pretty similar to before on the "good" file....

That said, I am new to what ffprobe.exe should be doing, but I am seeing alot of unrelated (I''d assume since its a command line exe and I haven't scripted anything) but I am not sure that something else isn't using it (since I just learned emby has ffprobe.exe in the system folder, I was using a ffmpeg binary that I installed seperatly)... so I am attaching an image showing the activity in disk monitor to see if that looks "normal"-ish...  (and if emby is doing ffprobe commands is there an obvious way I can add something that checks the integrity for my purposes? (rather than me trying to do something from scratch)

 

TIA.

-G

[Garbonzo17 26]

On 22/08/2020 at 16:09, chef said:

Yeah, 

It looks at each entire season, then filters out what you have, which leaves only what you don't have

Puts that data in a table.

Is there anyone you might recommend that can help with determining which files I have that are corrupt? ffprobe was suggested, and actually there is a emby function that gets me most of the way there... when I so a "scan library" on a Series with 4 seasons that I just re-ripped, looking at disk in resource manager ffprobe.exe pops up so quickly I can't even click on it to see what files it is looking at....

HOWEVER, I scanned Psych, which has some "known bad" videos, and this is what I see...

and those sat there for various times, 10-30 seconds-ish and after most cleared a couple more came up, but all together about a dozen or so over the 8 seasons... so if these take so much longer to clear/fail/whatever, if there was a way to log it I could then move those files to another location and then have Sonarr let me know what is missing...  I know it sounds convoluted, but I clicked on all of the files listed there and the all "failed to render" in my local media player, but every other file in those same folders works fine... so I feel like it is within reach.. just a use case that maybe nobody has needed...

Also, do you have an opinion on if I should I break this out into a new topic that may be better served? 

TIA

-G

[rbjtech 4291]

The fundamental issue is you have no baseline to compare your files to ( ie no CRC/HASH)

Unless you have this - I don't believe there is a way to quickly see if a file is missing 'data' - could be a couple of bits resulting in a single frame being partially corrupted, it could be reference frames missing, resulting in temporary 'hangs' until the next ref frames are synced up or it could be missing 'critical' information meaning the file does not play at all.

If ffprobe is 'quickly' scanning files - then I would expect it is just scanning the headers - it is not looking at the actual frame by frame content.

ffprobe -v error -count_frames -threads 0 -i filename.mp4

It is possible to force ffprobe to do this - where it literally reads every frame and reports back on any corrupt frames - this is what I think you need to run to be 100% sure the file is not corrupt.

However - it is extremely processor intensive and will take a long time ... it may be quicker to re-rip the files ... 

edit - I'm going to look into this some more myself - as I technically have no way of knowing if my 'backup' is actually 100% good -but with a CRC/HASH check, I should be able to confirm this ...

 

 

 

Edited August 27, 2020 by rbjtech

[Garbonzo17 26]

6 hours ago, rbjtech said:

If ffprobe is 'quickly' scanning files - then I would expect it is just scanning the headers - it is not looking at the actual frame by frame content.

Yeah, I am just trying to get a kinda first pass list of files that are most likely bad.  By the fact that it scans my re-ripped Sanctuary (4 seasons) in a few seconds without alot of overhead, yet Psych (8 seasons) hangs around for many, many minutes with certain files getting pegged in resource monitor for extended times, I would think there is some delta in ms (or seconds in this case) that could be logged.  Again, it's just over my head in my current grasp of what the software is doing when this is triggered (by manually re-scanning files).

I am going to get a couple more drives as soon as I can afford, and then proceed to have offsite backup (for security) and then keep drivepool's mirroring on (for resiliency)... but it's about filling in missing (sonarr is doing that well enough) and corrupt files (fail to render in VLC or MPC)... the latter is my current dilemma.

Thanks for your insights, and let me know if you run across anything if you further dive into it.

-G

[rbjtech 4291]

Hi - I did look into it a little more.  My backup software does 'Integrity' checking - this is basically creating an MD5 hash of the file and keeping a record in a database of what it is.  For you - this would have been ideal as you could then have re-calculated the MD5 hash post the 'incident'  - and if any were different, then you could say without doubt that the file is 'corrupt' or needs replacing.  For me, I wanted to compare MD5 hashes from the original source to the backup destination - and there doesn't appear to be an easy way for me to do that.   

I will probably end up writing a script to create the .md5 file to keep alongside the video file - then for any given query, I can simply compare the original .md5 file to a newly calculated .md5 file from the backup - if they are identical, then so are the files. 

I believe drive pooling software may use some form of integrity checking ? - but I don't know enough about it to comment further.

 

[Garbonzo17 26]

6 hours ago, rbjtech said:

 

I believe drive pooling software may use some form of integrity checking ? - but I don't know enough about it to comment further.

It may, but unfortunately I messed things up while moving my data to the pool in the first place (ignored a sync warning about it doing a mirror sync instead of a one way) so my pool got sync'd with a drive containing LESS files (lucky it was early enough on in the process that I hadn't overwritten most of the other drives).

I need to buckle down and get into fprobe to see how long it is churning on these bad files and then figure out a way to log it...

Tonight I am messing with Radarr Collection and People Manager up... I got turned onto it looking for a way to show missing movies in collections... I think you were active alot in that thread actually.

[Garbonzo17 26]

Ok updata. dont know what took me so long to examine the server log file, but it has what I need in it.... clearly states "ffprobe failed for file:xxxxx" so now I just need to pull the data to the right of that statement.  What is the easiest way for a non-programer to grab that data from a text file?
I am going to do some googling to see if I can answer myself, but I thought i'd ask anyway.

Here is an example of what I am seeing...

   
2020-08-29 03:11:53.458 Info MediaProbeManager: ProcessRun 'ffprobe' Process exited with code 1
2020-08-29 03:11:53.459 Error App: Error in ffprobe
    *** Error Report ***
    Version: 4.4.3.0
    Command line: C:\Users\PK\AppData\Roaming\Emby-Server\system\EmbyServer.dll -noautorunwebapp
    Operating system: Microsoft Windows NT 6.2.9200.0
    64-Bit OS: True
    64-Bit Process: True
    User Interactive: True
    Runtime: file:///C:/Users/PK/AppData/Roaming/Emby-Server/system/System.Private.CoreLib.dll
    System.Environment.Version: 3.1.4
    Processor count: 4
    Program data path: C:\Users\PK\AppData\Roaming\Emby-Server\programdata
    Application directory: C:\Users\PK\AppData\Roaming\Emby-Server\system
    System.Exception: System.Exception: ffprobe failed for file:"Z:\!Media\TV\(R-Z)\Sherlock\Season 01\Sherlock.S01E01.A.Study.in.Pink.mkv" - streams and format are both null:

Sorry for tagging a couple of you that have already been helpful, but I am not having much luck extracting the info I need from the log files. I would probably get more traction on this if I just asked it as a new question, but I didn't know if I was out of line to duplucate the question of if it's worse to @ a couple of people... like @chef @rbjtech @PenkethBoy @CBers and @Gilgamesh_48 who have ALL been very helpful thus far. 

I think this is my final hurdle... and I should be in good shape once these files are identified definitively.

TIA

-G

 

 

Edited September 2, 2020 by Garbonzo17
tagging a couple people

[CBers 6772]

@Garbonzo17 There are a few ways of doing it, but the easiest is probably in a CMD window and using the find command.

Something like:

find /i "ffprobe failed" serverlogfilename

That should return all instances of "ffprobe failed" in the file. 

The /i ignores case. 

Edited September 3, 2020 by CBers

[rbjtech 4291]

find /i "ffprobe failed" serverlogfilename > errors.txt

.. redirect the output to a file which will then be a complete text file list from the emby log.

 

[PenkethBoy 2063]

another alternative would be any good text editor that has a "find in files" option

e.g textpad (which i use) or Notepad++

i have search all the logs in my log folder for "error" - see dialog thats in bottom right (transparent)

in the search results you can double click an entry and it will be opened in the top at that point - see highlighted lines

search results can be saved to a file if necessary

Obviously you could just search for ffprobe failed if you only have one log file - textpad has a nice feature to highlight all occurrences found rather than slowly going from one occurance to the next etc

[Garbonzo17 26]

I about crapped my pants when the initial text file I got back was 45k+ lines long, but I noticed some duplicates, so I filtered for unique entries and its a much more manageable 1500 or so... Still alot of bad files, but I just made one big batch file MOVE to another non-scanned folder location, and at least emby isn't choking on the bad files during scans anymore.

Thanks again for everyone's help... now I gotta get to work re-acquiring what I can that is bad...

-G