Jump to content


Photo

Emby Just Deleted ALL of my Movies


  • This topic is locked This topic is locked
198 replies to this topic

#21 ebr OFFLINE  

ebr

    Chief Bottle Washer

  • Administrators
  • 52011 posts
  • Local time: 09:44 PM

Posted 04 March 2019 - 04:59 PM

Hi.  We are not meaning to imply there was anything you did wrong.  Just that the error occurred during the delete requests as opposed to the refresh metadata actually causing deletions.  We're 100% sure that is not the case so I just didn't want people to be afraid of that function.

 

Thanks.


  • Angelblue05 likes this

#22 speechles OFFLINE  

speechles

    Advanced Member

  • App Developer
  • 6088 posts
  • Local time: 06:44 PM

Posted 04 March 2019 - 05:35 PM

There is no consoling this wound. The damage is done. What can be done is use this to learn. Assign value to the collection that must now be rebuilt entirely however way it was done. There is no pointing fingers if you ripped this yourself or borrowed it from usenet/torrents. This is situational where if you value the library you protect it. Some users let Emby protect it for them without using other precautions to keep Emby in check. In other words, they trusted Emby with all their media. This trust is now violated. This must immediately be repaired and this user should be consulted with directly and offered some kind of special PM where they discuss how to solve this. None of us have experienced this so the sense of loss and violation of trust can only be speculated how this feels. What should happen is Luke and Ebr should directly contact dbott and work to come up with a complete solution. The person who experienced this should be included in the solution to be sure this would satisfy in rebuilding their trust. That Emby holds itself fully responsible when mistakes happen. Be more responsible in the future and move on from this. This may already be occuring and they are working to a solution. They will do the right thing.


Edited by speechles, 04 March 2019 - 05:38 PM.

  • rbjtech likes this

#23 dbott OFFLINE  

dbott

    Advanced Member

  • Members
  • 46 posts
  • Local time: 08:44 PM

Posted 04 March 2019 - 06:26 PM

So what function should I be afraid of?

At the moment I am afraid of the whole product!

 

@Luke mentioned the fix is in the next release, so from that I take it I am currently on a version with this issue and if all the things line up BOOM the server decides all my media gets deleted.

 

the OP lost everything, I have stuff in my collection that I ripped years ago from DVDs and BR disks, I have at least 3 TV Shows I ripped (hundreds of episodes) and manually converted to mkv's with hundreds of hours of work going into that. If this is at risk I would like to know about it.

 

Thats my point. :)  As I mentioned, not even sure what I can or can't touch at this point.  Will it do the same to TV and photos and music?  If so, what should I not do?  Again...No notices. 
 
Understand...To me...You have said yes, there is a major issue...yet at the same time not putting out a notice make me feel like I did something.  Not the case.  If you know for a fact that refresh did not make for this issue, and I can see the logic, then you may care to change the title of the thread at least to be correct as to what made for the issue.  Do you actually know for sure what at this point what actually caused it?  Have you guys gone and reproduce the issue? (Note that I have done delete media via my Kodi box that is using the Emby plugin without such issue.  The deletes in this case that I did were done via the Web Browser interface.  Not sure how the code works...But just adding that.
 
And not knowing what I should NOT do kills me. Again, scared for my TV shows, music, and photos so for now, Emby if off-line and I needed to rebuild Kodi and using a Kodi database to what shows. (At least it did not take hours to scan all the movies.) 
 
I am starting to get lost for words and more upset actually about nothing happening to get the word out by Emby or a roll out a release NOW that fixes the issue vs waiting for the next release. At least people with auto update on would then get a fix and can be safe.  This could be a bomb waiting to happen for how many people?
 
Luke, I don't post here a lot. I really have had very little need to. I did by a lifetime license as I liked the product at the time over Plex who actually gave me a free license (all products, plugins and all)  so thats does say something. :)  You know however who I am, or maybe you don't. (If not, reach out as you may care to.)  I surely could write a post on other communities about how Emby deleted my collection which surely would not go over to well and would for sure be picked up by all the major geek and AV sites. But no, I am taking it for what it is...Someone just messed up. I am not looking to take down Emby which I truly have enjoyed using.  But it is what is happening, or I should say lack of happening now, that is upsetting to me.
 
Sure, this was a movie collection and I DID not have a backup of my collection. I am running unRAID so if I lost a drive I could rebuild, so I do have some protections in place. But not for something like this. Even unRAID's Recycle Bin plugin would not have saved me as it only works if a delete happens from over the network, not from a local call on the server. I did post this over on the unRaid community under the Emby Docker thread to let people know. Know what I got? People blasting me for not having a backup and thus it was "my fault", not Emby, for not having a backup if I "really cared about my data".  I do backup items that are very important offsite.  But Movies, TV, & Music I did not think need it that much protection as I was protected from a drive failure at least. (But yes, there are movies I cannot get back as they can no longer be found.) My photos are on Google Drive so they are safe. 
 
Side Note...My wife and I for the last 5 years live full time in a motor coach traveling the US. As such, my downloads all come from AT&T and Verizon JetPacks. So, well, speed and data use is an issue. So I will not be able to get 17TB of movies back nor could I have sent backups off site as that is just a lot of data use. :)  Right now we are just trying to remember ones we had that we still needed to watch. 
 
Shameless plug...The movie "The Challenger Disaster" (2019) is an independent movie that my wife and I Executive Produced just came out on January 24th. The true story of how a few engineers tried to stop the Challenger Shuttle from being launched. So we did add that one back to the collection already. :)


#24 ebr OFFLINE  

ebr

    Chief Bottle Washer

  • Administrators
  • 52011 posts
  • Local time: 09:44 PM

Posted 04 March 2019 - 06:48 PM

So what function should I be afraid of?

 

The only feature that deletes your media files is the explicit "Delete" function.  If you are worried about this happening, then do not use the Delete feature in an Emby app (or disable it for all users) and then this can never happen to your library.

 

This is extremely unfortunate but errors do happen.  We believe this one was very rare as this has happened to two or three people out of tens of thousands.  Of course, that is no consolation to those two or three folks so we are sincerely sorry about that.

 

Also, this isn't a situation where there is a bug in the release that causes this to happen.  We believe this happened due to a problem in the conversion process from an older version of the server to the newer one.  The problem was data-related as opposed to logic related and requires a complex set of circumstances for it to have happened.



#25 dbott OFFLINE  

dbott

    Advanced Member

  • Members
  • 46 posts
  • Local time: 08:44 PM

Posted 04 March 2019 - 07:21 PM

The only feature that deletes your media files is the explicit "Delete" function.  If you are worried about this happening, then do not use the Delete feature in an Emby app (or disable it for all users) and then this can never happen to your library.

 

This is extremely unfortunate but errors do happen.  We believe this one was very rare as this has happened to two or three people out of tens of thousands.  Of course, that is no consolation to those two or three folks so we are sincerely sorry about that.

 

Also, this isn't a situation where there is a bug in the release that causes this to happen.  We believe this happened due to a problem in the conversion process from an older version of the server to the newer one.  The problem was data-related as opposed to logic related and requires a complex set of circumstances for it to have happened.

 

So what you are saying now is that I have a corrupt database caused by an Emby conversion (that took place between what ver and what version?) and thus is an issue just waiting to happen.  What is wrong with the database and HOW DO WE FIX IT!!!!  

 

If this is the case, then you have a larger issue on your hands seeing it deleted based on the database!  Now you need to fix everyones databases it would seem as mine would have converted just like anyone else's.  I run nothing special. No hacks. Nothing! 

 

So you say "We believe this happened..."  which now gives me even more cause for concern as it would seem to indicate you are not really sure.

 

Now I think I do need to write something up about this as it seems you already knew about such an issue as it already took place to others, yet here I am with lost data that you all KNEW could happen to others. But what the heck, it's only a few people, so no biggie, right?  

 

This is now reads as "Oh, sorry, you are one of the unlucky ones." (Maybe I should play the lottery then.) However for me it now becomes... "ALERT: Emby makers knew of an issue that could make for TOTAL DATA LOSS and does nothing to alert their users!"  Oh, Plex is going to love this one. 

 

WOW! Just WOW!


Edited by dbott, 04 March 2019 - 07:21 PM.

  • angusleung likes this

#26 revengineer OFFLINE  

revengineer

    Advanced Member

  • Members
  • 774 posts
  • Local time: 09:44 PM

Posted 04 March 2019 - 07:44 PM

A long time ago, I had a single movie disappear and I could not figure out how it happened. The precaution I took after that  is to let emby only touch copy of my movies. This way I can fix everything with a single mirror. However, my movies are only around 5 TB, which is more manageable than the larger collection of the OP.



#27 cayars OFFLINE  

cayars

    Advanced Member

  • Alpha Testers
  • 3393 posts
  • Local time: 09:44 PM

Posted 04 March 2019 - 07:51 PM

Around 2019-03-02 08:30 AM ish
A Simple Favor (2018)
The 1517 to Paris (2018)
 
Around 2019-03-02 12:10 PM ish
The Humanity Bureau (2017)
One Under the Sun (2017)
Finding Your Feet (2017)
911 (2017)
12 Years a Slave
The 13th Warrior
27 Dresses
The 33
About Time
 
Can you confirm you did in fact delete the above manually?
 
At this point it all looks good....Then BAM!!!

I'm not convinced this is an Emby issue at all.

If you look at the logs you'll see many directories already missing before the refresh took place.

Start looking at around 2019-03-02 12:17:14.417 where it's working with image files and can't because the directories don't exist.

 

There are no entries in the log that I saw of actual physical deletes except for the manual ones.  I do see however a lot of database deletes starting with the refresh (as the folders don't exist anymore so they are being removed from the DB).

 

Am i missing something?

 

Now as to the actual physical removal.  That could have been from a bug in code and files/folders got wacked when one of the manual ones was deleted.  It could have been a file system error just as easily.  It only takes one cross linked entry of a file to have this effect.

 

I'm not a huge fan of "deletes" in media software at all myself and don't use it.  However it would probably be a great idea to not ever delete anything but "move" the files to /EmbyDelete or similar on each drive.  Have it work similar to Windows Recycle bin.  It's one thing to remove DB entries but quite another to physically remove the media.  Having some type of "recycle" bin would be advantageous as well if a user with delete privs removes something the admin didn't really want removed as well.



#28 dbott OFFLINE  

dbott

    Advanced Member

  • Members
  • 46 posts
  • Local time: 08:44 PM

Posted 04 March 2019 - 08:09 PM

Not a file system issue.  It took out two different user shares.  DVD and Movies.  Both directories wiped empty. 



#29 mickle026 OFFLINE  

mickle026

    Advanced Member

  • Members
  • 411 posts
  • Local time: 02:44 AM

Posted 04 March 2019 - 08:25 PM

I had this or very similar "file system error" PRIOR to the database update.  A while ago now that wiped out my movie collection of 10 years of stb recordings.

I had been using the Web interface a long time before this suddenly happened out of the blue.

 

I know it was Emby related, but couldn't find what it was.  In my case it was hurtful but not the end of the world  - just image if it were home movies  ?

 

I do agree though that Emby could do this differently.  Even just a looped/recursive delete so x amount delete files happen only rather than a file system folder delete instruction.  How fast does it need to be?, it doesn't even need to be on the main GUI thread.

 

 

This must NEVER happen and there are simple things that can be done to stop it, now Im not telling the creators how to program but ...

 

 

if DeletePath != RootPath    '<----the base directory

{

       Directory.Delete(Path);

}

else

{

       MessageBox.Show("Really delete everything in this Library Path ?? OMG... Are you SURE ??? ");

}

 

 

 

Or something simple like prompting the user

 

"There are 3 Movie files and 10 images to delete - are you sure?"  <-- with a button to show a list of them

 

as opposed to:

 

"There are 40,000 files to delete - 15,000 Movie file types and 25,000 images - Im not going to ask, just going to start deleting ......"


Edited by mickle026, 04 March 2019 - 08:27 PM.

  • angusleung likes this

#30 jaquestati OFFLINE  

jaquestati

    Advanced Member

  • Members
  • 170 posts
  • Local time: 06:44 PM

Posted 05 March 2019 - 01:55 AM

for once i'm glad that 99% of my stuff is on a read only unionfs gdrive that emby/plex can't write to :) :(

if i lost triple digit terabytes i'd go out and play in traffic :) :(


  • denz, cayars and Caascada like this

#31 cayars OFFLINE  

cayars

    Advanced Member

  • Alpha Testers
  • 3393 posts
  • Local time: 09:44 PM

Posted 05 March 2019 - 03:03 AM

Not a file system issue.  It took out two different user shares.  DVD and Movies.  Both directories wiped empty. 

That is even more indication of what I was saying that I think this was not Emby's doing.

 

It's one thing to delete files under a specific folder/directory but now you're saying it's two different shares?

This sounds like something else to me.  I also don't see the deletes in the logs, only "delete DB entries" because the folders weren't present on the disc (during the refresh).

 

I would check the OS logs and any other software also running on that system.  Check discs, etc.

Thus far there is no proof of Emby doing this, just speculation.



#32 cayars OFFLINE  

cayars

    Advanced Member

  • Alpha Testers
  • 3393 posts
  • Local time: 09:44 PM

Posted 05 March 2019 - 03:47 AM

Luke said:

"Ok it looks like you did actually use the delete media function, and for whatever reason it may have gone up an extra level when deleting the movie folder. I will have to add more logging to confirm this. I believe we have noticed this issue being associated with the database upgrade from 3.5, but it does not occur on newly created libraries. It is already resolved for the next release so that it will not happen again. I apologize for the disruption."

 

However, I'm not sure that is correct either because there were manual deletes (what Luke might have seen in the logs) which did work correctly it appears.  It was not until the REFRESH was run that everything was logged and removed from the DB.

 

Notice Luke said "might have went up a level" and he added additional checks for this.  It's a guess and he added further protection to make sure this doesn't happen (not that it did happen).  It's certainly just as likely that the file system had cross linked files/folders and the delete of one of the files caused this as well.

 

There were folders already missing before the "DELETE" happened if you look at the logs.  Emby was logging errors when trying to work with images that didn't exist anymore (before the deletes).  The simple fact is we DO NOT KNOW what caused this and the logs show issues already on the system.

 

I'm certainly all for as much protection in the code as possible when it comes to deletes or file modification of any kind.  I just don't agree with the overall sentiment about "publishing a problem" or the over reacting that has taken place in the thread when we don't know what the cause is.  I also don't agree that the logs show "clearly" what took place.  The only thing I see clearly is the delete from the DB when the refresh took place which should have happened if the folders didn't exist.



#33 RobbyBobby OFFLINE  

RobbyBobby

    Advanced Member

  • Members
  • 30 posts
  • Local time: 01:44 AM

Posted 05 March 2019 - 05:15 AM

Let's be honest, this is the about the most unfun thing that can happen to a user's media collection and I speak from personal experience. I once had a hard drive (in enclosure) connected to my Mac via USB. I connected it to a Windows virtual machine using the VM interface and BAM! The entire directory of the hard drive just vanished. 

 

So I feel the OP's pain. (And it was only a couple of TB, probably far less.)

 

I went the route of upgrading Emby 3.x to Emby 4. I asked on these very forums whether I would be better off doing a fresh install and the word was it *should* all be fine.

 

I'd like some clarity on what's gone on here to be honest and whether I should do as I originally intended and start afresh? I think it would put everyone's mind at rest to ascertain precisely what happened.



#34 ~Achilles~ OFFLINE  

~Achilles~

    Member

  • Members
  • 22 posts
  • Local time: 05:44 PM

Posted 05 March 2019 - 06:31 AM

We all empathize with the OP. Some of us have and some of haven't had to learn the hard way that RAID/parity is not a backup. Backups have to be part of your system design planning from the start--otherwise you are just playing with fire.

 

It is unfortunate that this happened to the OP. For this very reason, I do not allow any media management software RW permissions to my content. Emby/Plex gets RO permissions to content dirs. The only dir it gets RW to is the DVR dir.



#35 cayars OFFLINE  

cayars

    Advanced Member

  • Alpha Testers
  • 3393 posts
  • Local time: 09:44 PM

Posted 05 March 2019 - 08:39 AM

This all lines up with what the dev team have outlined as the issue and the symptoms look consisted with the edge case.

Yes it might be something else, disk or system failure or cosmic rays or the OP deleted all the files and forgot he did or his cat did it. But from what we can see it looks like Emby did it.

 

Next time you are having a heated discussion with your partner and they are doing the freak out tell them they are "over reacting", then after you have put out all the fires remember this thread :-)

Again, I'm not trying to say it couldn't possibly have been an Emby issue. I'm just stating it could be other things as well as Emby that is responsible. The fact that multiple shares came up missing is tell tail IMHO. I do think the idea to drop everything and post a warning message is premature however.

 

I myself like hard facts and not speculation. Just my opinion.

 

We all empathize with the OP. Some of us have and some of haven't had to learn the hard way that RAID/parity is not a backup. Backups have to be part of your system design planning from the start--otherwise you are just playing with fire.

 

It is unfortunate that this happened to the OP. For this very reason, I do not allow any media management software RW permissions to my content. Emby/Plex gets RO permissions to content dirs. The only dir it gets RW to is the DVR dir.

Can't say more about proper backups.  It's not a matter of if you'll need them, but when.

 

Another pretty simple way to protect against media deletion lies in file permissions on pretty much ever OS.  Of course you can't do this if you INTEND on using delete features.

 

I too only allow RW privs to my DVR section.  Everything else is Read Only for the media files.



#36 dbott OFFLINE  

dbott

    Advanced Member

  • Members
  • 46 posts
  • Local time: 08:44 PM

Posted 05 March 2019 - 11:50 AM

To me, you tell Emby you have these two directories that have movies in them. So Emby uses them to look in and do things to anything you ask it to do within the directories as they were put into it as they contain Movies.  So you tell Emby to refresh Movies it does not just look at ONE directory, it handles any and all them listed in that section as a directory containing movies.   

 

So yeah, I surely can see how Emby went though and removed it all just as it can go scan it all to make your collection and manage it. I was dealing with Movies, to that Emby that means X Y Z directories are involved in whatever I ask of it. 


  • viperkc likes this

#37 cayars OFFLINE  

cayars

    Advanced Member

  • Alpha Testers
  • 3393 posts
  • Local time: 09:44 PM

Posted 05 March 2019 - 01:02 PM

So you had two different paths added to your movie library and both got removed correct?



#38 dbott OFFLINE  

dbott

    Advanced Member

  • Members
  • 46 posts
  • Local time: 08:44 PM

Posted 05 March 2019 - 01:07 PM

Correct, they were both listed under FOLDERS in the Library Setup area for Movies.  The older one was DVD back in the, well, DVD days as they were different. The came alone just making one for Movies for the new era.

 

Like this...

 

Movies...

 

Folder:  /mnt/user/Movies

Optional: nfs://192.168.1.200/mnt/user/Movies

 

Folder:  /mnt/user/DVD

Optional: nfs://192.168.1.200/mnt/user/DVD


Edited by dbott, 05 March 2019 - 01:29 PM.

  • cayars likes this

#39 dbott OFFLINE  

dbott

    Advanced Member

  • Members
  • 46 posts
  • Local time: 08:44 PM

Posted 05 March 2019 - 06:41 PM

Hey...

 

Seeing that there is mention of a database conversion/upgrade issue that may be the reason this happened...Should we be wiping out the database starting over or something?  It would be really nice to have some guidance here.

 

Thanks



#40 Happy2Play OFFLINE  

Happy2Play

    Trial and Error

  • Moderators
  • 19168 posts
  • Local time: 06:44 PM
  • LocationWashington State

Posted 05 March 2019 - 06:54 PM

Guidance will be hard as the odds of what happened to you will ever happen to me.  I can Refresh/replace all images all day long and never have this happen to me with Emby having full control of my media.

 

But the odd are high that the issue resides/resided in DVD rips library though.  But not knowing exactly how you deleted the media in Metadata manager, it is hard to say exactly.  But I know no matter how many times I delete in the metadata manager or Refresh libraries I can not get my entire library to delete.  






0 user(s) are reading this topic

0 members, 0 guests, 0 anonymous users