Jump to content


Photo

Emby Just Deleted ALL of my Movies


  • This topic is locked This topic is locked
198 replies to this topic

#41 dbott OFFLINE  

dbott

    Advanced Member

  • Members
  • 46 posts
  • Local time: 11:55 PM

Posted 05 March 2019 - 08:48 PM

Just to be clear, this was not the Meta Data refresh that caused the DELETES, this action just triggered the clean up of all the missing items that were no longer on disk.

The recursive delete happened when the DELETE action (one of your manual ones) got confused and went back too many directories to start the full deep recursive DELETE nuking everything.

This was due to a DB update that happened in upgrading from 3.5 -> 4.x, more than likely a field that should have been updated was not or it was when it should not have been etc.

 

So I ask again....WHAT SHOULD WE BE DOING? For if this was the case, then it also could be the case for anything thing else.  Photos, Music, TV.  Do they have something that will go through and check everyones database? Should we kill the database and have it rebuild from scratch? (If so, they should provide a way to be sure to get watched status back. I use Trakit but guessing not everyone does.)

 

The issue is Emby caused it...PERIOD.  How to be sure we are now safe is the question at hand before it happens again.  



#42 Happy2Play OFFLINE  

Happy2Play

    Trial and Error

  • Moderators
  • 19207 posts
  • Local time: 09:55 PM
  • LocationWashington State

Posted 05 March 2019 - 10:02 PM

Okay testing this if you delete a DVD Rip within a library, it clears the entire library folder. Deleting on the item or the metadata manager.

process to reproduce (Have not tried on a new database yet.)
Backuped up a test library.

Put one DVD rip in that library
Selected Delete on that DVD rip via itemdetails, after clicking delete there was a error pop-up

There was an error deleting the item from Emby Server. Please check that Emby Server has write access to the media folder and try again.

Looked at library folder (explorer) and it is now empty (one misc empty sub folder remained). This happened fast as majority of the item were dummy files, but folder size was 32GB.

This behavior only happens on Disc rips, I currently do not have a Bluray disc to see if the same happens with them also. 

I thought we already identified this though.
4.0.2.0 and 4.1.0.12

 

Deleting media that is not a disc does not show this destructive behavior.

Attached Files


Edited by Happy2Play, 05 March 2019 - 10:05 PM.

  • CBers and FrostByte like this

#43 dbott OFFLINE  

dbott

    Advanced Member

  • Members
  • 46 posts
  • Local time: 11:55 PM

Posted 05 March 2019 - 10:07 PM

Happy2Play....Ouch!!!!  Wow!  Nice going!   But scary to say the least!  

 

I have done the user non-delete setting.  But the thought is that there is still a time bomb waiting to happen to others if this was an upgrade to the database issue.  That is why I asking if we should rebuild from scratch. If it is a DB issue, they need to release a repair/check tool or something.  Or like I said, it could be waiting to happen to others.

 

But now it seems like it might not be DB related but how the code interprets as a directory structure. 

 

THANK YOU for looking more at it.  I have not heard from anyone else on it at all.  I hope what you found they then can look to fix quickly as it seems this could burn others.


Edited by dbott, 05 March 2019 - 10:14 PM.


#44 Happy2Play OFFLINE  

Happy2Play

    Trial and Error

  • Moderators
  • 19207 posts
  • Local time: 09:55 PM
  • LocationWashington State

Posted 05 March 2019 - 10:18 PM

Also note I had two folders attached to this library and it only affected the folder that contained the DVD rip.

 

Will perform the same test on a new database next.


Edited by Happy2Play, 05 March 2019 - 10:21 PM.


#45 Happy2Play OFFLINE  

Happy2Play

    Trial and Error

  • Moderators
  • 19207 posts
  • Local time: 09:55 PM
  • LocationWashington State

Posted 05 March 2019 - 10:41 PM

Only tested in 4.1.0.12, with a new database I got the same behavior as post 58.

 

5c7f32c97c1ca_mm.jpg

 

Everything in Movies - NFO was deleted by deleting this DVD rip.

 

Doubt it matters but CollapseVideoFolders=true.  Also only tested via local web client.


Edited by Happy2Play, 05 March 2019 - 11:04 PM.

  • CBers, SikSlayer and maegibbons like this

#46 mickle026 OFFLINE  

mickle026

    Advanced Member

  • Members
  • 416 posts
  • Local time: 05:55 AM

Posted 06 March 2019 - 05:02 AM

Someone has pointed out before (months ago) in this Beta forum that deleting a DVD causes this, just cant remember who or when.



#47 PenkethBoy OFFLINE  

PenkethBoy

    Advanced Member

  • Members
  • 4315 posts
  • Local time: 05:55 AM
  • LocationWarrington,UK

Posted 06 March 2019 - 07:51 AM

I can reproduce what H2P found and have tested a BluRay folder

 

With DVD Folder

 

If i delete a Movie (DVD folder) from a library - it deletes the Library ROOT folder - hence why all movies are disappearing for users

But the notification is a debug notification so unless a user has that enabled they wont see anything in the logs initially and the subsequent boat load of debug errors related to missing content

2019-03-06 11:20:09.219 Info HttpServer: HTTP DELETE http://192.168.1.56:8096/emby/Items/2171. UserAgent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/72.0.3626.119 Safari/537.36
2019-03-06 11:20:09.226 Info App: Deleting item, Type: Movie, Name: Godzilla, Path: F:\EmbyTest\MovieTest2\Godzilla, Id: 2171
2019-03-06 11:20:09.226 Debug App: Deleting path F:\EmbyCore\programdata\metadata\library\ed\edcde71b0f4d6fa0bb7a8920a5547544
2019-03-06 11:20:09.229 Debug App: Deleting path F:\EmbyTest\MovieTest2
2019-03-06 11:20:09.232 Debug LibraryMonitor: New file refresher created for F:\EmbyTest\MovieTest2\22 Bullets (2010)\22 Bullets (2010).Forced.Eng.srt
2019-03-06 11:20:09.744 Debug LibraryMonitor: Resetting file refresher from F:\EmbyTest\MovieTest2\22 Bullets (2010)\22 Bullets (2010).Forced.Eng.srt to F:\EmbyTest\MovieTest2\22 Bullets (2010)
2019-03-06 11:20:09.884 Debug LibraryMonitor: New file refresher created for F:\EmbyTest\MovieTest2\Godzilla\AUDIO_TS
2019-03-06 11:20:09.884 Debug LibraryMonitor: Resetting file refresher from F:\EmbyTest\MovieTest2\Godzilla\AUDIO_TS to F:\EmbyTest\MovieTest2\Godzilla
2019-03-06 11:20:09.899 Debug LibraryMonitor: Resetting file refresher from F:\EmbyTest\MovieTest2\22 Bullets (2010) to F:\EmbyTest\MovieTest2
2019-03-06 11:20:09.899 Error LibraryMonitor: Error in Directory watcher for: F:\EmbyTest\MovieTest2
	*** Error Report ***
	Version: 4.1.0.12
	Command line: F:\EmbyCore\system\EmbyServer.dll
	Operating system: Microsoft Windows NT 6.2.9200.0
	64-Bit OS: True
	64-Bit Process: True
	User Interactive: True
	Processor count: 8
	Program data path: F:\EmbyCore\programdata
	Application directory: F:\EmbyCore\system
	System.ComponentModel.Win32Exception: System.ComponentModel.Win32Exception (5): Access is denied
	Source: 
	TargetSite: 
	No Stack Trace Available

First line is me deleting Godzilla (DVD) manually

Fourth Line is Emby deleting the ROOT Library Folder !

22 Bullets - is another movie (not a DVD) i had in this test library.

When you do the delete (from Movie Page) - you get the normal warning that its going to delete the movie from disc - fine thats what i want - poof all movie disappear from the movie Page

 

With a BluRay folder

 

basically the same result

2019-03-06 11:29:22.622 Info HttpServer: HTTP DELETE http://192.168.1.56:8096/emby/Items/2210. UserAgent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/72.0.3626.119 Safari/537.36
2019-03-06 11:29:22.623 Info App: Deleting item, Type: Movie, Name: 300, Path: F:\EmbyTest\Movies 3D\300, Id: 2210
2019-03-06 11:29:22.623 Debug App: Deleting path F:\EmbyCore\programdata\metadata\library\17\17c85177b0e251fe3462b45500897a8c
2019-03-06 11:29:22.623 Debug App: Deleting path F:\EmbyTest\Movies 3D
2019-03-06 11:29:22.624 Debug LibraryMonitor: New file refresher created for F:\EmbyTest\Movies 3D\300\300.nfo
2019-03-06 11:29:22.624 Debug LibraryMonitor: Resetting file refresher from F:\EmbyTest\Movies 3D\300\300.nfo to F:\EmbyTest\Movies 3D\300
2019-03-06 11:29:22.770 Debug LibraryMonitor: New file refresher created for F:\EmbyTest\Movies 3D\Justice League (2017)\banner.jpg
2019-03-06 11:29:22.770 Debug LibraryMonitor: Resetting file refresher from F:\EmbyTest\Movies 3D\Justice League (2017)\banner.jpg to F:\EmbyTest\Movies 3D\Justice League (2017)
2019-03-06 11:29:23.278 Debug LibraryMonitor: Resetting file refresher from F:\EmbyTest\Movies 3D\300 to F:\EmbyTest\Movies 3D
2019-03-06 11:29:23.278 Error LibraryMonitor: Error in Directory watcher for: F:\EmbyTest\Movies 3D
	*** Error Report ***
	Version: 4.1.0.12
	Command line: F:\EmbyCore\system\EmbyServer.dll
	Operating system: Microsoft Windows NT 6.2.9200.0
	64-Bit OS: True
	64-Bit Process: True
	User Interactive: True
	Processor count: 8
	Program data path: F:\EmbyCore\programdata
	Application directory: F:\EmbyCore\system
	System.ComponentModel.Win32Exception: System.ComponentModel.Win32Exception (5): Access is denied
	Source: 
	TargetSite: 
	No Stack Trace Available

First line is me deleting 300 (BluRay) manually

Fourth Line is Emby deleting the ROOT Library Folder !

Differences i see is that on the Movie home screen - other movies do not disappear after i deleted "300" in fact the entry still worked for Justice League in that you could browse the entry see all images/metadata etc

And even after running a library scan - the entry is still there only missing its primary image

[edit] - Justice League db entry - does not get deleted with a library scan - another bug - this will lead to all sorts of extra problems of duplicate entries as and when a user re adds stuff to their library

 

So dont delete - disc image folders from you Library!

 

Not tested on a TV library or other types of library - but i suspect its the same result


Edited by PenkethBoy, 06 March 2019 - 08:47 AM.

  • CBers, SikSlayer, FrostByte and 1 other like this

#48 cayars OFFLINE  

cayars

    Advanced Member

  • Alpha Testers
  • 3418 posts
  • Local time: 12:55 AM

Posted 06 March 2019 - 08:34 AM

Now it's time for a Hot Fix and potential warning. :)

 

We probably need to test some other potential disc formats as well.  ie  Multiple movies in the same directory, mixed movies in folders and not in folders, multi-versions folders, etc.  Basically everything that's not a single movie in a single folder.


  • CBers, SikSlayer, d00zah and 2 others like this

#49 PenkethBoy OFFLINE  

PenkethBoy

    Advanced Member

  • Members
  • 4315 posts
  • Local time: 05:55 AM
  • LocationWarrington,UK

Posted 06 March 2019 - 08:48 AM

I look forward to the results of your testing  :P


  • CBers, SikSlayer, denz and 1 other like this

#50 cayars OFFLINE  

cayars

    Advanced Member

  • Alpha Testers
  • 3418 posts
  • Local time: 12:55 AM

Posted 06 March 2019 - 09:41 AM

I look forward to the results of your testing  :P

Working on different combinations on my test machine.

DVD, Blueray I confirm (started with them).

Testing the rest in about an hour.

I'll post what I tested.


  • SikSlayer, pĆ¼nktchen and Senna like this

#51 Senna OFFLINE  

Senna

    iOS - iPadOS - tvOS

  • Members
  • 1257 posts
  • Local time: 06:55 AM

Posted 06 March 2019 - 10:06 AM

Someone has pointed out before (months ago) in this Beta forum that deleting a DVD causes this, just cant remember who or when.

It is in this topic, which bleeded to dead, with no responses from Devs anymore in that topic, after the findings unfortunately.....

https://emby.media/c...e-4#entry703995


Edited by Senna, 06 March 2019 - 10:27 AM.


#52 PenkethBoy OFFLINE  

PenkethBoy

    Advanced Member

  • Members
  • 4315 posts
  • Local time: 05:55 AM
  • LocationWarrington,UK

Posted 06 March 2019 - 10:45 AM

Looks like at least four users lost their libraries - one more than once - from a quick read of that thread

 

One other point - re my posts above from today - the deletion of files was on a brand new db (deleted library db and let emby recreate it with .12 beta) - so the upgrade from an older DB is now very unlikely to have any influence on when this happens.


Edited by PenkethBoy, 06 March 2019 - 10:47 AM.

  • CBers likes this

#53 clarkss12 OFFLINE  

clarkss12

    Advanced Member

  • Members
  • 1256 posts
  • Local time: 12:55 AM
  • LocationMichigan, USA

Posted 06 March 2019 - 11:04 AM

This is a scary situation.  I have been slowly upgrading my DVD movies, to BlueRay OR 4k.  I guess I have been lucky so far, since I use Windows explorer to delete the folder that contains my movie that I just upgraded.

 

BUT, when I see a duplicate of a TV recording using Emby, I delete the duplicate right from the Emby Android app.  Been lucky so far.



#54 ebr OFFLINE  

ebr

    Chief Bottle Washer

  • Administrators
  • 52103 posts
  • Local time: 12:55 AM

Posted 06 March 2019 - 11:37 AM

Only tested in 4.1.0.12, with a new database I got the same behavior as post 58.

 

5c7f32c97c1ca_mm.jpg

 

Everything in Movies - NFO was deleted by deleting this DVD rip.

 

Doubt it matters but CollapseVideoFolders=true.  Also only tested via local web client.

 

Was there only ONE DVD rip in that test or multiples?



#55 jaquestati OFFLINE  

jaquestati

    Advanced Member

  • Members
  • 170 posts
  • Local time: 09:55 PM

Posted 06 March 2019 - 02:23 PM

they should just put out a release version and a beta with delete disabled/removed... and warn everyone to down/upgrade... at least temporarily ...
i'm having nightmares over this : ) : (



#56 ebr OFFLINE  

ebr

    Chief Bottle Washer

  • Administrators
  • 52103 posts
  • Local time: 12:55 AM

Posted 06 March 2019 - 02:25 PM

they should just put out a release version and a beta with delete disabled/removed... and warn everyone to down/upgrade... at least temporarily ...
i'm having nightmares over this : ) : (

 

Hi.  If you disable the delete feature on your server, then this cannot happen.



#57 dbott OFFLINE  

dbott

    Advanced Member

  • Members
  • 46 posts
  • Local time: 11:55 PM

Posted 06 March 2019 - 02:30 PM

Hi.  If you disable the delete feature on your server, then this cannot happen.

 

Hummm...That really is not an option....I don't think people want to do that on their entire server.  This is after all an Emby issue and thus should be resolved by Emby, as I sure it will be, but still not pulling or putting out a notice is really poor.

 

After all, as you can see above, a number of people have reproduced what happened to me...The question is...Why have you all not done so when it was talked about in the past?  If it was handled, I would not have lost 17TB of data!


Edited by dbott, 06 March 2019 - 02:34 PM.


#58 jaquestati OFFLINE  

jaquestati

    Advanced Member

  • Members
  • 170 posts
  • Local time: 09:55 PM

Posted 06 March 2019 - 02:39 PM

well 99% of my stuff is on a read only unionfs drive... (which i'm usually annoyed by.. but now i'm thankful... or i might have lost like 100s of terabytes... and taken to the bridge...)

: (

but for the non technical out there... releasing safe versions till it's fixed seems the best way to go... imho...



#59 dbott OFFLINE  

dbott

    Advanced Member

  • Members
  • 46 posts
  • Local time: 11:55 PM

Posted 06 March 2019 - 02:47 PM

well 99% of my stuff is on a read only unionfs drive... (which i'm usually annoyed by.. but now i'm thankful... or i might have lost like 100s of terabytes... and taken to the bridge...)

: (

but for the non technical out there... releasing safe versions till it's fixed seems the best way to go... imho...

 

I think that is great for you. Even more do with the data you could loose. :)  I would have to think most users of Emby, like Plex, don't have the knowledge or understanding of what is needed to be really safe.  They use media managers to control their media and make life easy.  So to them, like I, I would have NEVER, EVER, thought that Emby or Plex could do such a thing. 

 

It is more upsetting to me to read above that it had been seen before, and we are not talking like two weeks ago.  Yet here I am with lost data.

 

What also gets me is it went though any directory listed as a MOVIE directory.  I had two directories...DVD and MOVIES, both were listed under movies so they could be seen and access and of course used. Seeing that BOTH were wiped, it could only be code that did this as it would have to be told to traverse both shares. 



#60 dbott OFFLINE  

dbott

    Advanced Member

  • Members
  • 46 posts
  • Local time: 11:55 PM

Posted 06 March 2019 - 02:51 PM

It seems like the word is getting out, even without me posting anything anywhere else, which could mean bad things for Emby, as this thread has now over 1700 views. At some point someone will write an article and then that WILL BE bad for Emby if they have not taken action. 

 

You would rather have it read..."Bug found in Emby that could loose all your media, but heres is the patch." with a spin that they acted quickly.  Vs.  "Bug found in Emby that could loose all your media! Shut it down to be safe!"  (Or something like that.)






0 user(s) are reading this topic

0 members, 0 guests, 0 anonymous users