Prevent Emby from removing missing media

[dominicM 2]

I had a drive that had a corrupt filesystem. This meant the files were and some still are offline. The issue is that Emby seems to have removed the media files from the library. When some of the files were restored they had a new imported date. This means that there is permanent metadata loss which is annoying as I primarily use Date added sorting option.

 

I have "Date added behavior for new content:" set to "Use date scanned into the library". The other option produces identical sorting order too presumably because there is no standard creation date in Linux.

 

Question is how can I prevent Emby Server from automatically removing files that are missing? I do not see any option in settings related to this.

[Luke 42078]

Hi there, can we look at an example? please attach the information requested in how to report a problem. thanks !

[dominicM 2]

Hi there, can we look at an example? please attach the information requested in how to report a problem. thanks !

 

Essentially one disk was removed from a mergerfs pool so files are simply missing to emby server. I assumed it was removed by emby when the library was scanned (drive was missing for at least 2 weeks). Was this not the expected default behavior? When I recovered some files they appeared at the top when sorted by Date added. Some content was not recoverable and was downloaded via radarr (mdate was set to release date). These files too appeared at the top. I looked at one file that is not yet recovered (currently missing) and it does not appear in Emby Library so it was removed which explains why the modified date is messed up.

 

I don't think the logs are of any use in this case as the library items would have been removed 2 weeks or more ago. 

[Luke 42078]

What version of emby server?

[dominicM 2]

What version of emby server?

 

Version 3.3.1.0 on Fedora 27

[dominicM 2]

I have seen this also and it was a pain, a drive failed in my pool, I replaced it and rebuild my files with my snapraid backup.

To do this I has to take the drive pool offline and Emby removed all my missing media items.

When I brought the pool back online Emby re-added all the items again which was fine but a lot of my watched statuses and imported dates were now reset.

 

I am not sure what can be done here though, I think it would be hard to work out what has legitimately been deleted compared to what is unexpectedly "missing".

 

Thanks for chiming in, now I know it's not just me. Seems to me like a single option like "do not remove missing files from the library" would solve this issue. There is a similar option in Sonarr and Radarr already.

[ebr 16182]

Thanks for chiming in, now I know it's not just me. Seems to me like a single option like "do not remove missing files from the library" would solve this issue.

 

How would you move or delete a large portion of your library outside of Emby?

 

The issue here is we designed Emby to be "self maintaining".  We did not want you to have to run an "import" process and then keep doing that to keep your Emby library in sync with your file system.  We wanted the system to "just work" without having to do that kind of maintenance.

 

So, then you end up with issues like this.  They can be solved but they aren't simple given our goal to be a self maintaining system.

[dominicM 2]

How would you move or delete a large portion of your library outside of Emby?

 

The issue here is we designed Emby to be "self maintaining".  We did not want you to have to run an "import" process and then keep doing that to keep your Emby library in sync with your file system.  We wanted the system to "just work" without having to do that kind of maintenance.

 

So, then you end up with issues like this.  They can be solved but they aren't simple given our goal to be a self maintaining system.

 

Importing is not the issue here at all. It's the removal that's the issue. I don't see how an option to prevent removal of missing files from the library cause any issues with importing. In my case it actually seems that watch state is maintained even after file was removed and re-downloaded. I am not sure what the intended behavior is exactly. It seems like it's just resetting date added in my case. This is still very annoying as I can't sort by date modified which s very important metadata. Worst thing is that with it will happen again and it's enough to just have one drive temporarily fail to mount for metadata to be permanently lost.

[dominicM 2]

Deleteing is important, as I point out this is going to be hard to determin actually deleted items and unexpectedly missing item.

 

Scenario:

I deleted a TV Show from my drive pool using Sonarr as all members of the family have finished watching it and we all voted to never watch that shit again.

How does the server treat that?

- Keeps it cos it might not actually have meant to be deleted?

- Removes it as it is now missing from the drive?

pick one, but how does the server know which.

 

So lets add an option to never remove items missing from disk.

Now how do I delete stuff from my systems unless its through the Emby server itself?

 

I am not sure there is an easy fix.

 

Perhaps caching an item and keeping the date added is if gets removed and re-added.

 

Valid point but at least in my case I will almost never be deleting stuff. If you actually want items to be removed then simply do not enable the option, seems like a fair option. Obviously it would be better if it was 100% automated but that's not always possible.

 

Actually thinking about it, why couldn't items simply be marked as missing? That would give a chance to remove it manually or have an option to remove missing files after X days (I would set it to something like 90 days). Even if items are not marked as missing but rather just removed (but kept internally as it seems to be now), date added could be kept as well solving the issue entirely. Right now re-added items seem to have all metadata except date added which is set again. I would consider this a bug as the date added should be the original date the item was added rather than when it was re-added.

Edited March 19, 2018 by dominicM

[dominicM 2]

Nah, I'll just get more spinning rust

 

Bug or not do you see any reason for resetting date added but not watched state or other metadata? I can't think of one which is why I think it's a bug.

 

 

I do think by far the easiest solution right now is to simply not reset date added when content is re-added. In the long run having missing tag would be ideal. Just for the love of god don't have an option hard coded to 7 days! This is nowhere long enough in many cases. I am still in recovery after 6 weeks, some files were lost and will take a long time to re-download and then to try to use them to recover others from parity.

[ebr 16182]

"Purge missing content" won't work well because we have a feature to track missing TV episodes and show them so that you can see what you don't have and may want to acquire.

 

Importing/deleting are tied together so that's why my explanation applies.

[dominicM 2]

"Purge missing content" won't work well because we have a feature to track missing TV episodes and show them so that you can see what you don't have and may want to acquire.

 

Importing/deleting are tied together so that's why my explanation applies.

 

Seems to me like there is a very simple solution. Simply have 2 variables for each media item instead of one. So you would track whether the item ever had a file associated or it was added to track missing items. For example instead of just "missing:bool" you would have something like "missing:bool" and "missingAlways:bool". If there was never a file associated with library item then do nothing as now. If there was a file associated in the past but it is now missing keep date added metadata when new file is added to same library item.

[Sp3kt3r 13]

Just want a say that I also agree to have a different maintenance for purging the media.

I also had power issue one of my external drive enclosure and didn't notice right away.

Emby removed all those movie and I had to scan library.

 

I actually deleted the automatic task from emby and run the maintenance my self and I make sure all drive are online.

[dominicM 2]

Just want a say that I also agree to have a different maintenance for purging the media.

I also had power issue one of my external drive enclosure and didn't notice right away.

Emby removed all those movie and I had to scan library.

 

I actually deleted the automatic task from emby and run the maintenance my self and I make sure all drive are online.

 

Can you elaborate on how your setup works now? If I can stop automatic tasks and do it manually that would be a better solution than risking metadata loss. In scheduled tasks I only see scan library task, is there a way to separate it into import/remove tasks?

[Carlo 4561]

Not sure what's hard about this.  Add a field "MissingDate" and during a scan fill in this field for content not found.  Leave all other fields intact.  Yes that means the content will still show up as normal and fail to play (big hint something is wrong).

 

Upon finding the media again remove the MissingDate and update the path if needed (if located in a new place).  All metadata will be just as it was to begin with.

 

Add a configurable input to each library to allow the Admin to set the amount of days required before data will be removed automatically.  This would allow me to set 365 days for my TV Shows and Movie Libraries but 0 Days for my News Library where I typically roll shows every few days and expect them to be removed.

 

If the admin or user with delete privs does this through an interface then delete it as usual.  If the DVR has a setting to keep only X shows or for X days then also honor that since it's a specific setting to only keep certain amount of data.

 

But the "MissingDate" would be applied to data that is just "gone" and would save these bulk removals by the system when we don't want this data removed.

[Happy2Play 9780]

Can you elaborate on how your setup works now? If I can stop automatic tasks and do it manually that would be a better solution than risking metadata loss. In scheduled tasks I only see scan library task, is there a way to separate it into import/remove tasks?

 

Click on the Task in Scheduled Tasks and you will see the Triggers and can add and delete them.

[dominicM 2]

Click on the Task in Scheduled Tasks and you will see the Triggers and can add and delete them.

 

There is only "Scan media library". Would disabling this not cause any other issues? I assume new items would still be added right?

[Happy2Play 9780]

There is only "Scan media library". Would disabling this not cause any other issues? I assume new items would still be added right?

 

All scanning does is look for changes in your Library, so no.  You may need to disable Real Time monitoring also.  It is a option per library.  Dashboard-Library-click on each Library

[dominicM 2]

All scanning does is look for changes in your Library, so no.  You may need to disable Real Time monitoring also.  It is a option per library.  Dashboard-Library-click on each Library

 

Ah ok, I got it now. Seems like a decent workaround, will keep it disabled from now on. Thanks!

[m33e 2]

I have a similar issue with this: Emby is Running on my NAS. But I recently ran out of space, so I moved some media to my htpc and mounted the external Volume on my NAS. For Power consumption reasons the htpc is in sleep mode while nobody is actively using the machine, which means that all the media on there is offline for Most of the day. So when the scheduled Library scan starts indexing & the Second pc is sleeping, emby removes all affected files & they will only get re-added once scanning coincides with a wake Phase of the htpc. I have 3 libraries (movies, tv, misc), each library has 2 folders assigned to it, where the First is a local Directory (NAS), and the second Always is a NFS Mount to the other machine).

 

I would really love to have a way to keep all my media indexed, offline or not, and still have emby pick up changes that occur on the NAS media folders during the day (newly Added media, actual file removals would be negligible).

Edited March 21, 2018 by m33e

[Luke 42078]

For those running the beta server there are still some in-progress changes that need to be completed.

[SamES 1057]

I think I've been hit by the same problem on server 3.3.1.9

 

I had samba crash on my storage server, and you can see in the log that watcher noticed the loss of network drive and then proceeded to delete all the metadata.  Now in the middle of a very long rescan.  I've had network paths go down before but have never experienced this happen.

 

@@Luke, I'll PM you the log

2018-03-28 10:44:41.770 Error LibraryMonitor: Error in Directory watcher for: \\10.55.1.2\Media\Demos
    *** Error Report ***
    Version: 3.3.1.9
    Command line: C:\Users\Home\AppData\Roaming\Emby-Server\system\EmbyServer.dll C:\Users\Home\AppData\Roaming\Emby-Server\system\EmbyServer.dll C:\Users\Home\AppData\Roaming\Emby-Server\system\EmbyServer.dll C:\Users\Home\AppData\Roaming\Emby-Server\system\EmbyServer.dll C:\Users\Home\AppData\Roaming\Emby-Server\system\EmbyServer.dll C:\Users\Home\AppData\Roaming\Emby-Server\system\EmbyServer.dll C:\Users\Home\AppData\Roaming\Emby-Server\system\EmbyServer.dll C:\Users\Home\AppData\Roaming\Emby-Server\system\EmbyServer.dll C:\Users\Home\AppData\Roaming\Emby-Server\system\EmbyServer.dll C:\Users\Home\AppData\Roaming\Emby-Server\system\EmbyServer.dll C:\Users\Home\AppData\Roaming\Emby-Server\system\EmbyServer.dll C:\Users\Home\AppData\Roaming\Emby-Server\system\EmbyServer.dll -noautorunwebapp
    Operating system: Microsoft Windows NT 6.2.9200.0
    64-Bit OS: True
    64-Bit Process: True
    User Interactive: True
    Processor count: 3
    Program data path: C:\Users\Home\AppData\Roaming\Emby-Server
    Application directory: C:\Users\Home\AppData\Roaming\Emby-Server\system
    System.ComponentModel.Win32Exception (0x80004005): The specified network name is no longer available
    System.ComponentModel.Win32Exception
    No Stack Trace Available

2018-03-28 10:44:41.770 Info LibraryMonitor: Stopping directory watching for path \\10.55.1.2\Media\Demos

.......

2018-03-28 12:48:28.779 Info App: Deleting item, Type: Folder, Name: MPEG, Path: \\10.55.1.2\Media\Demos\MPEG, Id: 393caf81-b8bb-7b8f-ddd8-956becde78af
2018-03-28 12:48:28.782 Debug App: Deleting path C:\Users\Home\AppData\Roaming\Emby-Server\metadata\library\39\393caf81b8bb7b8fddd8956becde78af
2018-03-28 12:48:28.782 Debug App: Deleting path C:\Users\Home\AppData\Roaming\Emby-Server\metadata\library\ad\ad661e54776345bacfabbc2023f5b852
2018-03-28 12:48:28.783 Debug App: Deleting path C:\Users\Home\AppData\Roaming\Emby-Server\metadata\library\73\7354c18885f4c43b06b55e02162dbf4b
2018-03-28 12:48:28.784 Debug App: Deleting path C:\Users\Home\AppData\Roaming\Emby-Server\metadata\library\c6\c6df814d3371000bf1db20a8fbad9c5c
2018-03-28 12:48:28.784 Debug App: Deleting path C:\Users\Home\AppData\Roaming\Emby-Server\metadata\library\93\93cdd69d9f9b44550f9452e791a89423
2018-03-28 12:48:28.785 Debug App: Deleting path C:\Users\Home\AppData\Roaming\Emby-Server\metadata\library\86\86b03a3785b2525b578f28faf9c8f68e
2018-03-28 12:48:28.785 Debug App: Deleting path C:\Users\Home\AppData\Roaming\Emby-Server\metadata\library\31\310ec5f330439982d71102d6bfdf1e58
2018-03-28 12:48:28.786 Debug App: Deleting path C:\Users\Home\AppData\Roaming\Emby-Server\metadata\library\54\54ca4690515be4b2dea41402925e044c
2018-03-28 12:48:28.842 Debug BaseItem: Removed item: \\10.55.1.2\Media\Demos\MPEG-Jelly
2018-03-28 12:48:28.842 Info App: Deleting item, Type: Folder, Name: MPEG-Jelly, Path: \\10.55.1.2\Media\Demos\MPEG-Jelly, Id: bc5965ac-96b7-8923-9c6a-0ecfb90d99dd
2018-03-28 12:48:28.843 Debug App: Deleting path C:\Users\Home\AppData\Roaming\Emby-Server\metadata\library\bc\bc5965ac96b789239c6a0ecfb90d99dd
2018-03-28 12:48:28.844 Debug App: Deleting path C:\Users\Home\AppData\Roaming\Emby-Server\metadata\library\87\87de93aa389d1dcd6d7b40089b5281de
2018-03-28 12:48:28.844 Debug App: Deleting path C:\Users\Home\AppData\Roaming\Emby-Server\metadata\library\0b\0b980c0959ddae7fab24d8755378ac15
2018-03-28 12:48:28.845 Debug App: Deleting path C:\Users\Home\AppData\Roaming\Emby-Server\metadata\library\0d\0da4cce8f332493e977392801b7e60a7
2018-03-28 12:48:28.847 Debug App: Deleting path C:\Users\Home\AppData\Roaming\Emby-Server\metadata\library\38\38b76ef41c4743026197b722e9c3fb94
2018-03-28 12:48:28.847 Debug App: Deleting path C:\Users\Home\AppData\Roaming\Emby-Server\metadata\library\32\3280c7d8c1357f37495d9c083e0a211d
2018-03-28 12:48:28.848 Debug App: Deleting path C:\Users\Home\AppData\Roaming\Emby-Server\metadata\library\3c\3cfaf8303d9e9a13ac99fba99c9f6ba6
2018-03-28 12:48:28.890 Debug BaseItem: Removed item: \\10.55.1.2\Media\Demos\MPEG-Camp
2018-03-28 12:48:28.890 Info App: Deleting item, Type: Folder, Name: MPEG-Camp, Path: \\10.55.1.2\Media\Demos\MPEG-Camp, Id: ea944465-658a-c9b8-7115-c3f01ce8aa73
2018-03-28 12:48:28.891 Debug App: Deleting path C:\Users\Home\AppData\Roaming\Emby-Server\metadata\library\ea\ea944465658ac9b87115c3f01ce8aa73

Edited March 28, 2018 by SamES

[Luke 42078]

Please try again with the .10 beta. Thanks.

[SamES 1057]

Please try again with the .10 beta. Thanks.

 

Thanks.  I'm on .10 now.  I restarted the server after the deletion and it updated the version at the same time.

 

On a positive note, after a 7 hour library scan it appears that all the resume, watched status and date added has been maintained.

 

Do you want me to simulate samba down again and see what happens?  Do you think this is resolved?

[dominicM 2]

Thanks.  I'm on .10 now.  I restarted the server after the deletion and it updated the version at the same time.

 

On a positive note, after a 7 hour library scan it appears that all the resume, watched status and date added has been maintained.

 

Do you want me to simulate samba down again and see what happens?  Do you think this is resolved?

 

 

I think you can just move a single folder somewhere else, re scan library, move it back and see if it's added with no metadata loss. For me watched state was maintained already on stable branch but date added was reset to the folder date modified.