Completed Do not clean database, if folder containing the library isn't available

[fbartels 9]

Hi,

 

little backstory: Over the weekend I upgraded the server Emby is running from Ubuntu 14.04 to 16.04. Unfortunately after the upgrade Emby mixed up the configured users so I ended up without a user with admin permissions in Emby and therefore decided to delete /var/lib/emby-server and start fresh. Setup wise I have one system (Synology NAS) providing the files and another system actually running Emby.

 

It happens every now and then (every day if you don't remove the "clean database" from sheduled tasks, that Emby decided to clean out my library, when he can't find the media files (because the NAS is turned off). While it makes sense to remove an item from the database, that cannot be found anymore Emby should become suspicious of every item in the library vanished.

 

Feature suggestion: if a mere ls on the folder of the library already does not give any results, skip clean for the library.

[tocks 4]

+1

[CharleyVarrick 277]

Something similar happened to me a while back and I wished a failsafe feature would have kicked in

[mastrmind11 717]

https://emby.media/community/index.php?/topic/39767-library-disappeared-due-to-failed-mount/

 

https://emby.media/community/index.php?/topic/42630-server-ping-nas-offset/

[CBers 6771]

How is Emby expected to keep things "online" if you go and turn the storage off

 

If you don't turn the NAS off, then Emby won't "delete" your content.

 

If you want to continue turning it off, then close Emby before you power off, then restart Emby after powering the NAS back on.

[CharleyVarrick 277]

The source could become offline or otherwise unreachable without the user's knowledge. A simple "are you sure you want to delete a gazillion media items" could avoid lost hours (if not days) of rebuilding because a kid unplugged a wire somewhere in the house.

Edited January 9, 2017 by jlr19

[Latchmor 584]

It's not the clean database that's doing it (well, not only) the Library Scan will remove items if media is offline. I've had to disable all Library Scan schedules. I manually scan when I know my NAS is online.

 

For my situation the NAS goes off nearly every night and back on when I return from work. I'm managing it fine now by only scanning when online so it's no big deal to me. However, I can see the appeal of a process where Emby goes "Woah, I'm going to remove an entire library here, maybe I'll stop..." but I've no idea how easily you would achieve that.

 

 

Edit: I should add I'm pretty sure this situation was handled better before the 'separate settings per library' were introduced. I used to have scheduled scans and scan on application startup on. When Emby auto updated through the day and scanned. My library was still there after a quick scan after putting my NAS online.

Edited January 9, 2017 by Latchmor

[CharleyVarrick 277]

The 2 links from mastrmind11 (post# 4) are both MUST read on this subject

[fbartels 9]

It's not the clean database that's doing it (well, not only) the Library Scan will remove items if media is offline. I've had to disable all Library Scan schedules. I manually scan when I know my NAS is online.

Ah, yes thats the beauty of having to redo all sheduled tasks, "scan library" is now also set to the usual on time of my nas.

[Luke 37068]

Thanks, this is being reviewed for the next release of Emby Server.

[fbartels 9]

Thanks you Luke

[mastrmind11 717]

Thanks Luke.

[guimello 1]

has a failsafe been added for this yet?

[darkassassin07 423]

If you guys are still looking for a good way to implement this, what about a toggle in each libraries configuration 'allow item removal during scheduled scans'

When enabled, scans work as they do now, but disabled they can only add new media/metadata to the library. This would not effect scans manually triggered. (odds are if you are scanning manually, you know your media is available to scan)

 

It would give people another option besides trying to schedule your scans during known downtime. That may not be possible with storage that sleeps unless being accessed, or offline sporadically.

 

Im not sure how others have their libraries setup, but media is rarely removed from my libraries (at least movies/shows) and when it is, its done manually where i rescan manually after anyway.

 

 

 

 

 

The other side of this is how to handle media that is apart of the library but not available.

Two options I can think of are either not displaying them at all, though that would require frequent monitoring of all media items to see if they are available or not; or leaving them visible untill a user attempts to stream unavailable content then notify them 'this content is currently unavailable'

Edited October 8, 2017 by darkassassin07

[Luke 37068]

has a failsafe been added for this yet?

 

Yes, are you experiencing a problem? If so, can you please describe it? Thanks !

[m33e 2]

Yes, are you experiencing a problem? If so, can you please describe it? Thanks !

 

I'm experiencing this issue as well. I have some older media on a network share of a pc that I do not use frequently at all. So I only boot it up on demand, and then it would be nice if Emby didn't have to scan this share again and again, after the media becomes unavailable & deleted over and over again.

 

What kind of failsafe is there? Maybe my setup doesnt't trigger the implementation? I actually only have one library called "Movies" added in Emby. In this library, I added multiple folders, all from different network shares of different machines.

[otispresley 81]

I have had this happen before. Mine runs on Ubuntu 16.04.3 inside a Hyper-V Virtual Machine. My libraries are via CIFS mounts to storage on the host machine with an Internal vSwitch connecting them. For a while, there was a CIFS problem in Ubuntu where sometimes mounts would experience I/O issues and unmount. If I didn't catch it quickly, or I was away from home, then Emby would delete all my metadata, resulting in loss of watched status, recently added, etc. I would say that the Library Scan should trigger a check first to see if the root folder of the Library path is available. If not, then do not scan that library and log that the path was unavailable.

[Luke 37068]

Can you please discuss an example and attach the information requested in how to report a problem? thanks.

[ebr 14913]

It is really best not to trouble-shoot inside of a completed feature request topic (we might lose track of it) so, if you don't mind, it would be great if you open a new topic in the server forum with the information requested above.

 

Thanks!

[otispresley 81]

Can you please discuss an example and attach the information requested in how to report a problem? thanks.

 

@@Luke,

 

I won't have logs or anything at this point. The CIFS issue seems to have been resolved in Ubuntu some time ago. Since then, I have started over so I could use the .Net Core version of Emby. I only wanted to say that I have experienced a loss of metadata before so you know this problem is not isolated to a few users or use cases.

 

A little more detail about my Emby library paths though is that I have my single CIFS mount for Emby in /etc/fstab so it is mounted at boot and before Emby loads. There are 2 other mounts as well, one for Nextcloud users and the other for access to another share. The mount used for the Emby library is mounted RW as the emby user (gid, uid) as follows:

//<IP_Address>/Media     /media/Media    cifs    credentials=/home/<USER>/.smbpass,uid=999,gid=999,sec=ntlm,vers=3.0    0       0

The media is actually stored on the Windows Server 2016 host that the Hyper-V VM running Emby is hosted on, so I have an Internal vSwitch for communication between the VM's and the host only. The drive hosting the media is a drive pool made up of all internal 3.5" disks using Stablebit DrivePool software. I do not have logs, but please let me know if you want further details about my setup and/or configuration.

Edited February 11, 2018 by otispresley

[Luke 37068]

Ok thanks for that info. It would be best to try and capture it next time it happens so that we can look at the specifics. Thanks.

[BiTeTHeBiGG 13]

Hi @Luke, I can provide logs when the software deleted the entries on my side.

I'm experiencing network loss as of a faulty network wall plug and one of my  server mess up my whole database.

The second server (testing purpose) did not scan at that time and is healthy.

Not only all Items were removed but on readding something is broken as only 60% were newly added back and on specific one (more for sure there) only added back 28 episodes out of 368.

I will attach later the log when the software is having error and deleting it. Let me just obfuscate  a little. Not all should know my IPs and paths .

Best regards

BiTeTHeBiGG

EDIT: Uploaded obfuscated log parts

Debug-obfuscated.txt

and as you can see all libraries are messed up. Here just a small screenshot of the anime library. I have to manually identify all the grayed items as they were not added correct back. this is a big bit of frustration.

Edited June 26, 2023 by BiTeTHeBiGG
added screenshot

[Luke 37068]

On 6/26/2023 at 11:31 AM, BiTeTHeBiGG said:

Hi @Luke, I can provide logs when the software deleted the entries on my side.

I'm experiencing network loss as of a faulty network wall plug and one of my  server mess up my whole database.

The second server (testing purpose) did not scan at that time and is healthy.

Not only all Items were removed but on readding something is broken as only 60% were newly added back and on specific one (more for sure there) only added back 28 episodes out of 368.

I will attach later the log when the software is having error and deleting it. Let me just obfuscate  a little. Not all should know my IPs and paths .

Best regards

BiTeTHeBiGG

EDIT: Uploaded obfuscated log parts

Debug-obfuscated.txt 121.16 kB · 0 downloads

and as you can see all libraries are messed up. Here just a small screenshot of the anime library. I have to manually identify all the grayed items as they were not added correct back. this is a big bit of frustration.

OK, regarding content getting cleared out of the database, I think this should be improved in the upcoming 4.8 server release.