Library Real Time Monitoring needs SAFETY CHECK

[MacGyver27 21]

Hello,

I have a scenario where one of my libraries sits on the external HDD enclosure. I want to have for it Real Time Monitoring enabled but there is a catch.. sometimes happens this external HDD enclosure has connectivity issues or after reboot it did not get properly mounted to the Linux file system and Emby starts purging everything related to it. This is huge pain in case such library is 50k home video files - everything needs to be rebuild from scratch, which can take weeks! I know it kind of makes sense of Real Time Monitoring feature but I would propose to implement a safety check that the library is actually present on the file system. I know this is difficult to detect properly when Emby server is inside docker, but something simple like: If any specified folder is entirely empty, skip the synchronization for it.

So before triggering the real time sync perform for both of this folders this empty folder check:

• IF contains SOMETHING -> sync
• IF contains NOTHING -> skip

I would do this safety check also when triggered manually, because many times user don't even notice the external HDD enclosure is not mounted.. 99% of cases user finds out when his media are missing in the Emby library.. In order to sync on these empty libraries user would need to disable safety check.

 

Solution:

Edited Wednesday at 10:44 AM by MacGyver27

[Luke 43010]

Hi there, please attach the Emby server log from when the problem occurred:

How to Report a Problem

Thanks!

 

[MacGyver27 21]

Hey.
Thanks fore the reply. Unfortunately this happened quite few weeks ago.. but can simulate if necessary.. should be pretty straight forward with docker. Let me know!

[RanmaCanada 583]

This is not an Emby problem, this is a problem with your hardware being faulty. 

Replace it.

[Lessaj 556]

If your storage is normally mounted to /srv/media then when it's not mounted /srv/media will exist as an empty folder, but /srv/media/YouTube would not. To my knowledge the server already has a check in place wherein if the mapped folder does not exist at all it does not try to scan and delete. You mentioned docker specifically, so I believe if you're running the container with mapped paths and those paths don't exist that it will still map them anyway (it will create the path on the host?), so the Emby server running inside the container would yes see the folders as being empty and then clear out.

It would be better to require your storage to be mounted before starting the docker container, or mapping a lower path instead so that your actual folders would not exist and you would not have that problem. So if you map /srv/media into the container, then only map the library paths as /srv/media/YouTube and /srv/media/.downloads/ADV, if /srv/media is not mounted those folders would not exist and it won't delete anything.

[MacGyver27 21]

20 hours ago, RanmaCanada said:

This is not an Emby problem, this is a problem with your hardware being faulty. 

Replace it.

My hardware is not faulty.. It is given when you are using an external HDD enclosure from time to time it wont mount after reboot - there are idle timeouts. When my server is down for longer than 10 minutes and I boot it, this HDD enclosure needs power-cycle, because it is in sleep mode or something..

[MacGyver27 21]

20 hours ago, Lessaj said:

If your storage is normally mounted to /srv/media then when it's not mounted /srv/media will exist as an empty folder, but /srv/media/YouTube would not. To my knowledge the server already has a check in place wherein if the mapped folder does not exist at all it does not try to scan and delete. You mentioned docker specifically, so I believe if you're running the container with mapped paths and those paths don't exist that it will still map them anyway (it will create the path on the host?), so the Emby server running inside the container would yes see the folders as being empty and then clear out.

Precisely! That's why I said it is tricky via Docker.. only fast solution would be this empty folder check.
 

 

20 hours ago, Lessaj said:

It would be better to require your storage to be mounted before starting the docker container, or mapping a lower path instead so that your actual folders would not exist and you would not have that problem. So if you map /srv/media into the container, then only map the library paths as /srv/media/YouTube and /srv/media/.downloads/ADV, if /srv/media is not mounted those folders would not exist and it won't delete anything.

Correct as well. I can make a health-check to prevent Emby from starting entirely when external mount is not present, but sometimes I maintain this external HDD enclosure and it is not in use and this would mean Emby would not start at all.. Usual scenario is when my RAID array needs a HDD replacement, for safety reasons I keep this HDD enclosure turned off and for this whole period of time - can be 30 days - I would not be able to run Emby server at all, until my RAID array is healthy again.

That is why I proposed this to be implemented into Emby server directly. So you mentioned you have a safety check to skip it in case folder is not preset at all - could you please extend this safety check of a scenario where the folder is also empty?

Thanks a lot!

Edited 5 hours ago by MacGyver27

[Lessaj 556]

This is essentially a feature request and should be in the correct forum, but if you mount the paths as suggested you won't need this at all.

[RanmaCanada 583]

1 hour ago, MacGyver27 said:

My hardware is not faulty.. It is given when you are using an external HDD enclosure from time to time it wont mount after reboot - there are idle timeouts. When my server is down for longer than 10 minutes and I boot it, this HDD enclosure needs power-cycle, because it is in sleep mode or something..

Your hardware is faulty. It is not a given that an external HDD enclosure won't mount from time to time after reboot. A proper working enclosure will always mount, ALWAYS. As for sleep time, you're just adding that now to defend yourself. Having it go to sleep when it's attached to the server just shows you have no clue what you're doing, which was apparent the moment you said you're using an external drive for your media..Have your system setup CORRECTLY.

I'd argue this is a 100% your problem, not an Emby problem.

[speechles 2109]

The issue is you are trying to blame Emby for something that Emby has no control over.

Why is your drive not staying mounted when it goes to sleep? It should still be mounted just goes to sleep. Mine stay this way on both Windows and Linux. The enclosures are USB and they sleep and stay mounted just fine. Why does it not always appear when you boot the PC? 

What type of drives are these? What is the enclosure? As stated by others this isn't an Emby problem. It is a hardware or OS issue. Fix your power plan that is turning off the hard drive after a period of time of no use. That is what this sounds like. You want the drive to always stay mounted and that power plan is causing it to unmount. It can still fall asleep and wake just fine.

A temporary solution is turn off real time monitoring. Use scan library instead. Then when adding new media navigate to the folder view of that type of library. In this case I've added new media to this particular folder inside my Movies library. Rather than scan the entire library I just want to update this folder. I can do that with scanning just the library in the folder. Did you know you could do this? If you add new media across your libraries just use the normal scan library to cover all. But when adding to just certain folders you can just scan those folders.

If you add a new episode to a series. You can also just click that series name on the 3 dots menu. Find the scan library and just scan in that series. Quickly. It works at any level and will just scan at that level.

Edited 3 hours ago by speechles

[C.S. 99]

1 hour ago, speechles said:

The issue is you are trying to blame Emby for something that Emby has no control over.

Don't hard drives die? Wouldn't that cause the exact issue OP is on about?

It's reasonable to expect Emby to deal gracefully with unfortunate situations that happen from time to time in the real world.

[speechles 2109]

9 minutes ago, C.S. said:

Don't hard drives die? Wouldn't that cause the exact issue OP is on about?

It's reasonable to expect Emby to deal gracefully with unfortunate situations that happen from time to time in the real world.

But if the hard drive is dead. The data is too. Unless you backed it up. Given the prices of drives today. Likely no backup and the data is gone. Most users data is just gone. Poof.

When people want to force an update of a library it is suggested to move that library out and scan. Then moving it back in and scan. This behavior is why that happens. It is expected when the drive isn't there that it isn't there for a reason and likely it will be shortly and can get scanned again.

If you want the OS to notify you the moment the drive becomes unmounted you can do that. But why would Emby need to? The issue has never been mentioned before. There is one person having the problem. This is cause and effect. The cause is unmounting the hard drive. The effect is it gets removed as a library during the scan. But figure out the cause and there is no effect. What is the cause?

Edited 1 hour ago by speechles

[C.S. 99]

15 minutes ago, speechles said:

But if the hard drive is dead. The data is too. Unless you backed it up. Given the prices of drives today. Likely no backup and the data is gone. Most users data is just gone. Poof.

My data is backed up. I'm not rich, just paranoid. As a paranoid person I would prefer if Emby took a beat before removing huge swaths of my library before I even have a chance to see that there's a problem. I would prefer Emby throw out an alert that says something like Hey a huge number of files are gone, do you want to continue scanning? Yes, the data is gone for a reason. The reason is an accident occurred. A very common accident. Help me out with a notification and give me a chance to fix things before moving ahead.

This is reasonable.

[Lessaj 556]

If you set up your library folders as I mentioned you wouldn't have that issue.

[C.S. 99]

Could you restate your advice for Windows users?

[Lessaj 556]

I haven't run on Emby on Windows in well over 10 years, I would need to test to provide any concrete advice, but if a drive isn't available then it wouldn't be there at all to scan, so that shouldn't be an issue. If, however, you had mapped say D:\ as a library folder, and you attach a new drive with that letter and the drive is blank, or contains other media, then sure it will scan it. If you had D:\Movies mapped instead, and now the newly mounted disk doesn't have that folder, it also shouldn't clear anything because that folder isn't there at all.