Library Real Time Monitoring needs SAFETY CHECK

[MacGyver27 21]

Hello,

I have a scenario where one of my libraries sits on the external HDD enclosure. I want to have for it Real Time Monitoring enabled but there is a catch.. sometimes happens this external HDD enclosure has connectivity issues or after reboot it did not get properly mounted to the Linux file system and Emby starts purging everything related to it. This is huge pain in case such library is 50k home video files - everything needs to be rebuild from scratch, which can take weeks! I know it kind of makes sense of Real Time Monitoring feature but I would propose to implement a safety check that the library is actually present on the file system. I know this is difficult to detect properly when Emby server is inside docker, but something simple like: If any specified folder is entirely empty, skip the synchronization for it.

So before triggering the real time sync perform for both of this folders this empty folder check:

• IF contains SOMETHING -> sync
• IF contains NOTHING -> skip

I would do this safety check also when triggered manually, because many times user don't even notice the external HDD enclosure is not mounted.. 99% of cases user finds out when his media are missing in the Emby library.. In order to sync on these empty libraries user would need to disable safety check.

 

Solution:

Edited Wednesday at 10:44 AM by MacGyver27

[Luke 43029]

Hi there, please attach the Emby server log from when the problem occurred:

How to Report a Problem

Thanks!

 

[MacGyver27 21]

Hey.
Thanks fore the reply. Unfortunately this happened quite few weeks ago.. but can simulate if necessary.. should be pretty straight forward with docker. Let me know!

[RanmaCanada 584]

This is not an Emby problem, this is a problem with your hardware being faulty. 

Replace it.

[Lessaj 559]

If your storage is normally mounted to /srv/media then when it's not mounted /srv/media will exist as an empty folder, but /srv/media/YouTube would not. To my knowledge the server already has a check in place wherein if the mapped folder does not exist at all it does not try to scan and delete. You mentioned docker specifically, so I believe if you're running the container with mapped paths and those paths don't exist that it will still map them anyway (it will create the path on the host?), so the Emby server running inside the container would yes see the folders as being empty and then clear out.

It would be better to require your storage to be mounted before starting the docker container, or mapping a lower path instead so that your actual folders would not exist and you would not have that problem. So if you map /srv/media into the container, then only map the library paths as /srv/media/YouTube and /srv/media/.downloads/ADV, if /srv/media is not mounted those folders would not exist and it won't delete anything.

[MacGyver27 21]

20 hours ago, RanmaCanada said:

This is not an Emby problem, this is a problem with your hardware being faulty. 

Replace it.

My hardware is not faulty.. It is given when you are using an external HDD enclosure from time to time it wont mount after reboot - there are idle timeouts. When my server is down for longer than 10 minutes and I boot it, this HDD enclosure needs power-cycle, because it is in sleep mode or something..

[MacGyver27 21]

20 hours ago, Lessaj said:

If your storage is normally mounted to /srv/media then when it's not mounted /srv/media will exist as an empty folder, but /srv/media/YouTube would not. To my knowledge the server already has a check in place wherein if the mapped folder does not exist at all it does not try to scan and delete. You mentioned docker specifically, so I believe if you're running the container with mapped paths and those paths don't exist that it will still map them anyway (it will create the path on the host?), so the Emby server running inside the container would yes see the folders as being empty and then clear out.

Precisely! That's why I said it is tricky via Docker.. only fast solution would be this empty folder check.
 

 

20 hours ago, Lessaj said:

It would be better to require your storage to be mounted before starting the docker container, or mapping a lower path instead so that your actual folders would not exist and you would not have that problem. So if you map /srv/media into the container, then only map the library paths as /srv/media/YouTube and /srv/media/.downloads/ADV, if /srv/media is not mounted those folders would not exist and it won't delete anything.

Correct as well. I can make a health-check to prevent Emby from starting entirely when external mount is not present, but sometimes I maintain this external HDD enclosure and it is not in use and this would mean Emby would not start at all.. Usual scenario is when my RAID array needs a HDD replacement, for safety reasons I keep this HDD enclosure turned off and for this whole period of time - can be 30 days - I would not be able to run Emby server at all, until my RAID array is healthy again.

That is why I proposed this to be implemented into Emby server directly. So you mentioned you have a safety check to skip it in case folder is not preset at all - could you please extend this safety check of a scenario where the folder is also empty?

Thanks a lot!

Edited Friday at 01:04 PM by MacGyver27

[Lessaj 559]

This is essentially a feature request and should be in the correct forum, but if you mount the paths as suggested you won't need this at all.

[RanmaCanada 584]

1 hour ago, MacGyver27 said:

My hardware is not faulty.. It is given when you are using an external HDD enclosure from time to time it wont mount after reboot - there are idle timeouts. When my server is down for longer than 10 minutes and I boot it, this HDD enclosure needs power-cycle, because it is in sleep mode or something..

Your hardware is faulty. It is not a given that an external HDD enclosure won't mount from time to time after reboot. A proper working enclosure will always mount, ALWAYS. As for sleep time, you're just adding that now to defend yourself. Having it go to sleep when it's attached to the server just shows you have no clue what you're doing, which was apparent the moment you said you're using an external drive for your media..Have your system setup CORRECTLY.

I'd argue this is a 100% your problem, not an Emby problem.

[speechles 2109]

The issue is you are trying to blame Emby for something that Emby has no control over.

Why is your drive not staying mounted when it goes to sleep? It should still be mounted just goes to sleep. Mine stay this way on both Windows and Linux. The enclosures are USB and they sleep and stay mounted just fine. Why does it not always appear when you boot the PC? 

What type of drives are these? What is the enclosure? As stated by others this isn't an Emby problem. It is a hardware or OS issue. Fix your power plan that is turning off the hard drive after a period of time of no use. That is what this sounds like. You want the drive to always stay mounted and that power plan is causing it to unmount. It can still fall asleep and wake just fine.

A temporary solution is turn off real time monitoring. Use scan library instead. Then when adding new media navigate to the folder view of that type of library. In this case I've added new media to this particular folder inside my Movies library. Rather than scan the entire library I just want to update this folder. I can do that with scanning just the library in the folder. Did you know you could do this? If you add new media across your libraries just use the normal scan library to cover all. But when adding to just certain folders you can just scan those folders.

If you add a new episode to a series. You can also just click that series name on the 3 dots menu. Find the scan library and just scan in that series. Quickly. It works at any level and will just scan at that level.

Edited Friday at 03:24 PM by speechles

[C.S. 100]

1 hour ago, speechles said:

The issue is you are trying to blame Emby for something that Emby has no control over.

Don't hard drives die? Wouldn't that cause the exact issue OP is on about?

It's reasonable to expect Emby to deal gracefully with unfortunate situations that happen from time to time in the real world.

[speechles 2109]

9 minutes ago, C.S. said:

Don't hard drives die? Wouldn't that cause the exact issue OP is on about?

It's reasonable to expect Emby to deal gracefully with unfortunate situations that happen from time to time in the real world.

But if the hard drive is dead. The data is too. Unless you backed it up. Given the prices of drives today. Likely no backup and the data is gone. Most users data is just gone. Poof.

When people want to force an update of a library it is suggested to move that library out and scan. Then moving it back in and scan. This behavior is why that happens. It is expected when the drive isn't there that it isn't there for a reason and likely it will be shortly and can get scanned again.

If you want the OS to notify you the moment the drive becomes unmounted you can do that. But why would Emby need to? The issue has never been mentioned before. There is one person having the problem. This is cause and effect. The cause is unmounting the hard drive. The effect is it gets removed as a library during the scan. But figure out the cause and there is no effect. What is the cause?

Edited Friday at 05:14 PM by speechles

[C.S. 100]

15 minutes ago, speechles said:

But if the hard drive is dead. The data is too. Unless you backed it up. Given the prices of drives today. Likely no backup and the data is gone. Most users data is just gone. Poof.

My data is backed up. I'm not rich, just paranoid. As a paranoid person I would prefer if Emby took a beat before removing huge swaths of my library before I even have a chance to see that there's a problem. I would prefer Emby throw out an alert that says something like Hey a huge number of files are gone, do you want to continue scanning? Yes, the data is gone for a reason. The reason is an accident occurred. A very common accident. Help me out with a notification and give me a chance to fix things before moving ahead.

This is reasonable.

[Lessaj 559]

If you set up your library folders as I mentioned you wouldn't have that issue.

[C.S. 100]

Could you restate your advice for Windows users?

[Lessaj 559]

I haven't run on Emby on Windows in well over 10 years, I would need to test to provide any concrete advice, but if a drive isn't available then it wouldn't be there at all to scan, so that shouldn't be an issue. If, however, you had mapped say D:\ as a library folder, and you attach a new drive with that letter and the drive is blank, or contains other media, then sure it will scan it. If you had D:\Movies mapped instead, and now the newly mounted disk doesn't have that folder, it also shouldn't clear anything because that folder isn't there at all.

[C.S. 100]

Oh I see where the disconnect is, no pun intended. OP had a whole library on a single disk. I'm looking at this from a drive pooling perspective, where Emby has no idea what drives are where, it just knows a crapload of files are suddenly missing.

[js28194 70]

OS Power Settings would step 1.  I think some external usb drive have external tools also such as seagate that allow modifications.

 

 

 

[js28194 70]

I've had drive's on for many many hours w/o seeing this issue.  I will say that there have been times when I have rebooted and drive did not come online automatically. I had unplug and plug either the power cord or USB for that drive but did not see EMBY directly wipe any data.  YMMV of course.

Edited 1 hour ago by js28194

[js28194 70]

A few more comments to add... I know for a fact that my BIOS has some power saving settings at the USB level also, but I'm not going to reboot and screenshot just for you.

To clarify my comment on "Emby wiping Data"  What I meant was that in some random occurrences when the drive wasn't online, what I saw in the EMBY web interface is that the title was still there, but a big X on the image, clear indication to me that a drive as off line.

In Windows you can create a powershell script that runs in task scheduler every 30 - 60 minutes that checks windows event log for certain event ID's.  I'm not going to pollute this thread with all of that.  Windows removed the send email alert directly in event viewer many moons ago.

Finally, and this is pure opinion and preference.  At the home level, I'm not into drive pooling nor RAID as aside from some redundancy, provide no real performance benefit on my Gig LAN.  For remote viewing everything has to be transcoded down to 4mbps anyway (directly on NVIDIA Silicon Yay!) even though I have 1Gig Fiber.  Why you may ask (or not) well it adds another layer of complexity to deal with.  If my media files are scattered as you can see above on multiple drives, the chance of recovering data from 1 drive is much easier and simplier than data pooling / raid recovery.  In those cases, your ENTIRE server is down until you fix vs only 1 drive which can be re scanned when fixed or replaced.  I am not in a mission critical high performance requirement environment.  I have grown up in Dell PowerVault Arrays and been knee deep in RAID 0/1/5/10 environments so I know what I'm doing but my home is NOT a high performance environment.

I am also very slowing shucking and migrating these drives to a single non-raid enclosure (https://www.amazon.com/dp/B07MD2LNYX) as I understand the desire to save physical space.  Will end up removing the two PCIe cards providing 14 USB 3.0 ports.  Drives were cheaper back in the day so I guess I lucked out.  24TB seagate for $180 bucks I think it was.

Also, you can run a script every once in a while to determine what's on the drive.  Example  Dir/w F:\Movies > C:\temp\fmovies.txt  You can also schedule this script in task scheduler.

Isn't drive pooling just another fancy term for JBOD?  I hope those that are doing this have hardware level RAID.  Almost, very almost feels like a fancy marketing term for software RAID, but really is file duplication I suppose. I've doing some reading on Stablebit but not convinced.  Seems like it takes up twice the space unless webster changed the definition of "file duplication".  That is a guess in a way redundancy but anything that is important to me (home videos and photos) are already burnt on multiple 50GB DVD-R discs.  Media files unimportant.

Finally, I'm going to agree with most others here that is a your problem / hardware / OS, not an Emby problem.  I can't call Microsoft and bitch because my mapped drive disconnected and when I try to save a file it defaults to my local my documents directory now and not my server.  All these years later and Persistent drive mapping is still a random issue.  Sigh.  My Net USE commands are just batch files now   A double click away.

Ok done ranting for now.  Live long and prosper and don't take anything I say personally please.

Edited 14 minutes ago by js28194