Non-destructive additive-only library scan

[Guest]

Right now the library scan is like a sync. A Library scan will add metadata to the library, but it also looks to remove entries that might no longer exist.

The problem is that emby is godawful at this, and will flagrantly delete gigabytes worth of metadata even though the library has not changed at all. SMB share too slow? Sorry, you lost all metadata for media on that share. Accidentally clicked "Scan metadata folder"? gone.

This behavior is awful and should not exist. It makes emby an ridiculously frustrating experience if you've got a large library with media on multiple targets.

Please, please, please, I beg you.. please just make a version of the library scan that ADDS data only and never deletes anything. There's never a time when I want metadata deleted, ever.

[ebr 16169]

2 hours ago, Klaitu said:

that ADDS data only and never deletes anything

Hi.  I do not believe that would be possible or practical.

Do you store your metadata in NFO files?  If so, this should make losing it pretty difficult.  You could also lock it to prevent changes.

Perhaps we should look at an example of whatever issue you had.

[Guest]

You don't think it would be possible to simply... not issue the commands to delete library information? Better yet, why bother calculating which files are missing to begin with? Just assume all previously existing files are present.

I would think this would be substantially similar to the current library scan and easy to implement since it's removing a function that already exists.

My emby is configured to store .nfo files, but it only does so sporadically because emby's SMB implementation has a bug in authentication that prevents it from getting write access to SMB shares most of the time (though it does work sporadically). However, the .nfo metadata isn't the main problem. The main problem is all the images. Rescanning the entire library might take 20 hours, but regenerating all the thumbnail images takes weeks. 

The root problem here is that emby misidentifies files that are missing and deletes 30+gb of metadata for tens of thousands of media folders. Maybe it's related to the SMB problem, maybe it isn't, nobody really knows for sure.. You can check my post history for some examples of my attempting to troubleshoot this. Unfortunately there seems to have been zero interest in addressing the issue. Twice.

I really like emby, it has a lot of things going for it, but if it can't even retain its own library, what use is it to me? I don't have time to constantly go and restore backups.

So, please, consider making an additive-only library scan, or make that an option in the library settings or something. I would much prefer to continue using emby instead of having to find an alternative that works.

 

[rechigo 364]

I run into this as well. Happened about a week ago actually.

You should look into doing some sort of backups to prevent having to rebuild everything from scratch. The emby backup & restore plugin does a great job and has saved me from this countless times. 

[Guest]

3 minutes ago, rechigo said:

I run into this as well. Happened about a week ago actually.

You should look into doing some sort of backups to prevent having to rebuild everything from scratch. The emby backup & restore plugin does a great job and has saved me from this countless times. 

I do use backup and restore, but it's got a problem.. consider this:

I have new media to add, so I initiate a library scan. Emby dumps 20,000 entries in the process.

So, I run the backup restore, which in itself may take 20 hours to restore..

Now I'm back in the position I was in before I wanted to add the new media, which means I have to do another library scan.. 

and during the library scan, emby dumps more media, and so the process begins again. The only thing that stops it is luck.

It sure would be nice if I had the option to run a scan that will only add, never delete. 

[ebr 16169]

18 hours ago, Klaitu said:

You don't think it would be possible to simply... not issue the commands to delete library information?

Sorry, no it would not be that simple.  We'd end up with things that can never update and a bunch of duplication.  Emby is designed to be self-maintaining - IOW adapting to the data that exists in the file system.  That is fundamental to everything it does.

[Guest]

2 minutes ago, ebr said:

Sorry, no it would not be that simple.  We'd end up with things that can never update and a bunch of duplication.  Emby is designed to be self-maintaining - IOW adapting to the data that exists in the file system.  That is fundamental to everything it does.

Then it seems we have a critical design flaw here, because it only works correctly about 30% of the time.

You're the expert, but I can tell you that emby is way way way too trigger happy about deleting things from its library. Maybe more redundant checks before deleting, or a "are you sure you want to remove 15,934 items from the library?" prompt. There must be something that can be done to mitigate this problem.

Might also want to look into why "scan metadata folder" will delete all images, or at least put a warning on it. "Refresh metadata" on the library screen will also delete all the associated images, and while it may pull new images, the video thumbnails will get nuked regardless of setting and will have to be regenerated.

In the meantime, I'm going to experiment with giving emby read-only access to existing content folders, thereby forcefully preventing such deletions.

[ebr 16169]

2 hours ago, Klaitu said:

because it only works correctly about 30% of the time

That would be an extremely rare experience.  Obviously, you can see how we wouldn't have survived to this point if that were true for any significant number of installations.  So, we should get to the bottom of your situation. Have you posted details of these issues?

[Neminem 1516]

Did you check if this is still happening or started happening again.

 

Edited January 22 by Neminem

[Guest]

7 minutes ago, Neminem said:

Did you check if this is still happening or started happening again.

 

This issue has never stopped happening, but is somewhat mitigated by my running scripts to keep the drives awake during the scan. There was an update later in the year where emby would remove and readd the same media in the same scan, but I don't know if that is related.

[Neminem 1516]

1 minute ago, Klaitu said:

but is somewhat mitigated by my running scripts to keep the drives awake during the scan.

Ok so you created a workaround, great, that did not help.

[Guest]

15 minutes ago, ebr said:

That would be an extremely rare experience.  Obviously, you can see how we wouldn't have survived to this point if that were true for any significant number of installations.  So, we should get to the bottom of your situation. Have you posted details of these issues?

I appreciate your perspective, but emby's library scan almost never functions correctly for me. I had no issues when my library was smaller. I am sure it would function better if the media and emby were on the same machine, but as it stands it's over 36,000 individual media files spread across 3 SMB shares.

Under these conditions, emby's library scan doesn't work as intended. You may want to review the forums here, because there are many other examples of users that have problems with emby dropping library data, it's just that a rescan for them isn't a big deal.

[Guest]

2 minutes ago, Neminem said:

Ok so you created a workaround, great, that did not help.

I wish I had more information for you. I don't know how to fix emby's library scan implementation, and apparently neither does anyone else, which is why I suggested having a scan that only adds media.. but it seems like an uphill battle. 

[Luke 42077]

We should continue the troubleshooting in the other topic to learn what's going on.