Library Corruption after Full Drive

[ShadowKindjal 19]

I recently had my boot SSD filled with a 200GB Unifi log and it caused transcoding errors because new files couldn't be generated. I resolved the drive space issue but for some reason all the files within my tv shows library disappeared and I have no idea why. Would anyone be able to help me figure out why and if there's a way to repair it without scanning my 55TB show library again? I also find it weird that it only affected my tv show library and not my anime, music, or movies library.

emby logs.7z

[Happy2Play 8304]

What exactly is happening with TV Shows?  There are a lot of Refreshes for /mnt/shows.

You appear to have permissions issues on your custom Metadata path.

Access to the path '/Emby/metadata/........................' is denied.

2022-06-07 13:29:40.329 Error ProviderManager: UnauthorizedAccessException: Emby Server is being write access to /Emby/metadata/library/f2/f27caa37e5142225cceded48f6553502/poster.png. Please check your file permissions.

Not positive on the logos issue but appears path related.

2022-06-07 13:46:58.339 Debug Skia: SKFileStream had a problem reading https://raw.githubusercontent.com/Theitfixer85/myepg/master/Logos/abc-us.png. Attempting workaround...
2022-06-07 13:46:58.341 Error Server: Error processing request
	*** Error Report ***
	Version: 4.7.2.0
	Command line: /system/EmbyServer.dll -programdata /config -ffdetect /bin/ffdetect -ffmpeg /bin/ffmpeg -ffprobe /bin/ffprobe -restartexitcode 3
	Operating system: Linux version 5.13.0-44-generic (buildd@lcy02-amd64-107) (gcc (Ubuntu 9.4.0-1ubuntu1~20.04.1) 9.4.0, GNU ld (GNU Binutils for Ubuntu) 2.34) #49~20.04.
	Framework: .NET 6.0.2
	OS/Process: x64/x64
	Runtime: system/System.Private.CoreLib.dll
	Processor count: 16
	Data path: /config
	Application path: /system
	System.IO.DirectoryNotFoundException: System.IO.DirectoryNotFoundException: Could not find a part of the path '/var/run/s6/services/emby-server/https:/raw.githubusercontent.com/Theitfixer85/myepg/master/Logos/abc-us.png'.
	   at Emby.Drawing.Skia.SkiaEncoder.GetSKFileStream(String path)
	   at Emby.Drawing.Skia.SkiaEncoder.EncodeImage(String inputPath, String outputPath, Boolean autoOrient, Nullable`1 orientation, Int32 quality, ImageProcessingOptions options, ImageFormat selectedOutputFormat)
	   at Emby.Drawing.ImageProcessor.ProcessImage(ImageProcessingOptions options, CancellationToken cancellationToken)
	   at Emby.Api.Images.ImageService.GetImage(ImageRequest request, Int64 itemId, BaseItem item, Boolean isHeadRequest)
	   at Emby.Server.Implementations.Services.ServiceController.GetTaskResult(Task task)
	   at Emby.Server.Implementations.Services.ServiceHandler.ProcessRequestAsync(HttpListenerHost appHost, IRequest httpReq, IResponse httpRes, RestPath restPath, String responseContentType, CancellationToken cancellationToken)
	   at Emby.Server.Implementations.HttpServer.HttpListenerHost.RequestHandler(IRequest httpReq, ReadOnlyMemory`1 urlString, ReadOnlyMemory`1 localPath, CancellationToken cancellationToken)
	Source: Emby.Drawing.Skia
	TargetSite: SkiaSharp.SKFileStream GetSKFileStream(System.String)
	Path: https://raw.githubusercontent.com/Theitfixer85/myepg/master/Logos/abc-us.png

Your SSL certificate is failing

2022-06-07 13:27:32.788 Error App: Error loading cert from /LE-SSL/privkey.pfx
	*** Error Report ***
	Version: 4.7.2.0
	Command line: /system/EmbyServer.dll -programdata /config -ffdetect /bin/ffdetect -ffmpeg /bin/ffmpeg -ffprobe /bin/ffprobe -restartexitcode 3
	Operating system: Linux version 5.13.0-44-generic (buildd@lcy02-amd64-107) (gcc (Ubuntu 9.4.0-1ubuntu1~20.04.1) 9.4.0, GNU ld (GNU Binutils for Ubuntu) 2.34) #49~20.04.
	Framework: .NET 6.0.2
	OS/Process: x64/x64
	Runtime: system/System.Private.CoreLib.dll
	Processor count: 16
	Data path: /config
	Application path: /system
	Interop+Crypto+OpenSslCryptographicException: Interop+Crypto+OpenSslCryptographicException: error:2006D002:BIO routines:BIO_new_file:system lib
	   at Interop.Crypto.CheckValidOpenSslHandle(SafeHandle handle)
	   at Internal.Cryptography.Pal.OpenSslX509CertificateReader.FromFile(String fileName, SafePasswordHandle password, X509KeyStorageFlags keyStorageFlags)
	   at System.Security.Cryptography.X509Certificates.X509Certificate..ctor(String fileName, String password, X509KeyStorageFlags keyStorageFlags)
	   at System.Security.Cryptography.X509Certificates.X509Certificate2..ctor(String fileName, String password)
	   at Emby.Server.Implementations.ApplicationHost.GetCertificate(CertificateInfo info)
	Source: System.Security.Cryptography.X509Certificates
	TargetSite: Void CheckValidOpenSslHandle(System.Runtime.InteropServices.SafeHandle)

Looks like a plugin issue also, look to be a version behind but don't know if it is still functional per its topic.

2022-06-07 13:53:13.809 Error Emby.Notifications.Discord: Something unexpected happened in the item update checker

You also have issues with these plugins also

2022-06-07 13:27:32.542 Error App: LoaderException: Method 'GetChannelItemMediaInfo' in type 'MediaBrowser.Plugins.TuneIn.TuneInChannel' from assembly 'MediaBrowser.Plugins.TuneIn, Version=1.0.6098.30150, Culture=neutral, PublicKeyToken=null' does not have an implementation.
2022-06-07 13:27:32.542 Error App: LoaderException: Could not load type 'MediaBrowser.Controller.Channels.ChannelMediaInfo' from assembly 'MediaBrowser.Controller, Version=4.7.2.0, Culture=neutral, PublicKeyToken=null'.

 

[ShadowKindjal 19]

First of all, thank you for taking the time to help me.

I will check the permissions issues on the custom metadata location. What had happened is that I woke up one morning and none of the tv shows seemed to have playable content. It's as if all of the tv show files had disappeared. The multiple refreshes was me frantically trying to  solve the problem and hoping a refresh would remap all of my data without having to manually scan the entire library again.

In regards to the SSL certs... I honestly need to just remove that. I have everything handled by a reverse proxy now so it's not even being used.

In regards to the notifications app, now that you mention it I have noticed that my discord server is not seeing notifications from emby anymore and it's worth looking into.

As for the last two plugins you mentioned. I actually don't recognize the name of those. I'm guessing I would have to check the plugins directory to see if any dll files may be generating that error?

[Happy2Play 8304]

7 minutes ago, ShadowKindjal said:

I'm guessing I would have to check the plugins directory to see if any dll files may be generating that error?

I know the first one is the TuneIn plugin but not positive on the second one.

[ShadowKindjal 19]

Small progress update:

I deleted the reference to the old metadata folder and I'm allowing that information to be saved in the default location. It was linked to an old folder structure that is no longer relevant but it could have prompted the issue in the first place. Changing this, however, didn't fix the issues with the tv shows. They seem to be slowing re-populating from the scan library scheduled task. I just don't know what happened to the previous data that would cause all the shows to not appear within the server. I assume I just have to wait for now. Another thing I've noticed, however, is that a lot of the actors and actresses seem to have missing art as seen below. Is there an option to refresh this data?

I deleted the SSL reference on the networking page. Hopefully that'll take care of this issue. Like I said, everything is handled by my nginx reverse proxy so this should resolve the errors in my log files.

I checked my discord notifications app and I was mistaken. It seems to be functioning fine. I took a second look at the error and it seems to be in relation to updates and when I checked the emby catalog it looked like the app is no longer available. Could that be why I'm getting this error message?

Lastly, I did find the tune in plugin dll in my plugins directory and deleted it but I Can't find this anything similarly named to this channels plugin. I've provided an ls of my current plugins directory below.

 

 

[Happy2Play 8304]

19 minutes ago, ShadowKindjal said:

Is there an option to refresh this data?

You would need to do a Library level Refresh Metadata as that image shows broken links to your metadata/people folder, assume your log is flooded with can not find paths as all the paths are recorded in the database so /metadata relocations can become problematic.  You can try the "Scan Metadata Folder" task if you have relocate these items.

 

19 minutes ago, ShadowKindjal said:

Could that be why I'm getting this error message?

The topics suggest the dev stopped updating the plugin and hasn't been online in awhile, but you should look throw the plugin topic for options.

19 minutes ago, ShadowKindjal said:

Lastly, I did find the tune in plugin dll in my plugins directory and deleted it but I Can't find this anything similarly named to this channels plugin. I've provided an ls of my current plugins directory below.

Would need to see another server log after restart to see if anything else stands out.

Edited June 8, 2022 by Happy2Play

[ShadowKindjal 19]

I woke up this morning and it seems that my TV Show directory has been corrupted again. Shows were slowly scanning back in when I went to bed last night but when I woke up this morning all of the episodes were gone in addition to the custom image I had uploaded for the library as the Primary. I'm working on getting the server logs.

 

 

Edited June 8, 2022 by ShadowKindjal

[ShadowKindjal 19]

I've attached the logs from my server.

emby_logs.zip

[Luke 37119]

Yea there's something going on with your database file, notice the database locked errors in your server log. Rebooting your machine is a good thing to try that will often clear up that error.

Before doing that, I would suggest removing these plugins:

	Anime 1.4.8.0
	Device List Cleaner 1.0.0.0
	DiscordNotifications 1.0.7.9
	Disk Space 1.0.6.0
	Email Notifications 3.1.6.0
	Intro Skip 1.0.0.0
	Open Subtitles 1.0.31.0
	Playback Reporting 1.9.0.5
	Statistics 2.1.3.0

 

[ShadowKindjal 19]

Thank you for the response Luke. I'll attempt a reboot to fix the issue. Before doing that could you explain why you would suggest the removal of these plugins? I do use quite a few of them on a daily basis. 

[Luke 37119]

Anytime there's a significant unknown problem such as this, that should be one of the first things that you do. From our perspective, if we didn't write the code for them, then it's not realistic for us to troubleshoot them. So first we have to rule out problems potentially caused by plugins, and then you can add them back. In this case, I don't expect them to be the problem, but we still have to go through the process to get to the bottom of it.

[ShadowKindjal 19]

It looks like things are fixed for the most part but there are some shows that are missing from Emby but are available in the directory with full permissions.

embyserver.zip

[Luke 37119]

Can you try refreshing the metadata on that series?

[ShadowKindjal 19]

Prior to performing a refresh ten episodes appeared on the show but nothing else. I performed a metadata refresh and scanned library files for this show and this was the end result.

embyserver (1).zip

[ShadowKindjal 19]

Deleting and restoring the media files from a backup seems to have fixed the issue. Could this mean that the media files themselves were corrupted and emby refused to import them?

[rotational467 31]

@ShadowKindjal in the past I've seen strange and difficult-to-pinpoint corruption issues in SQL-type DBs when the drive where the DB files are stored runs out of space while the DB is otherwise open/active.  I'd consider the current database to be potentially suspect - if it were my system, I would reset Emby entirely and rebuild to be safe.  Chasing around DB gremlins is no fun.