Scan media library failed and how to restore the library

[NikaNika 0]

Hello everyone!
So, here's my issue. I have Emby media library with movies and TV series, which are all located on a 4Tb HDD and the HDD is connected to my Android TV Box via USB. So, I'm using the Android app of Emby server, installed on my TV Box.
I really love using Emby, it's so great in many aspects, but there's one issue that really irritates me and takes a lot of my time.
At the beginning everything worked fine, but once when I added new movies and tv series, library scan failed (when I clicked "scan library files"), so they weren't added no matter how many times I tried. Restarting the server and also restarting the TV Box didn't help. I found someone with similar problem on this forum, and read the advice that I should rename or delete the library.db file and rescan the library. I did so and it helped. But! I lost all of my manually added metadata, so I had to edit it again from the very beginning. And it was really tiresome, because (not all, but a lot of) the movies and series were incorrectly recognized automatically, and a lot of them are quite rare, so don't have descriptions and posters in my native language and I had to search for them in the internet, even translate some descriptions and manually add to Emby via "edit metadata". And I really can't do all this every time, because it really takes like the whole day for me to do it + a lot of effort.
I thought that the scan failure happened because I tried to scan the library while the upload of new media files to the HDD was in progress. I thought that this was my mistake, I just wouldn't do this anymore and everything would be fine.
Everything really worked fine for a week or so, but today library scan failed once again (right before that I just deleted one movie I already watched and renamed one of the folders with TV series via different (not Emby) file manager). I'm not really sure, maybe it also was my mistake and I shouldn't have done this?
Then I did everything posted here [https://support.emby.media/support/solutions/articles/44002210894-corrupt-database]: deleted library.db-shm & library.db-wal, checked my library.db file (Integrity Check) and it returned an integrity_check back of "OK" with no errors reported. So I had no choice but to delete the old library.db file and to start anew once again. It works, but once again I lost all of my manually added metadata. I saved the copy of the old library.db file to my laptop just in case, maybe it'll be possible to restore my old library from it. I don't know...
But I think I'm ready to sacrifice one more day to edit all the metadata manually for the 3rd time if there's guarantee that it won't disappear again (someone pls tell me what I should do to avoid this). Btw today I installed the Server Configuration Backup plugin, here I read that it would help. Btw it failed to create a backup file with my old library (when it was already impossible to scan it).
But I think now when my library started its existence from the very beginning once again and it's working and scanning fine, I think it will create the backup. But will it help to restore all of my manually added metadata, if the scan fails once again?
Someone help me please, and tell me what I should do, I'm really tired of these scan issues and especially of adding metadata manually over and over again for lots of my movies and series.

Overview (scan media library failed)

Spoiler

Exception of type 'SQLitePCL.pretty.SQLiteException' was thrown.
at SQLitePCL.pretty.SQLiteException.CheckOk (SQLitePCLEx.sqlite3 db, System.Int32 rc) [0x00017] in <1319f47a49f34f6db3822eb2e43ff130>:0
at SQLitePCL.pretty.SQLiteException.CheckOk (SQLitePCLEx.sqlite3_stmt stmt, System.Int32 rc) [0x00006] in <1319f47a49f34f6db3822eb2e43ff130>:0
at SQLitePCL.pretty.StatementImpl.MoveNext () [0x0004a] in <1319f47a49f34f6db3822eb2e43ff130>:0
at Emby.Server.Implementations.Data.SqliteItemRepository.ExecuteWithStringFormatParam (SQLitePCL.pretty.IDatabaseConnection db, System.String query, System.String valueText, System.Int64[] values) [0x00034] in <05979d9270ab4dafac52ba5587e41c17>:0
at Emby.Server.Implementations.Data.SqliteItemRepository.DeleteItemsInTransaction (SQLitePCL.pretty.IDatabaseConnection db, System.Int64[] ids, System.Boolean supportsPeople, System.Boolean supportsChapters, System.Boolean supportsCollections, System.Boolean supportsMediaStreams, System.Boolean supportsListItems, System.Boolean supportsExternalDatabases) [0x00114] in <05979d9270ab4dafac52ba5587e41c17>:0
at Emby.Server.Implementations.Data.SqliteItemRepository.DeleteItemsInChunks (System.Int64[] ids, SQLitePCL.pretty.IDatabaseConnection db, System.Boolean supportsPeople, System.Boolean supportsChapters, System.Boolean supportsCollections, System.Boolean supportsMediaStreams, System.Boolean supportsListItems, System.Boolean supportsExternalDatabases) [0x00027] in <05979d9270ab4dafac52ba5587e41c17>:0
at Emby.Server.Implementations.Data.SqliteItemRepository.DeleteItems (System.Int64[] ids) [0x00048] in <05979d9270ab4dafac52ba5587e41c17>:0
at Emby.Server.Implementations.Data.SqliteItemRepository.DeleteItems (MediaBrowser.Controller.Entities.BaseItem[] items) [0x0002b] in <05979d9270ab4dafac52ba5587e41c17>:0
at Emby.Server.Implementations.Library.LibraryManager.DeleteItem (MediaBrowser.Controller.Entities.BaseItem item, MediaBrowser.Controller.Library.DeleteOptions options, MediaBrowser.Controller.Entities.BaseItem parent, System.Boolean notifyParentItem) [0x00265] in <05979d9270ab4dafac52ba5587e41c17>:0
at MediaBrowser.Controller.Entities.Folder.ValidateChildrenInternal (System.IProgress`1[T] progress, System.Threading.CancellationToken cancellationToken, System.Boolean recursive, System.Boolean refreshChildMetadata, MediaBrowser.Controller.Providers.MetadataRefreshOptions refreshOptions, MediaBrowser.Controller.Providers.IDirectoryService directoryService) [0x0022e] in <42f6882bd6d34115bbdd5b6f8b48c222>:0
at MediaBrowser.Controller.Entities.Folder.ValidateSubFolders (MediaBrowser.Controller.Entities.BaseItem[] children, MediaBrowser.Controller.Providers.IDirectoryService directoryService, System.IProgress`1[T] progress, System.Threading.CancellationToken cancellationToken) [0x0013f] in <42f6882bd6d34115bbdd5b6f8b48c222>:0
at MediaBrowser.Controller.Entities.Folder.ValidateChildrenInternal (System.IProgress`1[T] progress, System.Threading.CancellationToken cancellationToken, System.Boolean recursive, System.Boolean refreshChildMetadata, MediaBrowser.Controller.Providers.MetadataRefreshOptions refreshOptions, MediaBrowser.Controller.Providers.IDirectoryService directoryService) [0x00395] in <42f6882bd6d34115bbdd5b6f8b48c222>:0
at Emby.Server.Implementations.Library.LibraryManager.PerformLibraryValidation (System.IProgress`1[T] progress, System.Threading.CancellationToken cancellationToken) [0x0014a] in <05979d9270ab4dafac52ba5587e41c17>:0
at Emby.Server.Implementations.Library.LibraryManager.ValidateMediaLibraryInternal (System.IProgress`1[T] progress, System.Threading.CancellationToken cancellationToken) [0x00092] in <05979d9270ab4dafac52ba5587e41c17>:0
at Emby.Server.Implementations.ScheduledTasks.ScheduledTaskWorker.ExecuteInternal (MediaBrowser.Model.Tasks.TaskOptions options) [0x001b0] in <05979d9270ab4dafac52ba5587e41c17>:0

 

embyserver-63767911477.txt

Edited September 22, 2021 by NikaNika

[GrimReaper 3314]

4 minutes ago, NikaNika said:

But! I lost all of my manually added metadata

 

4 minutes ago, NikaNika said:

but once again I lost all of my manually added metadata.

 

4 minutes ago, NikaNika said:

Someone help me please, and tell me what I should do

You should enable saving NFOs and images in your media folders. That way you will retain all your edits and artwork upon any rebuild, as they will not be scraped but read from local files. 

Edited September 22, 2021 by GrimReaper
Typo

[GrimReaper 3314]

7 minutes ago, NikaNika said:

But I think now when my library started its existence from the very beginning once again and it's working and scanning fine, I think it will create the backup. But will it help to restore all of my manually added metadata, if the scan fails once again?

Server Configuration Backup Plugin will only restore your library definitions, it doesn't store your metadata and/or artwork. 

[NikaNika 0]

Thanks!

Started my library anew and now I'm adding all the required metadata manually. Before that I created a new folder and enabled saving NFOs and images into it. But later I searched in this folder and didn't find any NFO files, just images (despite the fact that I already edited some descriptions). It worries me, because all the posters which I uploaded were automatically added anyway (after I started my library anew before), but descriptions and the rest of metadata were not. 

Edited September 22, 2021 by NikaNika

[GrimReaper 3314]

Make sure you enabled metadata saver not (only) reader in your library settings. 

Edit: Which AndroidTV box is that? 

Edited September 22, 2021 by GrimReaper
Append

[NikaNika 0]

52 minutes ago, GrimReaper said:

Edit: Which AndroidTV box is that? 

Ugoos X3 Plus

[NikaNika 0]

55 minutes ago, GrimReaper said:

Make sure you enabled metadata saver not (only) reader in your library settings.

Yeah, it finally found those settings and enabled it.

Already edited manually a couple of descriptions, but not many. Probably I need to delete the library.db file now and start my library anew again just to make sure that these edits haven't disappeared...

[Carlo 4330]

That's what I would do.  Let Emby rebuild the database for you.

Make sure you make this change for NFO saving for all libraries first.

[Happy2Play 8317]

Has this delete issue causing "SQLitePCL.pretty.SQLiteException: Constraint: FOREIGN KEY constraint failed" on Android server been identified?

Is there a better way to recover then a new database?

Edited September 22, 2021 by Happy2Play

[Carlo 4330]

Not that I know of.  To me it seems like a "show stopper" for running Emby Server on Android right now.

@Luke anything we can do to assist?

[Luke 37132]

We are looking into it. Thanks.

[Carlo 4330]

4.7.0.18 (beta) fixes the SQLite Foreign Key (corruption) issue.
You can download it from our website: https://emby.media/server-android.html

You can install this over your current system and be able to run a Full Library Scan without any of the SQLite errors.

[twalker294 2]

Carlo, is there a beta for Windows that will fix this issue as well? I have to delete the library.db-shm and -wal almost every time I add a new video to my library then rescan.

 

Thanks

Todd

[Luke 37132]

8 hours ago, twalker294 said:

Carlo, is there a beta for Windows that will fix this issue as well? I have to delete the library.db-shm and -wal almost every time I add a new video to my library then rescan.

 

Thanks

Todd

Windows does not have the same issue. Please open a new topic to describe your issue. Thanks.

[Carlo 4330]

On 11/15/2021 at 1:11 AM, twalker294 said:

Carlo, is there a beta for Windows that will fix this issue as well? I have to delete the library.db-shm and -wal almost every time I add a new video to my library then rescan.

Like Luke mentioned it's never been a problem on Windows, just that one platform.
If you see something like this on any platform now (assuming your running the latest version) please report it as it should be fixed.

[twalker294 2]

Thanks I got it figured out. I had installed emby as a service previously and it was also running as a startup item so it was running twice and I think one wasn’t shutting down on reboot so it was corrupting the db. I disabled the service and haven’t had any problems since. 

[Luke 37132]

Thanks for the feedback.

[hbbs 6]

This error seems to be back on NSTV running Android TV 11

Already running the latest Emby Server Beta  4.7.0.20

 

Emby-Server-log.txt

[Happy2Play 8317]

1 hour ago, hbbs said:

This error seems to be back on NSTV running Android TV 11

Already running the latest Emby Server Beta  4.7.0.20

 

Emby-Server-log.txt 6.42 kB · 0 downloads

Please post the server log to show us the exact sql error being thrown.

 

[hbbs 6]

20 minutes ago, Happy2Play said:

Please post the server log to show us the exact sql error being thrown.

 

Here it is.

Hope that helps it.

Please, let me know otherwise.

embyserver.txt

[Happy2Play 8317]

2 hours ago, hbbs said:

Here it is.

Hope that helps it.

Please, let me know otherwise.

embyserver.txt 45.64 kB · 0 downloads

Sorry there are no sql errors in that log only ffmpeg errors.

[Luke 37132]

7 hours ago, hbbs said:

Here it is.

Hope that helps it.

Please, let me know otherwise.

embyserver.txt 45.64 kB · 1 download

Why do you think the library scan failed?

[hbbs 6]

1 hour ago, Luke said:

Why do you think the library scan failed?

I have no idea. 

After I upgraded to the latest Android 11 version. I went to try to play something on emby and noticed that emby server wasn't running. Yes, if you reboot NSTV running Android 11, emby server has to be manually started.

After I manually started. I noticed that my movies were all gone. 

My TV Shows were also gone. But its art covers were there. But only that. There's nothing actually to be played.

The same for audiobooks. I even deleted my whole library from the dashboard. And the re-added everything.

 

That was when I first noticed the warnings with SQL errors.

 

[Luke 37132]

5 hours ago, hbbs said:

I have no idea. 

After I upgraded to the latest Android 11 version. I went to try to play something on emby and noticed that emby server wasn't running. Yes, if you reboot NSTV running Android 11, emby server has to be manually started.

After I manually started. I noticed that my movies were all gone. 

My TV Shows were also gone. But its art covers were there. But only that. There's nothing actually to be played.

The same for audiobooks. I even deleted my whole library from the dashboard. And the re-added everything.

 

That was when I first noticed the warnings with SQL errors.

 

There are a couple threads recently about the new Shield OS update. I would just follow along in the android server section of the community:

Thanks.

[nickysilver 9]

UPDATE: The Nvidia Shield hot fix, did just that. The drive storing emby became readable. I had to scan the movies/tv shows again. Interesting that emby retained SOME data.  For instance, custom artwork for collections was there. (Mind you the collections weren't. But when I recreated them, the artwork was in place.)

Sadly, the custom sort names did not come back. So, I spend about 36 hours, redoing that. (Yes, I know, high strung, but every movies gets edited so all 2400 appear in order from favorite-- to junk I'll probably never watch again.) Oh well..... I've done it before and I'm getting pretty fast at it. Used to take me a week!