cappapp 7 Posted January 6, 2023 Share Posted January 6, 2023 Looks like I have a corrupt library.db file, most likely from some recent-ish hard reboots when I thought the server froze, but didn't. Server still seems to be working fine, the only error I'm getting at the moment is a failing thumbnail extraction job at 2am, but my thumbnails old and new seem fine. For now So I guess I should do something about it before it gets worse. I have some backups of my appdata folder, which will include a copy of this file. Is it worth it to try and restore it? I'm not sure how long this has been happening and the backup might be a copy of corrupted version. Or should I just delete and rebuild the file? I know I need to rescan the library, but will that miss any data I have manually updated in terms of identifying correct movies or shows? Or will that stuff be OK, because the metadata should be in the .NFO files stored with the media? Also my library is over 100TB, so I imagine it will take a long time. Let me know any tips or tricks or checks before I begin. Cheers ERRORS: ---Alerts Emby Dashboard Thumbnail image extraction Failed on <servername> 1/6/2023, 2.00.16 AM Running time: 16 Seconds ---Error from info popup in Alerts Dashboard Emby Exception of type 'SQLitePCL.pretty.SQLiteException' was thrown. at SQLitePCL.pretty.SQLiteException.CheckOk(sqlite3 db, Int32 rc) at SQLitePCL.pretty.StatementImpl.MoveNext() at Emby.Sqlite.SqliteExtensions.ExecuteQuery(IStatement This)+MoveNext() at Emby.Server.Implementations.Data.SqliteItemRepository.QueryItemsInTransaction[T](InternalItemsQuery query, IDatabaseConnection db, String[] columnsToSelect, Func`5 rowReaderFn, Boolean singleResult, String methodName) at Emby.Server.Implementations.Data.SqliteItemRepository.GetInternalItemIdsList(InternalItemsQuery query) at Emby.Providers.MediaInfo.ChapterImagesTask.GetAllItemIdsToRefresh(Boolean extractImages) at Emby.Providers.MediaInfo.ChapterImagesTask.Execute(CancellationToken cancellationToken, IProgress`1 progress) at Emby.Server.Implementations.ScheduledTasks.ScheduledTaskWorker.ExecuteInternal(TaskOptions options) ---Error message in Emby log file 2023-01-06 21:54:50.407 Error TaskManager: Error *** Error Report *** Version: 4.7.11.0 Command line: /system/EmbyServer.dll -programdata /config -ffdetect /bin/ffdetect -ffmpeg /bin/ffmpeg -ffprobe /bin/ffprobe -restartexitcode 3 Operating system: Linux version 5.15.46-Unraid (root@Develop) (gcc (GCC) 11.2.0, GNU ld version 2.37-slack15) #1 SMP Fri Jun 10 11:08:41 PDT 2022 Framework: .NET 6.0.8 OS/Process: x64/x64 Runtime: system/System.Private.CoreLib.dll Processor count: 24 Data path: /config Application path: /system SQLitePCL.pretty.SQLiteException: Corrupt: database disk image is malformed SQLitePCL.pretty.SQLiteException: Exception of type 'SQLitePCL.pretty.SQLiteException' was thrown. at SQLitePCL.pretty.SQLiteException.CheckOk(sqlite3 db, Int32 rc) at SQLitePCL.pretty.StatementImpl.MoveNext() at Emby.Sqlite.SqliteExtensions.ExecuteQuery(IStatement This)+MoveNext() at Emby.Server.Implementations.Data.SqliteItemRepository.QueryItemsInTransaction[T](InternalItemsQuery query, IDatabaseConnection db, String[] columnsToSelect, Func`5 rowReaderFn, Boolean singleResult, String methodName) at Emby.Server.Implementations.Data.SqliteItemRepository.GetInternalItemIdsList(InternalItemsQuery query) at Emby.Providers.MediaInfo.ChapterImagesTask.GetAllItemIdsToRefresh(Boolean extractImages) at Emby.Providers.MediaInfo.ChapterImagesTask.Execute(CancellationToken cancellationToken, IProgress`1 progress) at Emby.Server.Implementations.ScheduledTasks.ScheduledTaskWorker.ExecuteInternal(TaskOptions options) Source: SQLitePCL.pretty TargetSite: Void CheckOk(SQLitePCLEx.sqlite3, Int32) ---Error on server logs file BTRFS warning (device sdb1): csum failed root 5 ino 309 off 144154624 csum 0xccaf53f8 expected csum 0x8188ffff mirror 2 ---Lookup corrupt file on server root@<servername>:/var/log# find /mnt/cache -inum 309 /mnt/cache/appdata/EmbyServer/data/library.db Link to comment Share on other sites More sharing options...
Happy2Play 8332 Posted January 6, 2023 Share Posted January 6, 2023 5 hours ago, cappapp said: Corrupt: database disk image is malformed If you have a working back that would be the fastest restore process otherwise you will have to rebuild the db. Shouldn't have any issues if you have existing nfo files as they will/should be read per your library settings. Corrupt Database : (emby.media) Link to comment Share on other sites More sharing options...
cappapp 7 Posted January 7, 2023 Author Share Posted January 7, 2023 Yep, I'll try that first. Actually I'm in luck. My backups go further backups than I thought. I still might try the latest one first, then oldest and worst case a rebuild. Link to comment Share on other sites More sharing options...
cappapp 7 Posted January 14, 2023 Author Share Posted January 14, 2023 Hi, So to update on this. Replacing the library.db file from backup has done the trick. Had to go back about a month which my current oldest backup. Which got me thinking, I might start a backup archive of individual important files that'll allow me to go back a year or two (pending space requirements). Beside library.db are there any other major files that'll save me time consuming a rebuild? Possibly the other .db files? Cheers Link to comment Share on other sites More sharing options...
cappapp 7 Posted January 14, 2023 Author Share Posted January 14, 2023 (edited) Ah, I have just realised all the playback data is lost up until that point. I should of acted faster on fixing the issue. I originally just thought it was a thumbnail issue, which at the time still looked fine to me. And only what Emby alerts was reporting. Not the DB malfunction Is there a way to get email alerts on errors like that? I guess I'm just lucky it wasn't everything and had to do a full rebuild. Edited January 14, 2023 by cappapp Link to comment Share on other sites More sharing options...
Solution Luke 37180 Posted January 14, 2023 Solution Share Posted January 14, 2023 11 hours ago, cappapp said: Hi, So to update on this. Replacing the library.db file from backup has done the trick. Had to go back about a month which my current oldest backup. Which got me thinking, I might start a backup archive of individual important files that'll allow me to go back a year or two (pending space requirements). Beside library.db are there any other major files that'll save me time consuming a rebuild? Possibly the other .db files? Cheers We don't really suggest selective backups where you choose to backup one piece of the data folder. I would backup the entire data folder of the server, while omitting the cache and transcoding-temp subfolders. 1 Link to comment Share on other sites More sharing options...
Happy2Play 8332 Posted January 14, 2023 Share Posted January 14, 2023 If Premiere user the server configuration backup plugin backs up user/userdata. If you copied/backup the corrupt db, you could possibly export the userdatas table from corrupt db to working db depending on what is corrupt. Link to comment Share on other sites More sharing options...
cappapp 7 Posted January 15, 2023 Author Share Posted January 15, 2023 (edited) 4 hours ago, Luke said: We don't really suggest selective backups where you choose to backup one piece of the data folder. I would backup the entire data folder of the server, while omitting the cache and transcoding-temp subfolders. Oh OK. Whoops. I can do that. Maybe I still will. I still have users blocked from access, as it's still building. It was a pretty big month of adding stuff. 3 hours ago, Happy2Play said: If Premiere user the server configuration backup plugin backs up user/userdata. If you copied/backup the corrupt db, you could possibly export the userdatas table from corrupt db to working db depending on what is corrupt. That sounds good. I am. Where abouts can I find this option? It is already installed, but I probably never configured it. EDIT: Oh, there it is, yep. Never config'd Hmmm, Would it be worth restoring the last backup as a separate instance to get this data. Thanks all Edited January 15, 2023 by cappapp Link to comment Share on other sites More sharing options...
Happy2Play 8332 Posted January 15, 2023 Share Posted January 15, 2023 12 minutes ago, cappapp said: 3 hours ago, Happy2Play said: If Premiere user the server configuration backup plugin backs up user/userdata. If you copied/backup the corrupt db, you could possibly export the userdatas table from corrupt db to working db depending on what is corrupt. That sounds good. I am. Where abouts can I find this option? It is already installed, but I probably never configured it. You can find the Server Configuration Backup plugin in the catalog. Then are install and restart click on the plugin to set a backup path. Link to comment Share on other sites More sharing options...
cappapp 7 Posted January 15, 2023 Author Share Posted January 15, 2023 Sorry, just made an edit. Yeah. EDIT: Oh, there it is, yep. Never config'd Hmmm, Would it be worth restoring the last backup as a separate instance to get this data. Trying to figure out if it's worth restoring the recent backup to get this data for 1 month. Actually more concerned I have made a bigger mistake by restoring only the .db file not the whole emby appdata folder. I assume I am more at risk of future Emby issues because of that? Link to comment Share on other sites More sharing options...
cappapp 7 Posted January 15, 2023 Author Share Posted January 15, 2023 (edited) Is there a best practice guide for Large Emby servers? 100TB+ etc? Like changing the DB cache size for performance or something like that? I saw that option but it says it's only for if you have a reason to change it. Not sure if this is a good enough reason. On the plus side, the current running restored version, whilst still updating, completed the thumbnail extraction process. Which was failing previously and haven't seen any other errors in the logs as yet. Edited January 15, 2023 by cappapp Link to comment Share on other sites More sharing options...
Happy2Play 8332 Posted January 15, 2023 Share Posted January 15, 2023 30 minutes ago, cappapp said: Sorry, just made an edit. Yeah. EDIT: Oh, there it is, yep. Never config'd Hmmm, Would it be worth restoring the last backup as a separate instance to get this data. Trying to figure out if it's worth restoring the recent backup to get this data for 1 month. Actually more concerned I have made a bigger mistake by restoring only the .db file not the whole emby appdata folder. I assume I am more at risk of future Emby issues because of that? Restoring your older db will only lose users watched data and have to reimport anything added in that timeframe (library scan). But with the SCB plugin you can restore only said users/userdata if you needed to in this scenario. But overall best practice is having a full backup of Emby programdata folder if possible. As for db cache size if you have the RAM you should set it to 1.5 to 2 times your library.db size to provide better performance as the default isn't really for such a large media collection. 1 Link to comment Share on other sites More sharing options...
cappapp 7 Posted January 15, 2023 Author Share Posted January 15, 2023 22 minutes ago, Happy2Play said: Restoring your older db will only lose users watched data and have to reimport anything added in that timeframe (library scan). But with the SCB plugin you can restore only said users/userdata if you needed to in this scenario. But overall best practice is having a full backup of Emby programdata folder if possible. As for db cache size if you have the RAM you should set it to 1.5 to 2 times your library.db size to provide better performance as the default isn't really for such a large media collection. Cool, thanks for clarifying. Thanks. I'll hold onto the known pre-corrupt full backup for awhile longer and see how this goes when it's finished rebuilding. If I mess around too much, I might risk making it worse? And thanks again Will also alter the current db cache after everything has settled down. The current DB size is about 600mb. It's actually been running not too bad on the default settings, maybe a little laggy from time to time. Hopefully I notice an improvement after. I only have about 12 users and they are not super active, often only 2-3 concurrent. Cheers 1 Link to comment Share on other sites More sharing options...
Recommended Posts
Create an account or sign in to comment
You need to be a member in order to leave a comment
Create an account
Sign up for a new account in our community. It's easy!
Register a new accountSign in
Already have an account? Sign in here.
Sign In Now