luziferius 0 Posted March 30, 2020 Share Posted March 30, 2020 Hi, I have a weird issue during the movie database import when using Emby for Kodi. Background: I am currently in the process of creating a media center using a Raspberry Pi 4 and Kodi. To access my Library on the Pi, I use the Emby for Kodi plugin. Kodi version: 18.6-1 from https://archlinuxarm.org/packages/armv7h/kodi-rbp4 Emby for Kodi version: 4.1.19, obtained from the stable repository here: https://kodi.emby.tv/ Problem: The SQLite3 write-ahead log (WAL) of the Kodi database file MyVideos116.db grows excessively in size during the movie database import. I have about 400 movies (ripped from my DVD collection). Right now, the import is 10% done, the database has a size of about 13MiB (that’s ok) and the related WAL (MyVideos116.db-wal) has a size of ~ 5.9GiB (not ok). I suspect that the WAL will take about 60GiB of hard disk space when the process finishes. Each individual movie takes about 30 seconds to import and adds ~ 100MiB to the WAL file size. Contrary, the tv series import was really fast and worked well, taking about some milliseconds per episode. Everything was fine with series episodes. Monitoring the database over SSH, import ~13% done: Link to comment Share on other sites More sharing options...
Luke 36879 Posted March 31, 2020 Share Posted March 31, 2020 Hi, that's strange, I wonder what it's putting in there. @@jachin99 have you seen this before? Does it put images into the database? Link to comment Share on other sites More sharing options...
jachin99 82 Posted March 31, 2020 Share Posted March 31, 2020 I'll have to look when I get a chance. My clients are all windows but I haven't run out of space on my one machine with 18.6 on it Link to comment Share on other sites More sharing options...
luziferius 0 Posted March 31, 2020 Author Share Posted March 31, 2020 The network itself is almost idle. And the WAL grows even for films without fan art. So it doesn’t look like it stores images, as I’d see that on the network monitor. I wanted to test what happens if i set the journal mode to the previous-default mode by executing $ sqlite3 MyVideos116.db <<< "PRAGMA journal_mode = 'delete';" Unfortunately, this is automatically reverted to WAL mode upon trying to start a synchronization, so I can’t test if it only manifests the issues in WAL mode… Needs further investigation… Link to comment Share on other sites More sharing options...
Luke 36879 Posted March 31, 2020 Share Posted March 31, 2020 @@sualfred do you know if the database holds the contents of images? That's the only thing I can think of that would cause this. Link to comment Share on other sites More sharing options...
jachin99 82 Posted March 31, 2020 Share Posted March 31, 2020 (edited) I'm not sure where the write ahead log is located on Windows so I peeked around my userdata folder in kodi under USERNAME/appdata/roaming/kodi. The total size of that folder is 3.17gb, and I have a few hundred files in the movie db alone most of which are bluray. This PC is on kodi 18.6, and E4k current stable. Edited March 31, 2020 by jachin99 Link to comment Share on other sites More sharing options...
luziferius 0 Posted March 31, 2020 Author Share Posted March 31, 2020 (edited) SQlite holds it’s files together. So the WAL file is always alongside the database file. It only exists during the lifetime of a database connection, it is removed when Kodi is closed. According to https://kodi.wiki/view/Databases and https://kodi.wiki/view/Userdata, under Windows it is located at: %APPDATA%\Kodi\userdata\Database On my main system, the database itself is ~20MiB large. Edit: I did the research that I should have done in the first place (as always…) This issue seems to be caused by my library layout: I kind of dislike redundancy. I didn’t want to have this directory structure when I initially set up my library years ago: Movie1/Movie1.mkv Movie2/Movie2.mkv Movie3/Movie3.mkv Movie4/Movie4.mkv So I built a flat structure. A single directory that simply contains all movie files. Simple and easy to browse and access using file managers. It works fine with Kodi. But it seems that Emby actually requires the structure above, with each file inside it’s own directory. The flat structure causes each movie to be reported with the backdrop of each movie. So this is a simple quadratic term, all of my 400 movies are reported to have each ~2500 backdrops. This is placed in the Kodi database by the Emby plugin, causing the database to explode during import. Here is a Kodi debug log of the first 2% of the import process: kodi.7z Beware, this is a 7zip compressed log file, it expands to about 1.5GiB in size if extracted, so your antivirus software might report a ZIP bomb. You can see these JSON response objects in the log (truncated to not overshoot the maximum forum post length): { "Genres": [ "Drama" ], "OfficialRating": "FSK-12", "Etag": "43475408900c657fb2d2dbfdf7710976", "TagItems": [], "RemoteTrailers": [ { "Url": "https://www.youtube.com/watch?v=T3MAp5D_YlE", "Name": "Deutsch HD" } ], "ServerId": "f6692b0e85154ea68a2b4179030270c3", "PresentationUniqueKey": "6e9719d6e1577bea604437ed5bf16ec3", "ProductionLocations": [ "United States of America" ], "SortName": "0000000008 mile", "Tags": [], "Type": "Movie", "IsFolder": false, "ProviderIds": { "Tmdb": "65", "Imdb": "tt0298203" }, "PremiereDate": "2002-11-07T23:00:00.0000000+00:00", "ProductionYear": 2002, "Path": "/mnt/Filme/Filme/8 Mile [2002].mkv", "LocalTrailerCount": 0, "BackdropImageTags": [ "7b5876b9c5b473ec7b92844531bd663e", "fb414172fb9f8c452413f2d4b627d033", "02f79ee40bdc876c6a3b8ac61cc9f79d", "bcfce063c25783971e0d1ecaa4c22db5", "271b2734fbdd47fa975952a7cba9f3fa", "9346bc8c85e6627ad7093c467e0df5b3", "e81a29491c6f3dbebe9c149f2a255adc", "33a08e55e53755a9f3203e0ec0797b32", "c7e667671b83f3064e5f4b478fb78a10", "a004f955c5a2d9d38a66c589af5e702b", "f244e90021197b765f535eca5523bcf4", "cfe481ee148cd47e6491823c4b1f8f6e", "0cc89acd1a022535634811c57003056d", "923b0ee496ce833bc662721812fb4cb2", "38932b2605c811c66d62458f294c9a38", "877d77231b6650f1feb913956865c492", "ef3ce84301b518f94cdb16104cc8a416", "b1cc16aba6b8fdda6acf76a2fcfb911e", "702db01d400f2d30ff4a7f223ac751f5", "e7fd85a00360ce648193ff180f569670", "d81c3119fb9d281bb925c24388fd133a", "23e863ade3e0a668166b48c1ab0c9866", "5dd6ce5027c00e9a831e84a5a1fc0de6", "fa1383f4793deb902d459afc0512654d", "a330b026b75f919b43975bc355282da2", "9307b694997516baf3b3d2744ef564d3", "542a68984ef322276890a4db183570e1", "e1f2abca5af4b2a132c3df3e62c3fb65", "6f67424ad3a26ecdbe96d9f6a44ad1f6", "01571eed813dd7b4003167df834d7e05", "4aa99b12e69a49ea1ab385652e73e3f2", "25e8b0eb84080cc809549508100a43ba", "3b0b4fd548714ae1d1fd9cd0f3121a2f", "68581b5b5764bafcf36c2914249f8600", "320ca5ffb1c06f1a5715b1cf1d8ca6f4", "b2af6259bcc06e70d5cd46b334dde183", "10f8731778d0cbd212e13ab6f8f356da", "cee4332559c699f5626c30397e22b017", "af455718065603ffeca31a3ca53c9d83", "a4d25475d710855a2c9c699ffb2e8e15", "070f7a3df2788591824df1edc335a7a7", "f100acc8e36f041facdc03f4f08d4cc4", "2dd41d5620476b828114dcf77a578dba", "ca340d0483c53158b79493b043e11133", "baea927181d2ba19701dc8fd2d87a1a3", "2896cfa1e7ac36b58cca2ebe351bb90d", "afae3b7777d7759228b2eb3f10d6e631", "a8019eb3ac0522007f649ed1fbeac95f", "7a61527e326ab62b26b7cc2415d94d58", "4c25ff6f61a0d5bc4a1354742840fd8f", "9bab9f1156e15c781669aa52a4d795e6", Edit 2: Yep. It seems to add all files contained in the "extrafanarts" directory (that the Kodi extrafanart downloader plugin placed there) as a backdrop for each movie in the database. That directory contains ~2500 files, which fits the observed data. So a proposed fix: Limit the number of certain elements when gathering data. Like limit database entries to have at most X (~ 10 - 100) backdrops. Edited April 1, 2020 by luziferius Link to comment Share on other sites More sharing options...
Recommended Posts