Jump to content

Emby for Kodi - Kodi Database WAL size explodes during Movie import


luziferius

Recommended Posts

luziferius

Hi,

 

I have a weird issue during the movie database import when using Emby for Kodi.

 

Background: I am currently in the process of creating a media center using a Raspberry Pi 4 and Kodi. To access my Library on the Pi, I use the Emby for Kodi plugin.

 

 

Kodi version: 18.6-1 from https://archlinuxarm.org/packages/armv7h/kodi-rbp4

Emby for Kodi version: 4.1.19, obtained from the stable repository here: https://kodi.emby.tv/

 

 

Problem: The SQLite3 write-ahead log (WAL) of the Kodi database file MyVideos116.db grows excessively in size during the movie database import.

I have about 400 movies (ripped from my DVD collection).

Right now, the import is 10% done, the database has a size of about 13MiB (that’s ok) and the related WAL (MyVideos116.db-wal) has a size of ~ 5.9GiB (not ok). I suspect that the WAL will take about 60GiB of hard disk space when the process finishes. Each individual movie takes about 30 seconds to import and adds ~ 100MiB to the WAL file size.

 

Contrary, the tv series import was really fast and worked well, taking about some milliseconds per episode. Everything was fine with series episodes.

 

Monitoring the database over SSH, import ~13% done:

post-165326-0-76634700-1585590022_thumb.png

Link to comment
Share on other sites

Hi, that's strange, I wonder what it's putting in there. @@jachin99 have you seen this before?

 

Does it put images into the database?

Link to comment
Share on other sites

jachin99

I'll have to look when I get a chance. My clients are all windows but I haven't run out of space on my one machine with 18.6 on it

Link to comment
Share on other sites

luziferius

The network itself is almost idle. And the WAL grows even for films without fan art. So it doesn’t look like it stores images, as I’d see that on the network monitor.

 

I wanted to test what happens if i set the journal mode to the previous-default mode by executing

$ sqlite3 MyVideos116.db  <<< "PRAGMA journal_mode = 'delete';"

Unfortunately, this is automatically reverted to WAL mode upon trying to start a synchronization, so I can’t test if it only manifests the issues in WAL mode…

Needs further investigation…

Link to comment
Share on other sites

@@sualfred do you know if the database holds the contents of images? That's the only thing I can think of that would cause this.

Link to comment
Share on other sites

jachin99

I'm not sure where the write ahead log is located on Windows so I peeked around my userdata folder in kodi under USERNAME/appdata/roaming/kodi.  The total size of that folder is 3.17gb, and I have a few hundred files in the movie db alone most of which are bluray.  This PC is on kodi 18.6, and E4k current stable. 

Edited by jachin99
Link to comment
Share on other sites

luziferius

SQlite holds it’s files together. So the WAL file is always alongside the database file. It only exists during the lifetime of a database connection, it is removed when Kodi is closed.

 

According to https://kodi.wiki/view/Databases and https://kodi.wiki/view/Userdata, under Windows it is located at:

%APPDATA%\Kodi\userdata\Database

On my main system, the database itself is ~20MiB large.

 

Edit:

 

I did the research that I should have done in the first place (as always…)

 

This issue seems to be caused by my library layout:

I kind of dislike redundancy. I didn’t want to have this directory structure when I initially set up my library years ago:

Movie1/Movie1.mkv
Movie2/Movie2.mkv
Movie3/Movie3.mkv
Movie4/Movie4.mkv

So I built a flat structure. A single directory that simply contains all movie files. Simple and easy to browse and access using file managers.

It works fine with Kodi. But it seems that Emby actually requires the structure above, with each file inside it’s own directory.

 

 

The flat structure causes each movie to be reported with the backdrop of each movie. So this is a simple quadratic term, all of my 400 movies are reported to have each ~2500 backdrops. This is placed in the Kodi database by the Emby plugin, causing the database to explode during import.

 

Here is a Kodi debug log of the first 2% of the import process: kodi.7z Beware, this is a 7zip compressed log file, it expands to about 1.5GiB in size if extracted, so your antivirus software might report a ZIP bomb.

 

You can see these JSON response objects in the log (truncated to not overshoot the maximum forum post length):

 

 

{
"Genres": [
    "Drama"
], 
"OfficialRating": "FSK-12", 
"Etag": "43475408900c657fb2d2dbfdf7710976", 
"TagItems": [], 
"RemoteTrailers": [
    {
        "Url": "https://www.youtube.com/watch?v=T3MAp5D_YlE", 
        "Name": "Deutsch HD"
    }
], 
"ServerId": "f6692b0e85154ea68a2b4179030270c3", 
"PresentationUniqueKey": "6e9719d6e1577bea604437ed5bf16ec3", 
"ProductionLocations": [
    "United States of America"
], 
"SortName": "0000000008 mile", 
"Tags": [], 
"Type": "Movie", 
"IsFolder": false, 
"ProviderIds": {
    "Tmdb": "65", 
    "Imdb": "tt0298203"
}, 
"PremiereDate": "2002-11-07T23:00:00.0000000+00:00", 
"ProductionYear": 2002, 
"Path": "/mnt/Filme/Filme/8 Mile [2002].mkv", 
"LocalTrailerCount": 0, 
"BackdropImageTags": [
    "7b5876b9c5b473ec7b92844531bd663e", 
    "fb414172fb9f8c452413f2d4b627d033", 
    "02f79ee40bdc876c6a3b8ac61cc9f79d", 
    "bcfce063c25783971e0d1ecaa4c22db5", 
    "271b2734fbdd47fa975952a7cba9f3fa", 
    "9346bc8c85e6627ad7093c467e0df5b3", 
    "e81a29491c6f3dbebe9c149f2a255adc", 
    "33a08e55e53755a9f3203e0ec0797b32", 
    "c7e667671b83f3064e5f4b478fb78a10", 
    "a004f955c5a2d9d38a66c589af5e702b", 
    "f244e90021197b765f535eca5523bcf4", 
    "cfe481ee148cd47e6491823c4b1f8f6e", 
    "0cc89acd1a022535634811c57003056d", 
    "923b0ee496ce833bc662721812fb4cb2", 
    "38932b2605c811c66d62458f294c9a38", 
    "877d77231b6650f1feb913956865c492", 
    "ef3ce84301b518f94cdb16104cc8a416", 
    "b1cc16aba6b8fdda6acf76a2fcfb911e", 
    "702db01d400f2d30ff4a7f223ac751f5", 
    "e7fd85a00360ce648193ff180f569670", 
    "d81c3119fb9d281bb925c24388fd133a", 
    "23e863ade3e0a668166b48c1ab0c9866", 
    "5dd6ce5027c00e9a831e84a5a1fc0de6", 
    "fa1383f4793deb902d459afc0512654d", 
    "a330b026b75f919b43975bc355282da2", 
    "9307b694997516baf3b3d2744ef564d3", 
    "542a68984ef322276890a4db183570e1", 
    "e1f2abca5af4b2a132c3df3e62c3fb65", 
    "6f67424ad3a26ecdbe96d9f6a44ad1f6", 
    "01571eed813dd7b4003167df834d7e05", 
    "4aa99b12e69a49ea1ab385652e73e3f2", 
    "25e8b0eb84080cc809549508100a43ba", 
    "3b0b4fd548714ae1d1fd9cd0f3121a2f", 
    "68581b5b5764bafcf36c2914249f8600", 
    "320ca5ffb1c06f1a5715b1cf1d8ca6f4", 
    "b2af6259bcc06e70d5cd46b334dde183", 
    "10f8731778d0cbd212e13ab6f8f356da", 
    "cee4332559c699f5626c30397e22b017", 
    "af455718065603ffeca31a3ca53c9d83", 
    "a4d25475d710855a2c9c699ffb2e8e15", 
    "070f7a3df2788591824df1edc335a7a7", 
    "f100acc8e36f041facdc03f4f08d4cc4", 
    "2dd41d5620476b828114dcf77a578dba", 
    "ca340d0483c53158b79493b043e11133", 
    "baea927181d2ba19701dc8fd2d87a1a3", 
    "2896cfa1e7ac36b58cca2ebe351bb90d", 
    "afae3b7777d7759228b2eb3f10d6e631", 
    "a8019eb3ac0522007f649ed1fbeac95f", 
    "7a61527e326ab62b26b7cc2415d94d58", 
    "4c25ff6f61a0d5bc4a1354742840fd8f", 
    "9bab9f1156e15c781669aa52a4d795e6",

 

 

 

Edit 2:

 

Yep. It seems to add all files contained in the "extrafanarts" directory (that the Kodi extrafanart downloader plugin placed there) as a backdrop for each movie in the database. That directory contains ~2500 files, which fits the observed data.

 

So a proposed fix: Limit the number of certain elements when gathering data. Like limit database entries to have at most X (~ 10 - 100) backdrops.

Edited by luziferius
Link to comment
Share on other sites

Guest
This topic is now closed to further replies.
×
×
  • Create New...