Jump to content


Photo

Emby for Kodi - Kodi Database WAL size explodes during Movie import


  • Please log in to reply
6 replies to this topic

#1 luziferius OFFLINE  

luziferius

    Newbie

  • Members
  • 7 posts
  • Local time: 03:04 PM
  • LocationGermany

Posted 30 March 2020 - 01:43 PM

Hi,

 

I have a weird issue during the movie database import when using Emby for Kodi.

 

Background: I am currently in the process of creating a media center using a Raspberry Pi 4 and Kodi. To access my Library on the Pi, I use the Emby for Kodi plugin.

 

 

Kodi version: 18.6-1 from https://archlinuxarm...rmv7h/kodi-rbp4

Emby for Kodi version: 4.1.19, obtained from the stable repository here: https://kodi.emby.tv/

 

 

Problem: The SQLite3 write-ahead log (WAL) of the Kodi database file MyVideos116.db grows excessively in size during the movie database import.

I have about 400 movies (ripped from my DVD collection).

Right now, the import is 10% done, the database has a size of about 13MiB (that’s ok) and the related WAL (MyVideos116.db-wal) has a size of ~ 5.9GiB (not ok). I suspect that the WAL will take about 60GiB of hard disk space when the process finishes. Each individual movie takes about 30 seconds to import and adds ~ 100MiB to the WAL file size.

 

Contrary, the tv series import was really fast and worked well, taking about some milliseconds per episode. Everything was fine with series episodes.

 

Monitoring the database over SSH, import ~13% done:

Attached File  Kodi_database.png   77.74KB   2 downloads



#2 Luke OFFLINE  

Luke

    System Architect

  • Administrators
  • 156872 posts
  • Local time: 09:04 AM

Posted 30 March 2020 - 11:34 PM

Hi, that's strange, I wonder what it's putting in there. @jachin99 have you seen this before?

 

Does it put images into the database?



#3 jachin99 OFFLINE  

jachin99

    Advanced Member

  • Members
  • 815 posts
  • Local time: 09:04 AM

Posted 31 March 2020 - 12:37 AM

I'll have to look when I get a chance. My clients are all windows but I haven't run out of space on my one machine with 18.6 on it

#4 luziferius OFFLINE  

luziferius

    Newbie

  • Members
  • 7 posts
  • Local time: 03:04 PM
  • LocationGermany

Posted 31 March 2020 - 06:12 AM

The network itself is almost idle. And the WAL grows even for films without fan art. So it doesn’t look like it stores images, as I’d see that on the network monitor.

 

I wanted to test what happens if i set the journal mode to the previous-default mode by executing

$ sqlite3 MyVideos116.db  <<< "PRAGMA journal_mode = 'delete';"

Unfortunately, this is automatically reverted to WAL mode upon trying to start a synchronization, so I can’t test if it only manifests the issues in WAL mode…

Needs further investigation…



#5 Luke OFFLINE  

Luke

    System Architect

  • Administrators
  • 156872 posts
  • Local time: 09:04 AM

Posted 31 March 2020 - 12:37 PM

@sualfred do you know if the database holds the contents of images? That's the only thing I can think of that would cause this.



#6 jachin99 OFFLINE  

jachin99

    Advanced Member

  • Members
  • 815 posts
  • Local time: 09:04 AM

Posted 31 March 2020 - 01:19 PM

I'm not sure where the write ahead log is located on Windows so I peeked around my userdata folder in kodi under USERNAME/appdata/roaming/kodi.  The total size of that folder is 3.17gb, and I have a few hundred files in the movie db alone most of which are bluray.  This PC is on kodi 18.6, and E4k current stable. 


Edited by jachin99, 31 March 2020 - 01:20 PM.


#7 luziferius OFFLINE  

luziferius

    Newbie

  • Members
  • 7 posts
  • Local time: 03:04 PM
  • LocationGermany

Posted 31 March 2020 - 03:52 PM

SQlite holds it’s files together. So the WAL file is always alongside the database file. It only exists during the lifetime of a database connection, it is removed when Kodi is closed.

 

According to https://kodi.wiki/view/Databases and https://kodi.wiki/view/Userdata, under Windows it is located at:

%APPDATA%\Kodi\userdata\Database

On my main system, the database itself is ~20MiB large.

 

Edit:

 

I did the research that I should have done in the first place (as always…)

 

This issue seems to be caused by my library layout:

I kind of dislike redundancy. I didn’t want to have this directory structure when I initially set up my library years ago:

Movie1/Movie1.mkv
Movie2/Movie2.mkv
Movie3/Movie3.mkv
Movie4/Movie4.mkv

So I built a flat structure. A single directory that simply contains all movie files. Simple and easy to browse and access using file managers.

It works fine with Kodi. But it seems that Emby actually requires the structure above, with each file inside it’s own directory.

 

 

The flat structure causes each movie to be reported with the backdrop of each movie. So this is a simple quadratic term, all of my 400 movies are reported to have each ~2500 backdrops. This is placed in the Kodi database by the Emby plugin, causing the database to explode during import.

 

Here is a Kodi debug log of the first 2% of the import process: Attached File  kodi.7z   1.07MB   0 downloads Beware, this is a 7zip compressed log file, it expands to about 1.5GiB in size if extracted, so your antivirus software might report a ZIP bomb.

 

You can see these JSON response objects in the log (truncated to not overshoot the maximum forum post length):

Spoiler

 

Edit 2:

 

Yep. It seems to add all files contained in the "extrafanarts" directory (that the Kodi extrafanart downloader plugin placed there) as a backdrop for each movie in the database. That directory contains ~2500 files, which fits the observed data.

 

So a proposed fix: Limit the number of certain elements when gathering data. Like limit database entries to have at most X (~ 10 - 100) backdrops.


Edited by luziferius, 01 April 2020 - 05:24 AM.





0 user(s) are reading this topic

0 members, 0 guests, 0 anonymous users