Artist view very slow after update 4.8.0.80

[letterman 40]

After update to 4.8.0.80 browsing in music library is very slow in general.

But even more slow is the artist view. It takes minutes to load (directly on the server, embyserver.exe is in full use in TaskManager). Using the app on iphone or ipad, only the name, the wiki information and the about section is shown. Songs, Albums, "more like this does" not appear (seems to be a timeout).

Browsing in movies is fine. Only music library is the problem.

It seems, that emby needs extreme time to collect the data from the database.

My library.db is 2427 MB, Cache 4196 MB. I did made full database scan and vacuum the database, Windows 16 GB memory.

What can we do to solve this problem? With this speed using emby is not possible anymore unfortunately

[pgriffith 88]

I don't think this is 4.8.0.80 is the cause of this. My artists page loads virtually instantly for me, it contains 2098 items.

Everything in music is superfast for me to load.

[letterman 40]

(1)
It was fast before the update, so there has been changed someting. Of course I can send logs to analyse.
Maybe my database is corrupt somehow, but how can I really check/analyse this? It is impossible for me to build a complete new setup/database, because I have lots of individuell settings/pictures/collections etc. 

(2)
To find a solution, I run integrity check of library.db. ok.
Despite that, I nonetheless recovered the DB (according to instructions https://emby.media/support/articles/Corrupt-Database.html).

Parsing errors:
Parse error near line 8: object name reserved for internal use: sqlite_sequence
Parse error near line 179: object name reserved for internal use: sqlite_stat1
Parse error near line 199: table sqlite_master may not be modified
Parse error near line 11868529: no such table: sqlite_stat1
Parse error near line 11868530: no such table: sqlite_stat1
Parse error near line 11868531: no such table: sqlite_stat1
Parse error near line 11868532: no such table: sqlite_stat1
Parse error near line 11868533: no such table: sqlite_stat1
Parse error near line 11868534: no such table: sqlite_stat1
Parse error near line 11868535: no such table: sqlite_stat1
Parse error near line 11868536: no such table: sqlite_stat1
Parse error near line 11868537: no such table: sqlite_stat1
Parse error near line 11868538: no such table: sqlite_stat1
Parse error near line 11868539: no such table: sqlite_stat1
Parse error near line 11868540: no such table: sqlite_stat1
Parse error near line 11868541: no such table: sqlite_stat1
Parse error near line 11868542: no such table: sqlite_stat1
Parse error near line 11868543: no such table: sqlite_stat1
Parse error near line 11868544: no such table: sqlite_stat1
Parse error near line 11868545: no such table: sqlite_stat1
Parse error near line 11868546: no such table: sqlite_stat1
Parse error near line 11868547: no such table: sqlite_stat1
Parse error near line 11868548: no such table: sqlite_stat1
Parse error near line 11868549: no such table: sqlite_stat1
Parse error near line 11868550: no such table: sqlite_stat1
Parse error near line 11868551: no such table: sqlite_stat1
Parse error near line 11868552: no such table: sqlite_stat1
Parse error near line 11868553: no such table: sqlite_stat1
Parse error near line 11868554: no such table: sqlite_stat1
Parse error near line 11868555: no such table: sqlite_stat1
Parse error near line 11868556: no such table: sqlite_stat1
Parse error near line 11868557: no such table: sqlite_stat1
Parse error near line 11868558: no such table: sqlite_stat1
Parse error near line 11868559: no such table: sqlite_stat1
Parse error near line 11868560: no such table: sqlite_stat1
Parse error near line 11868561: no such table: sqlite_stat1
Parse error near line 11868562: no such table: sqlite_stat1

BUT integrity check of new DB: ok

I changed DB and started EMBY.
Nothing happens. Not even a log file is created. Just the embytray.exe is running.

(3)
I started debugging to find out, why it takes so long. I can show whole debug file, if needed.
Unusual seems to be the following lines (more of these in the log):

2024-02-03 16:47:33.182 Debug ImageProcessor: Getting image size for item Audio C:\XXXXXXX\metadata\library\cd\cd1c1ef26bfc7aba375dbf36c0d4fa41\poster.jpg
2024-02-03 16:47:33.184 Debug Server: http/1.1 Response 200 to ‌‍‍::1‌. Time: 2103ms. GET http://‌‍‍localhost‌:XXXX/emby/Users/XXXXX/Items?IncludeItemTypes=MusicVideo&Limit=12&SortBy=SortName&Fields=PrimaryImageAspectRatio,ProductionYear&SortOrder=Ascending&Recursive=true&CollapseBoxSetItems=false&ArtistIds=1055921&ImageTypeLimit=1&EnableTotalRecordCount=false&X-Emby-Client=Emby Web&X-Emby-Device-Name=Chromium Windows&X-Emby-Device-Id=XXXXX-Emby-Client-Version=4.8.0.80&X-Emby-Token=‌XXXXXXX-Emby-Language=en-us 

2024-02-03 16:48:05.592 Debug ImageProcessor: Getting image size for item Audio C:\XXXXXX\metadata\library\43\43839cbf3494efe444fde032e88435e1\poster.jpg
2024-02-03 16:48:05.593 Debug Server: http/1.1 Response 200 to ‌‍‍::1‌. Time: 34513ms. GET http://‌‍‍localhost‌:XXXX/emby/Users/XXXXX/Items?IncludeItemTypes=MusicAlbum&Recursive=true&SortBy=ProductionYear,SortName&SortOrder=Descending&Fields=BasicSyncInfo,CanDelete,PrimaryImageAspectRatio,ProductionYear&Limit=12&ContributingArtistIds=1055921&X-Emby-Client=Emby Web&X-Emby-Device-Name=Chromium Windows&X-Emby-Device-Id=XXXX-Emby-Client-Version=4.8.0.80&X-Emby-Token=‌XXXXXXX-Emby-Language=en-us
2024-02-03 16:48:05.606 Debug ImageProcessor: ImageHeader.GetDimensions succeeded for C:\XXXXXXX\metadata\library\43\43839cbf3494efe444fde032e88435e1\poster.jpg

Why are there these high response times? The required poster.jpg files do exist.
Can I just delete my library-folder incl. subfolders trying to solve this problem?

Edited February 3, 2024 by letterman

[Luke 42079]

HI, please attach the complete emby server log. Thanks.

[letterman 40]

@LukeI sent you a PM with the log file

Edited February 4, 2024 by letterman

[letterman 40]

I do still make lots of researches to my problem, because I think it must be a bug in Emby, which needs to be solved. Can anyone here assess whether this link could have something to do with it? Jellyfin is based on Emby, so it might be a big step to a solution: https://github.com/jellyfin/jellyfin/issues/7206

[Luke 42079]

How does the albums tab compare?

[letterman 40]

Somehow same same. Click an an album, it takes lots of time until the songs and more like that appears. I can send you debug file, too if you want. But I think it is the same problem with response time of database query like I told above. 

[Luke 42079]

How about the albums tab?

[letterman 40]

What do you mean? Same like my last answer

[letterman 40]

Hi Luke, unfortunately I did not get an answer yet. Hope you will analyze my logfile soon.
 

To advance on this topic, I have rebuilt the entire database on a new system (Z2 Tower G4 Intel i7-8700K 6x 3.7GHz 64GB 1TB SSD Quadro P2000/5GB) from scratch. Nothing was imported. The library.db remains 2.5 GB in size (same like before). Notably, database queries still take an extremely long time.

Excerpt from the debug logfile:

2024-02-28 21:35:05.603 Debug SqliteItemRepository: ItemsService.GetItems.GetItems query time (slow 6x): 4882ms. Query: select count(*) OVER() AS TotalRecordCount,A.Id,A.IndexNumber,A.Name,A.Path,A.ParentIndexNumber,A.RunTimeTicks,A.ParentId,A.Album,A.AlbumId,A.Images,UserDatas.IsFavorite,UserDatas.Played,UserDatas.PlaybackPositionTicks,(Select ShareLevel from UserItemShares join AncestorIds2 on AncestorIds2.AncestorId=UserItemShares.ItemId where UserItemShares.UserId=1 and UserItemShares.ShareLevel not null and AncestorIds2.ItemId=A.Id order by Distance limit 1) as ShareLevel from MediaItems A Join ItemLinks2 ItemLinksFilters on ItemLinksFilters.ItemId=A.Id and ItemLinksFilters.LinkedId=984366 left join UserDatas on A.UserDataKeyId=UserDatas.UserDataKeyId And UserDatas.UserId=1 where A.Type=11 AND Coalesce(ShareLevel, 0) > 0 AND A.ExtraType is null Group by A.PresentationUniqueKey ORDER BY UserDatas.PlayCount DESC,A.SortName collate NATURALSORT ASC LIMIT 16

and

2024-02-28 21:35:18.414 Debug Server: http/1.1 Response 200 to ::1. Time: 17697ms. GET http://localhost:xxxx/emby/Users/343434/Items?IncludeItemTypes=MusicAlbum&Recursive=true&SortBy=ProductionYear,SortName&SortOrder=Descending&Fields=BasicSyncInfo,CanDelete,PrimaryImageAspectRatio,ProductionYear&Limit=16&ContributingArtistIds=984366&X-Emby-Client=Emby Web&X-Emby-Device-Name=Google Chrome Windows&X-Emby-Device-Id=23232323-Emby-Client-Version=4.8.1.0&X-Emby-Token=x_secret1_x&X-Emby-Language=en-us
(pls tell me, if you need the new logfile, too)

Since the update, there's definitely something off with the SQLite queries. Please check this, as for example, opening an artist's page currently takes about 30 seconds. On our older system (problem threat above), it takes over a minute. This cannot be the intention of Emby and it was much faster in the previous version for sure. Pls fix this bug!

 

[letterman 40]

To keep on going: of course, I did vacuum database and I deactivated all unnecessary plugins without any change. Database cache is 8192 MB. 
I found a nice advice for jellyfin users on github https://github.com/jellyfin/jellyfin/issues/7206 changing the configuration of the db.

PRAGMA main.page_size = 4096 (I think this parameter is analog to database cache within emby settings)
PRAGMA main.cache_size=10000
PRAGMA main.locking_mode=EXCLUSIVE
PRAGMA main.synchronous=NORMAL
PRAGMA main.journal_mode=WAL (I think this is same same to emby)
PRAGMA main.cache_size=5000
PRAGMA main.temp_store = MEMORY

Is it possible to implement these parameters somewhere in the startup.files, maybe an entry in system.xml? Or it it possible with another kind of script? Or to develop a plugin doing this during startup?

Edited February 29, 2024 by letterman

[Luke 42079]

We already match those exactly, have been for a long time. cache size you can set from the emby ui.

[Luke 42079]

Is the database on an ssd?

[letterman 40]

Yes. Z2 Tower G4 Intel i7-8700K 6x 3.7GHz 64GB 1TB SSD Quadro P2000/5GB All other programs are more then fast enough.

 

Edited February 29, 2024 by letterman

[letterman 40]

@LukeI give you all the information you need, so you can somehow fix this bug. I'm really desperate, because using emby (in detail for me primary listing to music and browing through the library and not watching movies) with this speed is not possible anymore. I know my library is large (about 1 Mio files, DB 2.5 GB), but that should not be the problem for a system of this speed (and it was better with the version before)

Edited February 29, 2024 by letterman

[adminExitium 355]

5 hours ago, Luke said:

those exactly

Even the synchronous=NORMAL & the temp_store=MEMORY settings? Just curious because those are not shown in the logs and they have helped in other sqlite-based programs.
If they ever become user-configurable, I would like to try it with synchronous set to OFF too.

[Luke 42079]

8 hours ago, adminExitium said:

Even the synchronous=NORMAL & the temp_store=MEMORY settings? Just curious because those are not shown in the logs and they have helped in other sqlite-based programs.
If they ever become user-configurable, I would like to try it with synchronous set to OFF too.

Yes those two are used.