Music library scanning method

[make_nz 4]

On 11/17/2022 at 5:56 PM, Luke said:

Hi, the upcoming 4.8 server release will have improved initial library scan time.

And also comments over here:

I admit I had left emby back in late 2019 - for that slow scan issue -- and then gave 4.8 a chance, but was no game changer.

So here I'm in 2024, gone through some hoops to get Emby BETA 4.9.0.11 running on TrueNAS Core, but things are not flying.
 

Quote

	Command line: /usr/local/lib/emby-server/system/EmbyServer.dll -os freebsd -ffdetect /usr/local/lib/emby-server/bin/ffdetect -ffmpeg /usr/local/lib/emby-server/bin/ffmpeg -ffprobe /usr/local/lib/emby-server/bin/ffprobe -programdata /var/db/emby-server -updatepackage emby-server-freebsd13_{version}_amd64.pkg -noautorunwebapp
	Operating system: FreeBSD 13.1-RELEASE-p9 FreeBSD 13.1-RELEASE-p9 n245429-296d095698e TRUENAS
	Framework: .NET 8.0.0
	OS/Process: x64/x64
	Runtime: usr/local/lib/emby-server/system/System.Private.CoreLib.dll
	Processor count: 2
	Data path: /var/db/emby-server
	Application path: /usr/local/lib/emby-server/system

As of writing -- and I guess long before finished -- it's scanning 45min by now

On the other hand I also just compiled Logitech Media Server(LMS) from ports.
(for those who don't know, it's more a server/client system, with hardware endpoints like Squezebox Radios, not so much a "streaming to app" system)

I only have on data source (_Music_HiRes) I use and its my CD collection, with many compilations of 2, 3,4 sometimes more discs and it's the same for Emby and LMS.
The entire scan on LMS is done:
 

Quote

 Status

Logitech Media Server Version: 8.4.0 - TRUNK @ UNKNOWN

Hostname: LMS

Server IP Address: xxx

Server HTTP Port Number: 9000

Operating system: FreeBSD - EN - utf8

Platform Architecture: amd64-freebsd

Database Version: SQLite

Total Players Recognized: 0

Perl and Module Versions

Perl Version: 5.36.3 - amd64-freebsd-thread-multi

Audio::Scan: 1.09

DBD::SQLite: 1.58 (sqlite 3.22.0)

IO::Socket::SSL: 2.085

Mozilla::CA:

Net::SSLeay: 1.94 - OpenSSL 1.1.1t-freebsd 7 Feb 2023

Library Statistics

Total Tracks: 17,916

Total Albums: 775

Total Artists: 5,671

Total Genres: 161

Total Playlists: 0

Total Playing Time: 1140:17:26

Media Scan Details

Discovering files/directories: /media/share/_Music_HiRes   (19510 of 19510)   Complete  00:00:44

---

Scanning new music files: /media/share/_Music_HiRes   (17888 of 17888)   Complete  00:11:35

---

Building full text index   (6 of 6)   Complete  00:00:12

---

Find Updated Coverart Files   (1 of 1)   Complete  00:00:00

---

Pre-caching Artwork   (765 of 765)   Complete  00:00:46

---

Database Optimize   (2 of 2)   Complete  00:00:03

---

The server has finished scanning your media library.
Total Time: 00:13:20 (Friday, 22. March 2024 / 09:01)

It also has the Total Artists: 5,671 because of all those compilations. Actual Album Artists are 347

In a way I'm asking, if it is time to maybe add a different method to scan Librarys?
One the user can simply select with "Just use my Tags and Images - no extra Metadata - Fast Scan/Less Metadata"

Or, as Emby is what it is now, give an option "Don't collect Metadata for Artists in compilations - Fast Scan" (if this would help to scan faster)

When you buy a compilation you get a cover and back you remember, not a booklet with 50+ pages for every Artist on that compilation AND that's just fine in my opinion!

My thoughts so far and I will add how the initial scan went.

Cheers

Edited March 22 by make_nz

[Luke 37066]

7 hours ago, make_nz said:

And also comments over here:

I admit I had left emby back in late 2019 - for that slow scan issue -- and then gave 4.8 a chance, but was no game changer.

So here I'm in 2024, gone through some hoops to get Emby BETA 4.9.0.11 running on TrueNAS Core, but things are not flying.
 

As of writing -- and I guess long before finished -- it's scanning 45min by now

On the other hand I also just compiled Logitech Media Server(LMS) from ports.
(for those who don't know, it's more a server/client system, with hardware endpoints like Squezebox Radios, not so much a "streaming to app" system)

I only have on data source (_Music_HiRes) I use and its my CD collection, with many compilations of 2, 3,4 sometimes more discs and it's the same for Emby and LMS.
The entire scan on LMS is done:
 

It also has the Total Artists: 5,671 because of all those compilations. Actual Album Artists are 347

In a way I'm asking, if it is time to maybe add a different method to scan Librarys?
One the user can simply select with "Just use my Tags and Images - no extra Metadata - Fast Scan/Less Metadata"

Or, as Emby is what it is now, give an option "Don't collect Metadata for Artists in compilations - Fast Scan" (if this would help to scan faster)

When you buy a compilation you get a cover and back you remember, not a booklet with 50+ pages for every Artist on that compilation AND that's just fine in my opinion!

My thoughts so far and I will add how the initial scan went.

Cheers

Hi, if that’s  what you want then just disable internet metadata fetchers on the library and that’s what it will do.

[Happy2Play 8282]

That is what I do on an initial rebuild when I had to.  Disable all providers per library (yes a little tedious) to allow only existing metadata and images to be used.

Or from a Music standpoint if you have all your Music in perfect structure Artist/Ablum/track and option set on library you can rely on existing folder image.  But from a Artist standpoint you are stuck as multi-artist album will not have images/metadata as they can not be disabled.  You would have to strip all additional artist from track tags for that.

[make_nz 4]

Quote

disable internet metadata fetchers

You know this and I do, too. But this what I mean by having a simple  "Just use my Tags and Images - no extra Metadata - Fast Scan/Less Metadata" - as a ONE CLICK option.
In times I have had 12-15 Music sources set up and going through the set up and unticking check boxes… a plain No, thanks I got good tags would be great.

Now, with all fetchers turned of, here is the result for that very same data source (_Music_HiRes):

Scan media library
Last ran 2 seconds ago, taking 1 hour 14 minutes and 9 seconds.
Scans your media library to check for new and updated files.

Above LMS: Total Time: 00:13:20 (Friday, 22. March 2024 / 09:01)

So Emby still is about 6 × slower

The above is my tiny data source on the same hardware(TrueNAS Core), no network NFS/SMB or so.
My biggest on this hardware is about 80 × larger, plus some in between and some SMB shares from other Computers on the LAN,
for which I would love a much faster scanner in Emby, but I know local music libs are not super popular these days.

[Luke 37066]

1 minute ago, make_nz said:

You know this and I do, too. But this what I mean by having a simple  "Just use my Tags and Images - no extra Metadata - Fast Scan/Less Metadata" - as a ONE CLICK option.
In times I have had 12-15 Music sources set up and going through the set up and unticking check boxes… a plain No, thanks I got good tags would be great.

Now, with all fetchers turned of, here is the result for that very same data source (_Music_HiRes):

Scan media library
Last ran 2 seconds ago, taking 1 hour 14 minutes and 9 seconds.
Scans your media library to check for new and updated files.

Above LMS: Total Time: 00:13:20 (Friday, 22. March 2024 / 09:01)

So Emby still is about 6 × slower

The above is my tiny data source on the same hardware(TrueNAS Core), no network NFS/SMB or so.
My biggest on this hardware is about 80 × larger, plus some in between and some SMB shares from other Computers on the LAN,
for which I would love a much faster scanner in Emby, but I know local music libs are not super popular these days.

Hi, if you can attach a server log then we might be able to provide some suggestions on how to speed up your scans.

[make_nz 4]

Upload right here? Or somewhere else?
Thanks

[Luke 37066]

3 minutes ago, make_nz said:

Upload right here? Or somewhere else?
Thanks

Since you’re already here this is fine. We can always extract and move to a separate topic if need be.

[make_nz 4]

here it is

embyserver-63846740990.txt

[make_nz 4]

Hmm, bigger issue -- no not my custom color CSS -- nothing is showing up after the scan

[make_nz 4]

BTW
This is the entire history from that jail
 

root@embyBETA:~ # history
     1  19:15   pkg update && pkg upgrade
     2  19:15   pkg update && pkg upgrade
     3  19:16   pkg install mc
     4  19:17   mc -u
     5  19:17   pkg update && pkg upgrade
     6  19:21   pkg install dotnet
     7  19:23   pkg install libass fontconfig freetype2 mbedtls fribidi gnutls iconv opus sqlite3 libtheora libva ocl-icd libvorbis webp libzvbi libraw ImageMagick6 tesseract intel-media-sdk libunwind libinotify openssl icu krb5 libexif libjxl libheif libimagequant openexr openjpeg orc pango tiff
     8  19:31   pkg add -f https://github.com/MediaBrowser/Emby.Releases/releases/download/4.9.0.11/emby-server-freebsd13_4.9.0.11_amd64.pkg
     9  19:33   sysrc emby_server_enable="YES"
    10  19:33   service emby-server status
    11  19:33   service emby-server start
    12  19:33   service emby-server status
    13  21:49   service emby-server status
    14  21:49   service emby-server restart
    15  21:50   service emby-server status
    16  22:01   mc -u
    17  22:08   pkg install htop
    18  22:08   htop
    19  22:15   service emby-server status
    20  22:15   history
root@embyBETA:~ #

root@embyBETA:~ # service emby-server status
emby_server is running as pid 51019.

Edited March 22 by make_nz

[Happy2Play 8282]

So library shows in library setup but not on Home Screen.  There have been a couple topics like this now.

Scan shows it probing the media.

Any applied parental controls/access?

Does it appear for a different user?

[make_nz 4]

2 minutes ago, Happy2Play said:

Any applied parental controls/access?

Does it appear for a different user?

None, only me as it's for testing

[Happy2Play 8282]

11 minutes ago, make_nz said:

None, only me as it's for testing

And if you created another user do they appear as for some reason this has happened to someone else.

[make_nz 4]

Emby drives server to 100% as soon as start doing something on the web interface.
I need to kill proc… restart service just hangs
 

root@embyBETA:~ # service emby-server restart
Stopping emby_server.
Waiting for PIDS: 52403

Edited March 22 by make_nz

[make_nz 4]

Will shut down Server and myself now -- different timezone

[make_nz 4]

Up and running again…
It seems to be USER related as the initial user emby now has the home screen populated, but the 2nd test user make has an empty home screen.
 

[make_nz 4]

18 hours ago, Happy2Play said:

That is what I do on an initial rebuild when I had to.  Disable all providers per library (yes a little tedious) to allow only existing metadata and images to be used.

Or from a Music standpoint if you have all your Music in perfect structure Artist/Ablum/track and option set on library you can rely on existing folder image.  But from a Artist standpoint you are stuck as multi-artist album will not have images/metadata as they can not be disabled.  You would have to strip all additional artist from track tags for that.

Yes, you have clearly been there!

My case is a bit extreme as I have huge data sets and many different  places where they live. I don't just do backups on some drives and put them in a drawer, I attach them do RBpis, Kodi, Android… and actually listen to the music in many places far away from my network.

But as for dealing with them in Emby a plain No, thanks I got good tags would be great, even as a general settings option I would like it very much.

And here is a great example why I wanna see my Tags and Images only -- something is missing and I rather fix it in my data source, then have a fetcher pull something.
It's very often the case, that covers vary from region to region and all auto-taggers I have used have pulled images which were not the actual CD cover I was holding in my hands.
And yes I have always the CD right next to me, when ripping and add the only this cover to tags and album folder, but here I missed placing one in the folder:
 

Edited March 23 by make_nz

[ebr 14913]

Will a moderator please split all this out?  Thanks.

[GrimReaper 3294]

7 hours ago, ebr said:

Will a moderator please split all this out?  Thanks.

Split.

[make_nz 4]

On 3/22/2024 at 9:53 PM, Luke said:

Since you’re already here this is fine. We can always extract and move to a separate topic if need be.

@GrimReaperThis is where I'd say it can be split and some parts deleted in each split. 

[make_nz 4]

On 3/23/2024 at 2:04 PM, make_nz said:

Up and running again…
It seems to be USER related as the initial user emby now has the home screen populated, but the 2nd test user make has an empty home screen.
 

@LukeNot the perfect place as off topic, but as it is not resolved with
emby-server-freebsd13_4.9.0.12_amd64.pkg
I'll add it anyway, as it came up here and you can still move it.


 

[make_nz 4]

On a newly created user (4.9.0.12_amd64.pkg) it is working.

[Happy2Play 8282]

And if you toogle parental controls on to any value, hit save, then off and hit save does the library appear?

[make_nz 4]

19 minutes ago, Happy2Play said:

And if you toogle parental controls on to any value, hit save, then off and hit save does the library appear?

Yes, it does now. +1 for this bug fixed, thanks.

On this topic

Quote

Cumulative changes since stable release:

• Improve library scan performance

May I ask, if I can expect a significant boost on my CD collection?
And how do I make sure I clean out everything from previous scans, in order to get real first run, please?
(As it is in a jail, I tent to revert to initial snapshot, with just the OS. But it takes time to set up everything again then)

Edited April 2 by make_nz

[Happy2Play 8282]

2 minutes ago, make_nz said:

19 minutes ago, Happy2Play said:

And if you toogle parental controls on to any value, hit save, then off and hit save does the library appear?

Yes, it does now. +1 for this bug fixed, thanks.

Yes this is an area @Lukeneeds to look as there have been a few topics with this disconnect on permissions and libraries.  I happened to figure it out when I tried to create a new database that did not go so well as everything is database only now.