Add new music files, Emby scans the whole music library, and fails ...

[jiggity 35]

Emby Server 4.8.3.0

HI,

This has been a peeve of mine for a while now. when i add new music to the library, Emby scans the entire music library, resulting in a log full of musicbrainz and discogs errors.

musicbrainz is constantly outputting bad request errors, a lot for items I had to spoof musicbranz album ids on, and discogs is always erroring for toomanyrequests, and there is nothing I can do about that.  it is an API issue.

Is there anything that can be done to hasten these scans and cut back on redundant scanning and/or reporting? Or have better identification of music files to prevent merging and this meta errors issues. Spoofing doesn't seem to be the solution, and the non-stop errors is causing more problems than the solution solved.

I have close to an hour of non stop errors because I added 10 new LPs?
Then there is the peeve where each imported LP doesn't use the embedded or included folder images until after the meta data scans, even when I have the library set to not scan meta providers for album image.
 

2024-03-07 11:52:22.386 Info HttpClient: GET https://musicbrainz.emby.tv/ws/2/release/10658506?inc=url-rels
2024-03-07 11:52:22.472 Error App: Error in MusicBrainz
	*** Error Report ***
	Version: 4.8.3.0
	Command line: /system/EmbyServer.dll -programdata /config -ffdetect /bin/ffdetect -ffmpeg /bin/ffmpeg -ffprobe /bin/ffprobe -restartexitcode 3
	Operating system: Linux version 6.1.74-Unraid (root@Develop-612) (gcc (GCC) 12.2.0, GNU ld version 2.40-slack151) #1 SMP PREEMPT_DYNAMIC Fri Feb  2 11:06:32 PST 2024
	Framework: .NET 6.0.25
	OS/Process: x64/x64
	Runtime: system/System.Private.CoreLib.dll
	Processor count: 24
	Data path: /config
	Application path: /system
	MediaBrowser.Model.Net.HttpException: MediaBrowser.Model.Net.HttpException: BadRequest
	   at Emby.Server.Implementations.HttpClientManager.CoreHttpClientManager.SendAsyncInternal(HttpRequestOptions options, String httpMethod)
	   at Emby.Server.Implementations.HttpClientManager.CoreHttpClientManager.SendAsync(HttpRequestOptions options, String httpMethod)
	   at MusicBrainz.MusicBrainzAlbumProvider.GetMusicBrainzResponse(String url, Boolean forceMusicBrainzProper, CancellationToken cancellationToken)
	   at MusicBrainz.MusicBrainzAlbumProvider.GetMetadata(AlbumInfo searchInfo, CancellationToken cancellationToken)
	   at Emby.Providers.Manager.MetadataService`2.ExecuteRemoteProviders(MetadataResult`1 temp, LibraryOptions libraryOptions, String logName, TIdType id, IRemoteMetadataProvider`2[] providers, MetadataRefreshOptions options, CancellationToken cancellationToken)
	Source: Emby.Server.Implementations
	TargetSite: Void MoveNext()
	
2024-03-07 11:52:22.772 Info HttpClient: GET https://api.discogs.com/releases/249086
2024-03-07 11:52:22.822 Error App: Error in Discogs
	*** Error Report ***
	Version: 4.8.3.0
	Command line: /system/EmbyServer.dll -programdata /config -ffdetect /bin/ffdetect -ffmpeg /bin/ffmpeg -ffprobe /bin/ffprobe -restartexitcode 3
	Operating system: Linux version 6.1.74-Unraid (root@Develop-612) (gcc (GCC) 12.2.0, GNU ld version 2.40-slack151) #1 SMP PREEMPT_DYNAMIC Fri Feb  2 11:06:32 PST 2024
	Framework: .NET 6.0.25
	OS/Process: x64/x64
	Runtime: system/System.Private.CoreLib.dll
	Processor count: 24
	Data path: /config
	Application path: /system
	MediaBrowser.Model.Net.HttpException: MediaBrowser.Model.Net.HttpException: TooManyRequests
	   at Emby.Server.Implementations.HttpClientManager.CoreHttpClientManager.SendAsyncInternal(HttpRequestOptions options, String httpMethod)
	   at Emby.Server.Implementations.HttpClientManager.CoreHttpClientManager.SendAsync(HttpRequestOptions options, String httpMethod)
	   at Discogs.Plugin.GetResponse(HttpRequestOptions request, String method)
	   at Discogs.Plugin.GetDeserializedResponse[T](HttpRequestOptions request, String method)
	   at Discogs.DiscogsAlbumProvider.GetMetadataResponse(AlbumInfo id, CancellationToken cancellationToken)
	   at Discogs.DiscogsAlbumProvider.GetMetadata(AlbumInfo id, CancellationToken cancellationToken)
	   at Emby.Providers.Manager.MetadataService`2.ExecuteRemoteProviders(MetadataResult`1 temp, LibraryOptions libraryOptions, String logName, TIdType id, IRemoteMetadataProvider`2[] providers, MetadataRefreshOptions options, CancellationToken cancellationToken)
	Source: Emby.Server.Implementations
	TargetSite: Void MoveNext()

 

[Luke 40079]

Hi there, please attach the complete emby server log. thanks.

[jiggity 35]

@LukeSure thing
Attached is today's log file

embyserver.txt

[Luke 40079]

OK I'm not sure yet what the issue is with musicbrainz, but we'll increase the throttle between requests to Discogs and that should resolve the too many requests error.

[jiggity 35]

On 28/03/2024 at 22:47, Luke said:

OK I'm not sure yet what the issue is with musicbrainz, but we'll increase the throttle between requests to Discogs and that should resolve the too many requests error.

My scans are now very much improved, and I am getting things that didn't identity appearing in the library now

looping back to

I set my scan library to a hard limit of 1 hour, and that really gives me a much smaller log to go through
I had a moment this weekend to look at the errors Muscbrainz is throwing and the specifics of them. It looks like a lot, for me right now, are related to the issue linked above. where the MBZ plugin will import the discogs release ID info, and that will be incorrect when queried against the Discogs plug in.

It is causing a double whammy, imo.
MBZ imports and writes bad info, then discogs tries to look that info up
I know you said you wanted to look into writing the embedded DISCOGS_RELEASE_ID tag, can you also look into seeing if the MBZ plug can NOT write discogs info? That seems to be the source of many problems.

Can there be a discussion about order of operations for lookups, I mean embedded vs online metadata,  and the info that is read or write for plug ins? I feel like this is something that will need community feedback to overhaul.

[jiggity 35]

@Luke
Has something changed with the Discogs plugin again?

I am back to having hours upon hours of too many requests errors in the logs and that is impeding Emby from doing any other scans
Today's logs have 6 hours so far of Discogs Errors.

Edited May 6, 2024 by jiggity

[Luke 40079]

3 hours ago, jiggity said:

@Luke
Has something changed with the Discogs plugin again?

I am back to having hours upon hours of too many requests errors in the logs and that is impeding Emby from doing any other scans
Today's logs have 6 hours so far of Discogs Errors.

Hi, can you please attach a new server log example? Thanks.

[jiggity 35]

1 minute ago, Luke said:

Hi, can you please attach a new server log example? Thanks.

embyserver-63850591796.txtThis is less than 12 hours of today

[Happy2Play 9441]

And are we rate limited to 25 requests a minute?   @Luke

Home - Discogs API Documentation

But see provide rate limits like this one a big issue for large libraries even if it is a one and done query.

[jiggity 35]

Part of the problem is the way look ups are performed, imo.

If the DISCOGS_RELEASE_ID was read from the embedded tag to the metadata manager, then that was used for the lookup, you would save on the redundancies.
Currently the plugin queries to discogs up to 5 times for the same file, for Artist, for Album, for Art, for Artist info, and for Album Info. Each is done as a separate request.  With the Release ID all that info can be obtained in one request, as far as I understand.

For any file that does not have any discogs embedded tags do not query them against discogs unless a metadata fill in missing info request is made by the user.

Similar goes for MBZ, with the caveat that MBZ should not write discogs info.

[Happy2Play 9441]

I would have to look but don't believe there is a one and done here.

Have you actually looked at releaseid info as it will not return all Artist info as it is about the release not the individual same with every provider there is.  It will provide api links requiring additional requests with provided known ids.  So there is no one payload with all info.

api.discogs.com/releases/788491

So at least 2 request Artist and Album (from a known info standpoint) in addition to all image request.

[jiggity 35]

17 hours ago, Happy2Play said:

I would have to look but don't believe there is a one and done here.

Have you actually looked at releaseid info as it will not return all Artist info as it is about the release not the individual same with every provider there is.  It will provide api links requiring additional requests with provided known ids.  So there is no one payload with all info.

api.discogs.com/releases/788491

So at least 2 request Artist and Album (from a known info standpoint) in addition to all image request.

I interface with the Discogs APi through MP3 tag on the regular, and calling with the Release ID will get the exact release with all the details, whereas doing a call with just the artist name and/or the album title will only return a list of results you need to refine down to select the exact release from. It is the same way the front end of discogs works.


How is it that Emby is getting such limited, or poorly defined info on an api call?

 

[Luke 40079]

What do you mean by limited or poorly defined? What info is limited?

[jiggity 35]

9 hours ago, Luke said:

What do you mean by limited or poorly defined? What info is limited?

What happy showed looked like a scrape of the website, it was limited in info compared to what i can pull through mp3tag, as a comparison

[Luke 40079]

11 hours ago, jiggity said:

What happy showed looked like a scrape of the website, it was limited in info compared to what i can pull through mp3tag, as a comparison

Limited how?

[jiggity 35]

11 hours ago, Luke said:

Limited how?

Did you read the conversation? go back to 

The discogs plug sucks, it does too many lookups and creates its own error loops.
The musicbrainz plug imports bad discogs data and causes a garbage in garbage out error loop

The ingestion of music doesn't import embedded data to the metadata manger, the plugs don't look up things from that embedded data

The never ending bad lookups continue

Discogs should only be queried for Album info based on the ReleaseID
If there is no releasID, there should be no Discogs query
The difference between what I get with a release ID api call and what Happy showed is the details I get are the required ones, and your API is a scrape of the website, with extraneous info that is useless, plus you do not get the required info.

Look at the examples provided.

[Luke 40079]

I understand that. It just sounded different when you used the wording of limited info.