Jump to content

Server is very slow at scanning for new content

Philip6666

By Philip6666
in Linux

 Share
Go to solution Solved by Neminem,

Recommended Posts

Philip6666

Philip6666 1

Posted
Posted

embyserver.txtUp until 2-3 weeks ago, everything worked perfectly with new content being available to watch almost immediately. Now, it can take days, even if i go to the specific show and press "Scan library files" or abort the library scan and start a new one. (It seems like the library scanning always get stuck at around 91%. 

I've changed both "Generate intro markers" and "Generate miniature images for video preview" to just run as scheduled task instead of scheduled task and when new media is added. But, it had little to no effect. 

Since the library scanning seems to get stuck around the same percentage each time, I wonder if there could be a specific file that's messed up.

I checked the logs, but the only errors I could find was some errors connecting to TMDB, but I don't really know what I'm looking at, so I've attached a log file

Any help or tips would be very appreciated. 

  • Solution
Neminem

Neminem 1518

Posted
  • Solution
Posted (edited)

It look like something is blocking your metadata providers.

OpenSubtitles.

Quote

Error searching for subtitles on Open Subtitles    *** Error Report ***
    Version: 4.8.11.0
    Command line: /opt/emby-server/system/EmbyServer.dll -programdata /var/lib/emby -ffdetect /opt/emby-server/bin/ffdetect -ffmpeg /opt/emby-server/bin/ffmpeg -ffprobe /opt/emby-server/bin/ffprobe -restartexitcode 3 -updatepackage emby-server-deb_{version}_amd64.deb
    Operating system: Linux version 6.8.12-11-pve (build@proxmox) (gcc (Debian 12.2.0-14+deb12u1) 12.2.0, GNU ld (GNU Binutils for Debian) 2.40) #1 SMP PREEMPT_DYNAMIC PMX
    Framework: .NET 6.0.36
    OS/Process: x64/x64
    Runtime: opt/emby-server/system/System.Private.CoreLib.dll
    Processor count: 4
    Data path: /var/lib/emby
    Application path: /opt/emby-server/system
    MediaBrowser.Model.Net.HttpException: MediaBrowser.Model.Net.HttpException: Connection to https://api.opensubtitles.com/api/v1/infos/languages timed out
     ---> System.Threading.Tasks.TaskCanceledException: The request was canceled due to the configured HttpClient.Timeout of 20 seconds elapsing.
     ---> System.TimeoutException: A task was canceled.
     ---> System.Threading.Tasks.TaskCanceledException: A task was canceled.
       at System.Threading.Tasks.TaskCompletionSourceWithCancellation`1.WaitWithCancellationAsync(CancellationToken cancellationToken)
       at System.Net.Http.HttpConnectionPool.GetHttp11ConnectionAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
       at System.Net.Http.HttpConnectionPool.SendWithVersionDetectionAndRetryAsync(HttpRequestMessage request, Boolean async, Boolean doRequestAuth, CancellationToken cancellationToken)
       at System.Net.Http.DiagnosticsHandler.SendAsyncCore(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
       at System.Net.Http.RedirectHandler.SendAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
       at System.Net.Http.DecompressionHandler.SendAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
       at System.Net.Http.HttpClient.<SendAsync>g__Core|83_0(HttpRequestMessage request, HttpCompletionOption completionOption, CancellationTokenSource cts, Boolean disposeCts, CancellationTokenSource pendingRequestsCts, CancellationToken originalCancellationToken)
       --- End of inner exception stack trace ---
       --- End of inner exception stack trace ---
       at System.Net.Http.HttpClient.HandleFailure(Exception e, Boolean telemetryStarted, HttpResponseMessage response, CancellationTokenSource cts, CancellationToken cancellationToken, CancellationTokenSource pendingRequestsCts)
       at System.Net.Http.HttpClient.<SendAsync>g__Core|83_0(HttpRequestMessage request, HttpCompletionOption completionOption, CancellationTokenSource cts, Boolean disposeCts, CancellationTokenSource pendingRequestsCts, CancellationToken originalCancellationToken)
       at Emby.Server.Implementations.HttpClientManager.CoreHttpClientManager.SendAsyncInternal(HttpRequestOptions options, String httpMethod)
       --- End of inner exception stack trace ---
       at OpenSubtitles.OpenSubtitleComDownloader.GetResponse(HttpRequestOptions requestOptions, String httpMethod, Int32 allowedRequestFrequencyMs)
       at OpenSubtitles.OpenSubtitleComDownloader.GetOpenSubtitlesLanguages(OpenSubtitleOptions options, CancellationToken cancellationToken)
       at OpenSubtitles.OpenSubtitleComDownloader.SearchInternal(SubtitleSearchRequest request, Boolean useFilename, OpenSubtitleOptions options, CancellationToken cancellationToken)
       at OpenSubtitles.OpenSubtitleComDownloader.Search(SubtitleSearchRequest request, OpenSubtitleOptions options, CancellationToken cancellationToken)
       at Emby.Providers.Subtitles.SubtitleManager.SearchSubtitles(SubtitleSearchRequest request, CancellationToken cancellationToken)
    Source: Emby.Server.Implementations
    TargetSite: Void MoveNext()
    InnerException: System.Threading.Tasks.TaskCanceledException: The request was canceled due to the configured HttpClient.Timeout of 20 seconds elapsing.
    Source: System.Net.Http
    TargetSite: Void HandleFailure(System.Exception, Boolean, System.Net.Http.HttpResponseMessage, System.Threading.CancellationTokenSource, System.Threading.CancellationToken, System.Threading.CancellationTokenSource)
       at System.Net.Http.HttpClient.HandleFailure(Exception e, Boolean telemetryStarted, HttpResponseMessage response, CancellationTokenSource cts, CancellationToken cancellationToken, CancellationTokenSource pendingRequestsCts)
       at System.Net.Http.HttpClient.<SendAsync>g__Core|83_0(HttpRequestMessage request, HttpCompletionOption completionOption, CancellationTokenSource cts, Boolean disposeCts, CancellationTokenSource pendingRequestsCts, CancellationToken originalCancellationToken)
       at Emby.Server.Implementations.HttpClientManager.CoreHttpClientManager.SendAsyncInternal(HttpRequestOptions options, String httpMethod)
    InnerException: System.TimeoutException: A task was canceled.
    Source: 
    TargetSite: 
    InnerException: System.Threading.Tasks.TaskCanceledException: A task was canceled.
    Source: System.Private.CoreLib
    TargetSite: Void ThrowForNonSuccess(System.Threading.Tasks.Task)
       at System.Threading.Tasks.TaskCompletionSourceWithCancellation`1.WaitWithCancellationAsync(CancellationToken cancellationToken)
       at System.Net.Http.HttpConnectionPool.GetHttp11ConnectionAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
       at System.Net.Http.HttpConnectionPool.SendWithVersionDetectionAndRetryAsync(HttpRequestMessage request, Boolean async, Boolean doRequestAuth, CancellationToken cancellationToken)
       at System.Net.Http.DiagnosticsHandler.SendAsyncCore(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
       at System.Net.Http.RedirectHandler.SendAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
       at System.Net.Http.DecompressionHandler.SendAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
       at System.Net.Http.HttpClient.<SendAsync>g__Core|83_0(HttpRequestMessage request, HttpCompletionOption completionOption, CancellationTokenSource cts, Boolean disposeCts, CancellationTokenSource pendingRequestsCts, CancellationToken originalCancellationToken)
    
 

TVDB

Quote

Error in TheTVDB    *** Error Report ***
    Version: 4.8.11.0
    Command line: /opt/emby-server/system/EmbyServer.dll -programdata /var/lib/emby -ffdetect /opt/emby-server/bin/ffdetect -ffmpeg /opt/emby-server/bin/ffmpeg -ffprobe /opt/emby-server/bin/ffprobe -restartexitcode 3 -updatepackage emby-server-deb_{version}_amd64.deb
    Operating system: Linux version 6.8.12-11-pve (build@proxmox) (gcc (Debian 12.2.0-14+deb12u1) 12.2.0, GNU ld (GNU Binutils for Debian) 2.40) #1 SMP PREEMPT_DYNAMIC PMX
    Framework: .NET 6.0.36
    OS/Process: x64/x64
    Runtime: opt/emby-server/system/System.Private.CoreLib.dll
    Processor count: 4
    Data path: /var/lib/emby
    Application path: /opt/emby-server/system
    MediaBrowser.Model.Net.HttpException: MediaBrowser.Model.Net.HttpException: Cancelling connection to https://api4.thetvdb.com/v4/login due to a previous timeout.
       at Emby.Server.Implementations.HttpClientManager.CoreHttpClientManager.SendAsyncInternal(HttpRequestOptions options, String httpMethod)
       at Emby.Server.Implementations.HttpClientManager.CoreHttpClientManager.SendAsync(HttpRequestOptions options, String httpMethod)
       at Tvdb.EntryPoint.EnsureToken(CancellationToken cancellationToken)
       at Tvdb.EntryPoint.GetResponse(HttpRequestOptions request, String method)
       at Tvdb.TvdbEpisodeProvider.FetchEpisodes(String url, Int32 page, CancellationToken cancellationToken)
       at Tvdb.TvdbEpisodeProvider.FetchEpisodesWithAllPages(String url, CancellationToken cancellationToken)
       at Tvdb.TvdbEpisodeProvider.FetchMainResult(String seriesId, String seasonType, CancellationToken cancellationToken)
       at Tvdb.TvdbEpisodeProvider.EnsureEpisodesInfo(String id, String seasonType, TimeSpan cacheTime, IDirectoryService directoryService, CancellationToken cancellationToken)
       at Tvdb.TvdbEpisodeProvider.GetEpisodeData(EpisodeInfo searchInfo, Boolean fillExtendedInfo, IDirectoryService directoryService, CancellationToken cancellationToken)
       at Tvdb.TvdbEpisodeProvider.GetMetadata(RemoteMetadataFetchOptions`1 options, CancellationToken cancellationToken)
       at Emby.Providers.Manager.MetadataService`2.ExecuteRemoteProviders(MetadataResult`1 temp, LibraryOptions libraryOptions, String logName, TIdType id, IRemoteMetadataProvider`2[] providers, MetadataRefreshOptions options, CancellationToken cancellationToken)
    Source: Emby.Server.Implementations
    TargetSite: Void MoveNext()
    
 

Emby License validation

Quote

Error checking registration status of MBSupporter    *** Error Report ***
    Version: 4.8.11.0
    Command line: /opt/emby-server/system/EmbyServer.dll -programdata /var/lib/emby -ffdetect /opt/emby-server/bin/ffdetect -ffmpeg /opt/emby-server/bin/ffmpeg -ffprobe /opt/emby-server/bin/ffprobe -restartexitcode 3 -updatepackage emby-server-deb_{version}_amd64.deb
    Operating system: Linux version 6.8.12-11-pve (build@proxmox) (gcc (Debian 12.2.0-14+deb12u1) 12.2.0, GNU ld (GNU Binutils for Debian) 2.40) #1 SMP PREEMPT_DYNAMIC PMX
    Framework: .NET 6.0.36
    OS/Process: x64/x64
    Runtime: opt/emby-server/system/System.Private.CoreLib.dll
    Processor count: 4
    Data path: /var/lib/emby
    Application path: /opt/emby-server/system
    MediaBrowser.Model.Net.HttpException: MediaBrowser.Model.Net.HttpException: Cancelling connection to https://mb3admin.com/admin/service/registration/validate due to a previous timeout.
       at Emby.Server.Implementations.HttpClientManager.CoreHttpClientManager.SendAsyncInternal(HttpRequestOptions options, String httpMethod)
       at Emby.Server.Implementations.HttpClientManager.CoreHttpClientManager.SendAsync(HttpRequestOptions options, String httpMethod)
       at Emby.Server.Implementations.Security.PluginSecurityManager.UpdateRegistrationStatus(String feature, String key, String version, CancellationToken cancellationToken)
    Source: Emby.Server.Implementations
    TargetSite: Void MoveNext()
    
 

This is usually due to Firewall, Security Software, Router config, DNS issues, VPN connection.

So you will have to fix that, and you should see a significant speed in crease.

Right now it waits for the timeout before proceeding.

Edited by Neminem
Neminem

Neminem 1518

Posted
Posted

Nice 😉 

@Philip6666How did you solve it, it might help others, if you tell how you did it. 

Philip6666

Philip6666 1

Posted
  • Author
Posted
2 minutes ago, Neminem said:

Nice 😉 

@Philip6666How did you solve it, it might help others, if you tell how you did it. 

Thank you for the detailed answer, I turned out to be a (tailscale) dns issue. I tried running: 

tailscale up --accept-dns=false

and now it's working. It's still quite slow, but it's atleast progressing. But I guess slow is to be expected with 50TB of media. 😅

  • Haha 1
Neminem

Neminem 1518

Posted
Posted

Yes that's a lot to injest. 

Have you enabled this?

As something like this will slow it down a lot. 

image.png.cb2fecaeae501204098bfce33fe17203.png

 

Philip6666

Philip6666 1

Posted
  • Author
Posted
Just now, Neminem said:

Yes that's a lot to injest. 

Have you enabled this?

As something like this will slow it down a lot. 

image.png.cb2fecaeae501204098bfce33fe17203.png

 

It used to be "as a scheduled task and when media is added" but after reading that it could slow down scanning I changed it to just "as a scheduled task" (for both tasks). Was that correct?

Neminem

Neminem 1518

Posted
Posted

Yes that should do it a lot faster.

After initial scan I have it set to "as a scheduled task and when media is added"

That way Initial scan will run a lot quicker, and then scheduled task will take over and then when media is added.

  • Like 1
  • Thanks 1

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now
 Share
Go to topic listing
×
×
  • Create New...