OMDb Timing Out

[adminExitium 355]

Version: 4.6.2.0

 

Stack Trace:

2021-06-05 14:55:57.824 Error HttpClient: Connection to https://private.omdbapi.com?apikey=fe53f97e&i=tt2781612&plot=short&tomatoes=true&r=json timed out
2021-06-05 14:55:57.826 Error App: Error in The Open Movie Database
        *** Error Report ***                                                                                                                                          Version: 4.6.2.0
        Command line: /app/emby/EmbyServer.dll -programdata /config -ffdetect /app/emby/ffdetect -ffmpeg /app/emby/ffmpeg -ffprobe /app/emby/ffprobe -restarte
xitcode 3
        Operating system: Linux version 5.12.8-xanmod1 (root@mascote) (gcc-11 (Debian 11.1.0-2) 11.1.0, GNU ld (GNU Binutils for Debian) 2.35.2) #0~git2021052
8.f7f420d SMP PREE
        Framework: .NET Core 3.1.13
        OS/Process: x64/x64
        Runtime: app/emby/System.Private.CoreLib.dll                                                                                                                  Processor count: 20
        Data path: /config
        Application path: /app/emby
        MediaBrowser.Model.Net.HttpException: MediaBrowser.Model.Net.HttpException: Connection to https://private.omdbapi.com?apikey=fe53f97e&i=tt2781612&plot
=short&tomatoes=true&r=json timed out
         ---> System.Threading.Tasks.TaskCanceledException: The operation was canceled.
         ---> System.IO.IOException: Unable to read data from the transport connection: Operation canceled.                                                            ---> System.Net.Sockets.SocketException (125): Operation canceled
           --- End of inner exception stack trace ---
           at System.Net.Sockets.Socket.AwaitableSocketAsyncEventArgs.ThrowException(SocketError error, CancellationToken cancellationToken)
           at System.Net.Sockets.Socket.AwaitableSocketAsyncEventArgs.GetResult(Int16 token)
           at System.Net.Security.SslStream.<FillBufferAsync>g__InternalFillBufferAsync|215_0[TReadAdapter](TReadAdapter adap, ValueTask`1 task, Int32 min, Int32 initial)
           at System.Net.Security.SslStream.ReadAsyncInternal[TReadAdapter](TReadAdapter adapter, Memory`1 buffer)                                                       at System.Net.Http.HttpConnection.FillAsync()
           at System.Net.Http.HttpConnection.ReadNextResponseHeaderLineAsync(Boolean foldedHeadersAllowed)                                                               at System.Net.Http.HttpConnection.SendAsyncCore(HttpRequestMessage request, CancellationToken cancellationToken)                                              --- End of inner exception stack trace ---
           at System.Net.Http.HttpConnection.SendAsyncCore(HttpRequestMessage request, CancellationToken cancellationToken)
           at System.Net.Http.HttpConnectionPool.SendWithNtConnectionAuthAsync(HttpConnection connection, HttpRequestMessage request, Boolean doRequestAuth, C
ancellationToken cancellationToken)
           at System.Net.Http.HttpConnectionPool.SendWithRetryAsync(HttpRequestMessage request, Boolean doRequestAuth, CancellationToken cancellationToken)
           at System.Net.Http.RedirectHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)                                                 at System.Net.Http.DecompressionHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
           at System.Net.Http.DiagnosticsHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)                                              at System.Net.Http.HttpClient.FinishSendAsyncBuffered(Task`1 sendTask, HttpRequestMessage request, CancellationTokenSource cts, Boolean disposeCts)
           at Emby.Server.Implementations.HttpClientManager.CoreHttpClientManager.SendAsyncInternal(HttpRequestOptions options, String httpMethod)
           --- End of inner exception stack trace ---
           at Emby.Server.Implementations.HttpClientManager.CoreHttpClientManager.SendAsyncInternal(HttpRequestOptions options, String httpMethod)
           at Emby.Server.Implementations.HttpClientManager.CoreHttpClientManager.SendAsync(HttpRequestOptions options, String httpMethod)
           at OMDb.Common.OmdbProvider.EnsureItemInfo(String imdbId, CancellationToken cancellationToken)
           at OMDb.Common.OmdbProvider.GetRootObject(String imdbId, CancellationToken cancellationToken)
           at OMDb.Common.OmdbProvider.Fetch[T](MetadataResult`1 itemResult, String imdbId, String language, String country, CancellationToken cancellationTok
en)
           at OMDb.Common.OmdbItemProvider.GetMovieResult[T](ItemLookupInfo info, CancellationToken cancellationToken)
           at MediaBrowser.Providers.Manager.MetadataService`2.ExecuteRemoteProviders(BaseItem originalItem, MetadataResult`1 temp, LibraryOptions libraryOpti
ons, String logName, TIdType id, IEnumerable`1 providers, CancellationToken cancellationToken)
        Source: Emby.Server.Implementations
        TargetSite: Void MoveNext()
        InnerException: System.Threading.Tasks.TaskCanceledException: The operation was canceled.
        Source: System.Net.Http
        TargetSite: Void MoveNext()
           at System.Net.Http.HttpConnection.SendAsyncCore(HttpRequestMessage request, CancellationToken cancellationToken)
           at System.Net.Http.HttpConnectionPool.SendWithNtConnectionAuthAsync(HttpConnection connection, HttpRequestMessage request, Boolean doRequestAuth, C
ancellationToken cancellationToken)
           at System.Net.Http.HttpConnectionPool.SendWithRetryAsync(HttpRequestMessage request, Boolean doRequestAuth, CancellationToken cancellationToken)
           at System.Net.Http.RedirectHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
           at System.Net.Http.DecompressionHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
           at System.Net.Http.DiagnosticsHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
           at System.Net.Http.HttpClient.FinishSendAsyncBuffered(Task`1 sendTask, HttpRequestMessage request, CancellationTokenSource cts, Boolean disposeCts)
           at Emby.Server.Implementations.HttpClientManager.CoreHttpClientManager.SendAsyncInternal(HttpRequestOptions options, String httpMethod)
        InnerException: System.IO.IOException: Unable to read data from the transport connection: Operation canceled.
        Source: System.Net.Sockets
        TargetSite: Void ThrowException(System.Net.Sockets.SocketError, System.Threading.CancellationToken)
           at System.Net.Sockets.Socket.AwaitableSocketAsyncEventArgs.ThrowException(SocketError error, CancellationToken cancellationToken)
           at System.Net.Sockets.Socket.AwaitableSocketAsyncEventArgs.GetResult(Int16 token)
           at System.Net.Security.SslStream.<FillBufferAsync>g__InternalFillBufferAsync|215_0[TReadAdapter](TReadAdapter adap, ValueTask`1 task, Int32 min, In
t32 initial)
           at System.Net.Security.SslStream.ReadAsyncInternal[TReadAdapter](TReadAdapter adapter, Memory`1 buffer)
           at System.Net.Http.HttpConnection.FillAsync()
           at System.Net.Http.HttpConnection.ReadNextResponseHeaderLineAsync(Boolean foldedHeadersAllowed)
           at System.Net.Http.HttpConnection.SendAsyncCore(HttpRequestMessage request, CancellationToken cancellationToken)
        InnerException: System.Net.Sockets.SocketException: Operation canceled
        Source:
        TargetSite:
        
 2021-06-05 14:55:57.976 Error ProviderManager: OmdbImageProvider failed in GetImageInfos for type Movie
        *** Error Report ***
        Version: 4.6.2.0
        Command line: /app/emby/EmbyServer.dll -programdata /config -ffdetect /app/emby/ffdetect -ffmpeg /app/emby/ffmpeg -ffprobe /app/emby/ffprobe -restarte
xitcode 3
        Operating system: Linux version 5.12.8-xanmod1 (root@mascote) (gcc-11 (Debian 11.1.0-2) 11.1.0, GNU ld (GNU Binutils for Debian) 2.35.2) #0~git2021052
8.f7f420d SMP PREE
        Framework: .NET Core 3.1.13
        OS/Process: x64/x64
        Runtime: app/emby/System.Private.CoreLib.dll
        Processor count: 20
        Data path: /config
        Application path: /app/emby
        MediaBrowser.Model.Net.HttpException: MediaBrowser.Model.Net.HttpException: Cancelling connection to https://private.omdbapi.com?apikey=fe53f97e&i=tt2
781612&plot=short&tomatoes=true&r=json due to a previous timeout.
           at Emby.Server.Implementations.HttpClientManager.CoreHttpClientManager.SendAsyncInternal(HttpRequestOptions options, String httpMethod)
           at Emby.Server.Implementations.HttpClientManager.CoreHttpClientManager.SendAsync(HttpRequestOptions options, String httpMethod)
           at OMDb.Common.OmdbProvider.EnsureItemInfo(String imdbId, CancellationToken cancellationToken)
           at OMDb.Common.OmdbProvider.GetRootObject(String imdbId, CancellationToken cancellationToken)
           at OMDb.Common.OmdbImageProvider.GetImages(BaseItem item, LibraryOptions libraryOptions, CancellationToken cancellationToken)
           at MediaBrowser.Providers.Manager.ProviderManager.GetImages(BaseItem item, LibraryOptions libraryOptions, CancellationToken cancellationToken, IRem
oteImageProvider provider, Int32 providerIndex)
        Source: Emby.Server.Implementations
        TargetSite: Void MoveNext()

 

Fetching the same URL manually (via Curl) takes around 45-50s for the first time and couple of seconds for every subsequent fetch, due to a cache possibly. The same errors seems to be occurring for almost every movie which seems reasonable considering everything cannot be in the cache at the same time.

Perhaps the timeout could be increased to 60s?

And also, possibly a way to specify our own OMDb API Key (This is mostly a wish list feature)?

[Luke 42078]

Hi, there is currently no way to provide your own api key.

[adminExitium 355]

And the timeout? Will I need to wait for a plugin update?

[Luke 42078]

I would give it a try again later and see if it's still happening.

[adminExitium 355]

That's what I thought too but it's been happening for the past 2 days now which is why I felt it was time to raise an issue.

Is raising the timeout that big of a change?

[Carlo 4561]

The problem is that this would make scanning even a longer process.

If you don't get a response back in 45 seconds then increasing it to 60 likely won't do anything.  But with multiple system all waiting longer it could just make the problem worse for everyone.

[adminExitium 355]

Not sure I follow your logic.

Fetching the URL manually via browser or curl gives me a response within 45-50 seconds (as noted in the first post) which seems to indicate that it's just a matter of the timeout in Emby being too conservative. 

And it doesn't matter if it waits longer or not, once the request is fired, the cache will be filled anyway. I have tried cancelling a manual request after 30 seconds of waiting and then checking again 30 seconds later gives me an instantaneous response which indicates that the cache is now filled (irrespective of the original request status).

[Carlo 4561]

What I mean is that if you had 10 media items and each took 30 seconds before timing out, that is 5 minutes of scan time.  If the timeout was increased to 60 seconds and each still timed out that is 10 minutes of scan time.

That kind of thing can make scanning take a long time.  Typically it's better to try those again later during another run when the source servers are faster and not timing out, which makes overall scanning faster.

There needs to be a balance of course between these but the timeout right now is reasonable. Could they be changed?  Of course but that's something likely Luke would need to decide on.

[adminExitium 355]

Is there any way to fetch only the OMDb metadata later then? I really don't want to do a full metadata refresh again since it takes quite some time with ~49k movies.

 

Btw, the timeout is still happening. I have just stopped the refresh for now and will try again later in case the cache gets filled automatically or if timeout becomes configurable.

[Luke 42078]

There isn't but you could just disable that one provider for now in your library settings. Current timeout by the way is 30 seconds.

[adminExitium 355]

Before I do that, enabling it later would just require a missing metadata refresh or a complete metadata refresh?

[Carlo 4561]

Missing only will likely do the trick for you but it really depends if you have providers in a specific order for any particular purpose.

[adminExitium 355]

Alright, thanks, will give it a try.

 

Worst case, I just write a script to fetch all the IMDB IDs from Emby and call the OMDb APIs myself (so that it's cached) at a minutely frequency before refreshing again in Emby.

[Carlo 4561]

Instead of doing anything like that you could try something like TinyMediaManager which can write NFO files and grab graphics for you.
Some people swear by it.

I myself just let Emby deal with meta-data provider errors and timeouts as it will resolve itself when it can.

[adminExitium 355]

Would be easier for me to just script it .. 

 

With ~80k Movies (49k was just one library) & ~500k Episodes, its much easier than moving to a new platform especially on a headless server, since TMM doesn't have any Web UI to manage it.

 

Btw, the current cache max-age (from the CloudFlare response) seems to be 7 days, so I expect this to start happening more frequently (if this was a recent change). I would still suggest bumping it up to 60s, if possible. 100s seems to be the CloudFlare limit for response from origin for any of the non-enterprise plans, which is what OMDb uses.

[Carlo 4561]

Dang, you have more movies than me.
Don't you know that's not allowed.  No help for you! LOL (soup naza reference).

So let me ask you about the timeouts you see. Is this for a full rescan or for just new media added?

 

[adminExitium 355]

Both actually. Depends on if it's a new or old movie. Newer ones generally get a cached response (seen from the cf-cache-status header in the Cloudflare response) whereas older ones timeout.

 

This specific log was from a full metadata refresh, as suggested in the other thread regarding the upgrade to 4.6.2 and the collections changes.

[adminExitium 355]

Just FYI, this is still happening. I have run my script so that the movie info of existing movies in Emby is cached but additions of older and/or relatively unknown movies are still timing out requiring me to refresh their metadata manually as and when I search the logs for the timeout and the corresponding movie.

[Happy2Play 9780]

So this is a isolated case where your responses exceed everyone else's and require a longer time?  So this would need to be configurable for you as I personally would not want to extend my time out and taking that much longer for any process to finish.

[adminExitium 355]

I very much doubt that it's limited specifically to my server or location since I have tried the same URL(s) via curl from multiple locations with them taking roughly the same amount of time everywhere i.e .40-50s.

Here are some to try out if you want to see for yourself:

* https://private.omdbapi.com?apikey=fe53f97e&i=tt0778877&plot=short&tomatoes=true&r=json

* https://private.omdbapi.com?apikey=fe53f97e&i=tt1236201&plot=short&tomatoes=true&r=json

* https://private.omdbapi.com?apikey=fe53f97e&i=tt1300572&plot=short&tomatoes=true&r=json

* https://private.omdbapi.com?apikey=fe53f97e&i=tt1199503&plot=short&tomatoes=true&r=json

* https://private.omdbapi.com?apikey=fe53f97e&i=tt0091995&plot=short&tomatoes=true&r=json

[Happy2Play 9780]

18 minutes ago, adminExitium said:

I very much doubt that it's limited specifically to my server or location since I have tried the same URL(s) via curl from multiple locations with them taking roughly the same amount of time everywhere i.e .40-50s.

Here are some to try out if you want to see for yourself:

* https://private.omdbapi.com?apikey=fe53f97e&i=tt0778877&plot=short&tomatoes=true&r=json

* https://private.omdbapi.com?apikey=fe53f97e&i=tt1236201&plot=short&tomatoes=true&r=json

* https://private.omdbapi.com?apikey=fe53f97e&i=tt1300572&plot=short&tomatoes=true&r=json

* https://private.omdbapi.com?apikey=fe53f97e&i=tt1199503&plot=short&tomatoes=true&r=json

* https://private.omdbapi.com?apikey=fe53f97e&i=tt0091995&plot=short&tomatoes=true&r=json

All of them open in less that 15s for me.

[adminExitium 355]

Huh, I wonder what's different on my end then.

 

A few more if you don't mind testing them please:

* https://private.omdbapi.com?apikey=fe53f97e&i=tt5110344&plot=short&tomatoes=true&r=json

* https://private.omdbapi.com?apikey=fe53f97e&i=tt1354011&plot=short&tomatoes=true&r=json

* https://private.omdbapi.com?apikey=fe53f97e&i=tt7060344&plot=short&tomatoes=true&r=json

* https://private.omdbapi.com?apikey=fe53f97e&i=tt0088568&plot=short&tomatoes=true&r=json

[Happy2Play 9780]

1 hour ago, adminExitium said:

Huh, I wonder what's different on my end then.

 

A few more if you don't mind testing them please:

* https://private.omdbapi.com?apikey=fe53f97e&i=tt5110344&plot=short&tomatoes=true&r=json

* https://private.omdbapi.com?apikey=fe53f97e&i=tt1354011&plot=short&tomatoes=true&r=json

* https://private.omdbapi.com?apikey=fe53f97e&i=tt7060344&plot=short&tomatoes=true&r=json

* https://private.omdbapi.com?apikey=fe53f97e&i=tt0088568&plot=short&tomatoes=true&r=json

Those took a little longer but under 30s for me.

Only thing odd I see is the missing "/" in the url after .com.  No idea if it really makes a difference though.

 

[Sentinel 12]

try removing &tomatoes=true

This works for me instantly every time.

 

https://www.omdbapi.com?apikey=fe53f97e&i=tt7060344&plot=short&r=json

https://www.omdbapi.com?apikey=fe53f97e&i=tt0088568&plot=short&r=json

 

Just noticed how old this post was, after i replied

 

Edited May 25, 2023 by Sentinel

[Luke 42078]

3 hours ago, Sentinel said:

try removing &tomatoes=true

This works for me instantly every time.

 

https://www.omdbapi.com?apikey=fe53f97e&i=tt7060344&plot=short&r=json

https://www.omdbapi.com?apikey=fe53f97e&i=tt0088568&plot=short&r=json

 

Just noticed how old this post was, after i replied

 

Hi, yea the problem is users like that feature.