Metadata/Image retrieval challenges

[accarshop 1]

I'm running into an issue getting metadata and images for content being added to my library.  The short-short is that it works fine when I first start emby but at some point, the indexer (such as Fanart.tv, TMDB, etc.) starts blocking ("connection refused") the requests.  Most of the time, powering off the box for a couple minutes and then bringing it back up clears the problem for a time.  Sometimes connections are still refused and the restart process has to be repeated.

The frequency in which this happens ranges from hours to days and I'm struggling to catch a correlation between the incidents and library change volume.  This seems like a threshold issue in that my configuration is generating too many requests, but I'm not seeing it.  Libraries are configured for real time monitoring with advance pulls.  Looking at scheduled tasks status' I see they are only firing when content is added.  In reviewing my configuration it also raised a question or two. 

So, here I am, asking for advice from those who love and know emby on what I'm missing and how to correct this. 

For the questions, the 2 biggest are;

1) How does emby generate/maintain API keys for the various indexers?  In looking at several generations of the server logs I see that the key is consistent for each indexer.  Is the key unique to each emby installation or shared by all installations?  The latter could be significant in my configuration as I route most all of my internet traffic, emby and otherwise, through a commercial VPN.  My ISP knows enough about me and my family.  They don't need to know more.

2) Is it possible to use my own api keys for the indexers?  If so, how? 
     NOTE: I'd really rather not have to go this route.

 

SERVER INFO:

      NOTE: All external requests do go through a VPN

 

Quote

2022-10-17 00:00:01.285 Info App: Application version: 4.7.8.0
2022-10-17 00:00:01.290 Info App: Emby
    Command line: C:\xxx\Emby\system\EmbyServer.dll C:\xxx\Emby\system\EmbyServer.dll -noautorunwebapp
    Operating system: Microsoft Windows 10.0.17763
    Framework: .NET 6.0.9
    OS/Process: x64/x64
    Runtime: C:/xxx/Emby/system/System.Private.CoreLib.dll
    Processor count: 16
    Data path: C:\xxx\Emby\programdata
    Application path: C:\xxx\Emby\system
2022-10-17 00:00:01.291 Info App: Logs path: C:\xxx\Emby\programdata\logs
2022-10-17 00:00:01.291 Info App: Cache path: C:\xxx\Emby\programdata\cache
2022-10-17 00:00:01.291 Info App: Internal metadata path: C:\xxx\Emby\programdata\metadata
2022-10-17 00:00:01.291 Info App: Transcoding temporary files path: C:\xxx\Transcode\transcoding-temp
2022-10-17 00:00:01.303 Info App: Plugins:
    Auto Box Sets 1.2.6.0
    Auto Organize 1.6.2.0
    Bluray Folder Support 1.0.0.0
    Cinema Intros 1.0.40.0
    Custom Person Provider 1.0.0.378
    Dlna 1.0.90.0
    Dvd Folder Support 1.0.0.0
    Emby Guide Data 1.0.7.0
    Fanart.tv 1.0.13.0
    IMVDb 1.1.0.0
    IPTV 1.1.9.0
    M3U TV Tuner 1.0.13.0
    MovieDb 1.6.2.0
    MusicBrainz 1.0.21.0
    Nfo Metadata 1.0.69.0
    OMDb 1.0.18.0
    Open Subtitles 1.0.31.0
    Podcasts 1.2.5.0
    Port Mapper 1.1.8.0
    Server Configuration Backup 1.4.9.0
    Statistics 3.0.5.1
    Studio Cleaner 3.2.6.0
    Studio Images 1.0.3.0
    SubDb 1.0.7.0
    TheAudioDb 1.0.16.0
    TheTVDB 1.3.2.0
    TMDB People Fix 1.0.0.140
    Webhooks 1.0.19.0
    XmlTV 1.0.8.0
2022-10-17 00:00:01.304 Debug TaskManager: Rotate log file Completed after 0 minute(s) and 0 seconds
2022-10-17 00:10:01.239 Debug TaskManager: DailyTrigger fired for task: Configuration Backup
2022-10-17 00:10:01.241 Debug TaskManager: Queueing task ScheduledBackupTask
2022-10-17 00:10:01.242 Debug TaskManager: Executing Configuration Backup
2022-10-17 00:10:01.267 Debug TaskManager: Configuration Backup Completed after 0 minute(s) and 0 seconds

 

Thanks in advance for any assistance you all can offer.

 

 

 

[visproduction 178]

I suspect the blocking from the providers comes from their security preferences and it probably is coded into their service.  Turning off those services is perhaps the best choice.  It might have something to do with VPN.  My rich media content is archived locally and no image fails from about about 15000 or so.  There may be a switch in setup to keep the content local.  Perhaps someone who sees this post has that answer.

[Luke 38500]

Hi, as a test, if you disable the VPN, how does that compare?

[Happy2Play 9060]

6 hours ago, accarshop said:

2) Is it possible to use my own api keys for the indexers?

Emby has its own key for each provider.

But it is possible the vpn acccess is getting flagged by provider.

[accarshop 1]

Thanks for the quick response.

Yes, routing the emby server traffic through my ISP instead of VPN does change the symptoms.  When traffic isn't routed through a VPN the issue happens less frequently.  

I should have also added that when connections start getting blocked, it appears they are being blocked at the indexer.  I can log into the emby server desktop and manually run the call in a browser.  Using a different or no API key and the call is successful.  Use the emby key and the connection is refused.

The 2 points above are why I was asking how keys are generated and if they are shared.

If possible, I would very much like to continue using a VPN.  My ISP is rather notorious for their data collection and I'm sure they get a nice return on what they market to the various aggregators.

[Happy2Play 9060]

9 minutes ago, accarshop said:

The 2 points above are why I was asking how keys are generated and if they are shared.

They are api keys assigned to this project by the providers.  So yes every Emby server utilizes the same api keys for that provider.  But provider detection systems control access when they suspect abuse and block that system/connection.  I don't think there is anything Emby can do when the provider targets that specific connection, and you have to do the same workarounds you are doing restarting host/VPN connection.  Guessing but could be the VPN service itself.

[accarshop 1]

2 minutes ago, Happy2Play said:

Emby has its own key for each provider.

But it is possible the vpn acccess is getting flagged by provider.

@Happy2Play, you hit at the crux of what I was afraid of.  Under that premise, my request volume is lumped in with that of any other emby user who happen to be using VPN and are assigned the same address.  I would expect this to become more prevalent with more and more people adopting VPN solutions.  It is a shame, I really like the idea of the api calls using emby issued keys.  

At this point I'm unhappy enough with my ISP that I don't want to give up VPN.  So, I go back to part to the part of my original questions that asks if there is a way to use our own keys?

And thanks to everyone for assisting with this.

[Happy2Play 9060]

2 minutes ago, accarshop said:

So, I go back to part to the part of my original questions that asks if there is a way to use our own keys?

Currently there is no way to provider your own api key to provider plugins.

[Luke 38500]

Can you please provide the complete server log from when this happens? Thanks.

[accarshop 1]

1 minute ago, Happy2Play said:

I don't think there is anything Emby can do when the provider targets that specific connection, and you have to do the same workarounds you are doing restarting host/VPN connection.  Guessing but could be the VPN service itself.

@Happy2Play, that is twice we've replied at the same time. 

I completely agree with your statement that Emby's hands are tied if everyone is using the same key. 

I see you just replied again adding there isn't an option for using our own keys.  That pretty much ties the user's hands if they want to use VPN with Emby without added work.

Your idea about restarting host/VPN has merit and challenges.  For me, my VPN tunnels are established on a firewall and then rules flag traffic for VPN at either the device or traffic type levels.  There isn't an option to restart the tunnels w/o a significant amount of work and leakage risk to other traffic.  The option of restarting the Emby server, at first blush, would seem to be the better option but I've seen too many times where I've rebooted the system and it comes back quickly enough that the original connection path is restored.  Ergo, blocked by the provider.  It could also be that it got a new IP and it has been blocked as well.  I'll have to start noting the IP before and after reboots.

I share in your 'Trial and Error' mantra and don't like giving up.  One option I'll look at is adding a VPN client to the Emby server and set the kill switch.  That will improve the likelihood of getting a new address.  There is also the option of getting the metadata and images from outside of Emby.  Not an ideal solution but could work. 

VPN PROVIDER NOTE: I use to use SurfShark.  While I liked the service, throughput, and price, I moved off of them because their client had a nasty habit of sporadically turning off the kill switch.  This happened around reboots and when SurfShark updated their VPN client.  It put me in a position of having to constantly check the connection status to make sure it was still turned on.  With the Nord / Surf purchase it will be interesting to see what happens.

 

[accarshop 1]

8 minutes ago, Luke said:

Can you please provide the complete server log from when this happens? Thanks.

@Luke, would you mind if posted obfuscated logs with examples of what you are looking for?  I'm assuming you would like to see a debug block where a call was successful and then another where it wasn't.  What else would you like?

[Luke 38500]

1 hour ago, accarshop said:

@Luke, would you mind if posted obfuscated logs with examples of what you are looking for?  I'm assuming you would like to see a debug block where a call was successful and then another where it wasn't.  What else would you like?

I want to see the whole thing to see if there are similar errors from other domains.

[accarshop 1]

@Luke, I looked for a log where the server started out with successful connections and then started experiencing connection refused errors.  Sadly, they have rolled out. 

In my day job I have to review application logs amongst other things so the idea of digging into Emby logs hasn't been very appealing.  That said, today I did dig deeper into the logs.  There were some interesting things to see.

Attached is a log for your review and there are a few curiosities that I believe you will see and questions that will come from it.  I have quite a few thoughts and questions but will defer most of them.  What remain are:

* There are MANY repeated connection attempts that I don't see payload information on (lines 4881 on).  Almost like something is stuck in a loop.
* When it is doing library scans Emby is re - requesting images and nfo files for what looks to be a surprising large percentage of the library.  As a swag I'd say greater than 20% in the TV library alone.  The VAST majority of the files I spot checked showed that there were existing image/nfo files for the media.  What triggers the refresh of the meta data for these?
* My library is very large and the majority of the content static.  Any way to still monitor for new content and only do full scans on demand?
* Recommendations on plugins to stay away from.
 

embyserver.txt

[Luke 38500]

In this log all outgoing communications are failing to multiple domains and now it looks more like you have something blocking the server from doing this.

If you want to reduce requests then you'll want to disable more library features, such as what image types the server will look for to try and download.

I would also check that your existing images are using naming conventions supported by the server in order to ensure that they're utilized.

Let us know if this helps. Thanks.

[accarshop 1]

Quote

In this log all outgoing communications are failing to multiple domains and now it looks more like you have something blocking the server from doing this.

Yes, the logs that show the connections working then failing rolled off.  That log is after a reboot and the providers continued to reject the connections.  If I ran the requests manually, where possible, or with a different key, they work.

Quote

I would also check that your existing images are using naming conventions supported by the server in order to ensure that they're utilized.

The file naming specifications are nice in how flexible they are.  The existing images and nfo are named/created by Emby.  I'm assuming this is in response to my question as to why SO many are being re-request on full scans. 

In the log there was a HUGE block of requests that I didn't see indications of what was triggering them. Any feedback on what they are and how to reduce the volume?

[Happy2Play 9060]

A guess would be multiple user across the same vpn address and provider flags as suspicious activity.  As this is the provider reacting to the VPN as you have already shown without the VPN it works. 

Honestly don't think there is anything Emby can do here beside a FR for allow personal api keys.  But I really see that as a temporary solution as it is a provider issue with VPN services.

 

[accarshop 1]

@Happy2Play,

Based on slow slog through some of my logs I do have concerns around the volume of requests going out.  Using the "Scan Media Library" scheduled task is a nice catch all cleanup for any misses from when  new media is added, BUT, when it runs, I see SO many re-requests for NFO and image data that is already in the library.  I'm backing the frequency back to weekly (as far as the gui allows) and see if that helps.

I'm also going to try to keep a closer eye on my plugin traffic to see what it adds. 

The net of all of this is I may reduce the frequency MY Emby install is hitting the providers.  Like you, my gut feeling is that at some point there will be an FR to allow personal provider keys.  I started to go ahead and put in an FR but thought I'd let this thread play out first.

 

[Happy2Play 9060]

Old topic on TVDB but probably still relavent.

We're behind cloudflare as you suspected and the captcha's would be coming from Cloudflare and not us. I think we could whitelist that IP but even if we did it wouldn't help much as your IP will keep changing. I'm not sure if there is going to be a reliable way for you to use the API through such a well known VPN.

I'm assuming you have some reason for not wanting to disable the VPN but quite honestly I think any solution you do find is likely to only be temporary as Cloudflare is going to present captcha's on any connections they think look suspicious, which means most VPN's will eventually get a captcha.