ryzen5000 12 Posted November 17, 2022 Author Share Posted November 17, 2022 32 minutes ago, softworkz said: An important part of library scanning is metadata retrieval: Identify items Retrieve data Query for available images etc. This procedure requires API calls to be made to various data providers. And most of them have rate limits - either technically implemented or from agreements we made with some. That means that when we have made an API call to a provider, we need to wait a certain time before we can make the next request to their API. If we wouldn't adhere to that - or we would allow you to configure/remove it - they would block you (some sooner, some later) - which would be the better outcome. The worst would be when all Emby users would get blocked (that's why there is no such configuration for this  ) To conclude: this is a hard limit that you cannot overcome with more/better hardware. You can only try to rule out all those things that you can change, like the two things I mentioned above. Ok, API calls to metadata providers are a limiting factor. Link to comment Share on other sites More sharing options...
ryzen5000 12 Posted November 17, 2022 Author Share Posted November 17, 2022 (edited) You gave me a good idea and it works! Now I added a second GPU to EMBY. ould Edited November 17, 2022 by ryzen5000 Link to comment Share on other sites More sharing options...
softworkz 3349 Posted November 17, 2022 Share Posted November 17, 2022 (edited) If you seriously need faster library scanning throughput than Emby can handle when running 24/7 on scanning, then you need a different strategy. Pimping up a single server does not help for this. In that case, I would completely take away the task of library scanning from your main server, because the problem you'll have when you add so much content so quickly, is that you will most of the time have items which are already there but not processed completely yet (I assume that your case is really that extreme). I would buy a number of average NUCs, install Emby Server on each of them, but for the sole purpose of scanning, metadata retrieval and image extraction. You don't need any GPUs or something and no user will ever connect to any of those "Staging Servers". You move new content to those staging servers first. You let them do all those things, have them save images right into the media folders and use the XML or NFO plugin to write the determined metadata as files into those folders. You can then move those items that have been processed to your main server. It won't need to download metadata or extract images - just probing. Then your main server is free from all that stuff and you can parallelize the scanning with more or less staging servers. Edited November 17, 2022 by softworkz Link to comment Share on other sites More sharing options...
softworkz 3349 Posted November 17, 2022 Share Posted November 17, 2022 Oh, and you should route some of the metadata IP traffic through some VPN, so it doesn't appear that it's all coming from the same source  Link to comment Share on other sites More sharing options...
ryzen5000 12 Posted November 17, 2022 Author Share Posted November 17, 2022 I happen to have 2 gaming laptops which I could install EMBY server on and 3 Unraid servers. So I could add metadata from 5Â Link to comment Share on other sites More sharing options...
ryzen5000 12 Posted November 17, 2022 Author Share Posted November 17, 2022 2 minutes ago, softworkz said: Oh, and you should route some of the metadata IP traffic through some VPN, so it doesn't appear that it's all coming from the same source  I have a VPN and also a new linksys 3200ACM sitting here with open VPN and 16 licenses with different providors I could make a special router just for metadata and using DD WRT I can hide my metadata snatching. Link to comment Share on other sites More sharing options...
softworkz 3349 Posted November 17, 2022 Share Posted November 17, 2022 Make a test with one and see how for you get. I can't say I've done that before, but I think it should work out. Link to comment Share on other sites More sharing options...
softworkz 3349 Posted November 17, 2022 Share Posted November 17, 2022 2 minutes ago, ryzen5000 said: I could make a special router just for metadata You shouldn't hide it - you don't need to. You should rather distribute it. A little bit this way and a little bit that way.. Link to comment Share on other sites More sharing options...
softworkz 3349 Posted November 17, 2022 Share Posted November 17, 2022 Some may throttle by source IP. When you route all the same way, you might loose the parallelization effect. Maybe even changing source IPs. There's one thing they are even more afraid of than their servers being overloaded, and that is that somebody might pull ("steal") the whole database. Link to comment Share on other sites More sharing options...
ryzen5000 12 Posted November 17, 2022 Author Share Posted November 17, 2022 (edited) I don't know if I can use my main router for VPN because it handles all the 1.5 Gbps traffic and allows me to download at extra ordinary speeds of 185MB/s. I need to find out if I can target just certain apps for the VPN and not others or use one of my other routers, one is https://www.newegg.ca/p/22Z-007C-00655?Item=9SIAK3VJCH9042 which does all the heavy lifting. and the other https://www.amazon.ca/gp/product/B07L56SN8M/ref=ppx_yo_dt_b_search_asin_title?ie=UTF8&psc=1 is an access point to share EMBY with my two flat screens and wifi for the whole house including security cameras and this one https://www.amazon.ca/gp/product/B01M9DV0OY/ref=ppx_yo_dt_b_search_asin_image?ie=UTF8&psc=1 is just sitting here I am not currently using it. But it makes sense to run some metadata through a VPN router but I have to connect it to the internet and the internal network I am not sure I can seperate the server so it only gets internet from the VPN because it has to also be connected to the network without a VPN. Edited November 17, 2022 by ryzen5000 Link to comment Share on other sites More sharing options...
ryzen5000 12 Posted November 17, 2022 Author Share Posted November 17, 2022 (edited) Mabye they can add a feature where I can use EMBY input open VPN details into the GUI and spread it that way? or I can run the EMBY docker through a VPN of Qbittorrent and openVPN and do it that way, using another app that has VPN already built in and just share the same network. If its just doing metadata I don't need to worry about connecting to it with the TV's. Edited November 17, 2022 by ryzen5000 Link to comment Share on other sites More sharing options...
softworkz 3349 Posted November 17, 2022 Share Posted November 17, 2022 You can directly make a VPN connection from the "staging servers". That's very easy. Link to comment Share on other sites More sharing options...
ryzen5000 12 Posted November 17, 2022 Author Share Posted November 17, 2022 Why even move the data back and forth between the staging servers? Can I not just share an SMB share with another server and run background extractions from all the unraid servers? The share would show up in the staging servers just as though the disks were directly inside it. Link to comment Share on other sites More sharing options...
softworkz 3349 Posted November 17, 2022 Share Posted November 17, 2022 5 minutes ago, ryzen5000 said: Why even move the data back and forth between the staging servers? Can I not just share an SMB share with another server and run background extractions from all the unraid servers? The share would show up in the staging servers just as though the disks were directly inside it. This will cause multiple servers to "see" the data and probably act on it. I'm not sure what kind of use you are aiming for with your server, but when it goes into a high-volume direction, then you would want to keep all the IO for extraction and other tasks away from the main server, so the main server can do its job without interference. When you have 20 parallel playbacks from a certain storage, you really don't want to have multiple extraction tasks in parallel on the same IO path. Link to comment Share on other sites More sharing options...
softworkz 3349 Posted November 17, 2022 Share Posted November 17, 2022 For the same reason, I wouldn't run any command line scripts on the main server for manual extraction. On the other hand: when you just want to collect a huge amount of content and there will only be a handful of users, then you probably wouldn't need to care.. Link to comment Share on other sites More sharing options...
ryzen5000 12 Posted November 17, 2022 Author Share Posted November 17, 2022 (edited) I want to complete thumbnail image extraction scans quicker, using additional staging servers I can mount the same shares in those servers and have them all attack the video files at once to complete the thumbnail scans and chapter scenes faster. Edited November 17, 2022 by ryzen5000 Link to comment Share on other sites More sharing options...
softworkz 3349 Posted November 17, 2022 Share Posted November 17, 2022 1 minute ago, ryzen5000 said: all attack the video files at once to complete the thumbnail scans and chapter scenes faster How can it be faster when they all do the same in parallel? It is the same software and works the same on all servers. When a file change is detected, all will do the same (= process the same file) Link to comment Share on other sites More sharing options...
Happy2Play 8356 Posted November 17, 2022 Share Posted November 17, 2022 That is what I was thinking unless you config A library per server they will all potentially do the exact same thing. 1 Link to comment Share on other sites More sharing options...
softworkz 3349 Posted November 17, 2022 Share Posted November 17, 2022 Parallelization always starts with partitioning the work load. Link to comment Share on other sites More sharing options...
ryzen5000 12 Posted November 17, 2022 Author Share Posted November 17, 2022 Link to comment Share on other sites More sharing options...
ryzen5000 12 Posted November 17, 2022 Author Share Posted November 17, 2022 (edited) 8 hours ago, ryzen5000 said: This is my backup server which just runs the security cams. its not doing anything most of the day it can run extractions too. And I have another server I can even seperate them to just scan TV on one staging server and Movies on another staging server? Â https://officerat.store/ Edited November 17, 2022 by ryzen5000 Link to comment Share on other sites More sharing options...
softworkz 3349 Posted November 17, 2022 Share Posted November 17, 2022 The secret to success here is to create and implement a thought-out strategy, not to throw hardware at it. Link to comment Share on other sites More sharing options...
ryzen5000 12 Posted November 17, 2022 Author Share Posted November 17, 2022 (edited) I got the remote shares added to the docker template on my 1st staging server. UNRAID OS PRO https://officerat.store/product/lime-technology-unraid-os-pro-6-10-3-server-os-software/ Edited November 17, 2022 by ryzen5000 Link to comment Share on other sites More sharing options...
ryzen5000 12 Posted November 17, 2022 Author Share Posted November 17, 2022 (edited) Ok its all working except the thumbnail extraction on the staging server keeps stopping it is only working on the main server, I even tried to cancel the main servers extraction and the staging server still stops after a few seconds when I click extraction, any idea why? Could it be because its still doing a library scan? Â https://officerat.store/ Â Â Edited November 17, 2022 by ryzen5000 Link to comment Share on other sites More sharing options...
ryzen5000 12 Posted November 17, 2022 Author Share Posted November 17, 2022 (edited) Also, can I add Emby premier license to more than one server? Edited November 17, 2022 by ryzen5000 Link to comment Share on other sites More sharing options...
Recommended Posts
Create an account or sign in to comment
You need to be a member in order to leave a comment
Create an account
Sign up for a new account in our community. It's easy!
Register a new accountSign in
Already have an account? Sign in here.
Sign In Now