chef 3810 Posted August 25, 2021 Posted August 25, 2021 (edited) 2 hours ago, Skyfay said: I use Linux Ubuntu 20.04… Do i have to use the latest beta that the plugin will work? You should be okay on Ubuntu. Try the fingerprinting task first. Allow it to finish, then try the title sequence detection next. Edited August 25, 2021 by chef
samuelqwe 317 Posted August 25, 2021 Posted August 25, 2021 2 hours ago, rbjtech said: Fingerprinting is pretty fast - but detection is 'slow' (set to 1 show at a time) but tbh, I've no idea what is involved here - cpu is sitting pretty much idle while the task is running - so it could be throttling itself to not kill the cpu ? The detection is basically just a bunch of complex math, so it should be able to use up quite a bit of CPU. I’m not sure why it doesn’t use that much, although I’ve noticed similar behaviour. Maybe the CPU can’t calculate faster than that? ¯\_(ツ)_/¯ 1
chef 3810 Posted August 25, 2021 Posted August 25, 2021 8 hours ago, STR8 said: Is this what your looking for embyserver.txt 8.17 MB · 0 downloads Sweet looks good!
Skyfay 19 Posted August 25, 2021 Posted August 25, 2021 22 minutes ago, chef said: You should be okay on Ubuntu. Try the fingerprinting task first. Allow it to finish, then try the title sequence detection next. Not work for me... Im on version 4.6.4.0 is that fine?
chef 3810 Posted August 25, 2021 Posted August 25, 2021 9 minutes ago, Skyfay said: Not work for me... Im on version 4.6.4.0 is that fine? pkease darf ich eine Log-Datei haben. may I have a log?
chef 3810 Posted August 25, 2021 Posted August 25, 2021 2 hours ago, rbjtech said: Probably best to wait for @chef as I'm now having issues with the sequence detection not working - as has been posted above. edit - it was me experimenting with lowering the 10 sec value to 5 to try and catch some small intro's - putting it back to 10 and everything is good again. I'll continue to experiment .. so far results have been 100% detection rate for me but that's only on 5 test shows/5 seasons ... but looking good. Fingerprinting is pretty fast - but detection is 'slow' (set to 1 show at a time) but tbh, I've no idea what is involved here - cpu is sitting pretty much idle while the task is running - so it could be throttling itself to not kill the cpu ? Set to 1? Interesting. That might take a while.
rbjtech 5284 Posted August 25, 2021 Posted August 25, 2021 1 minute ago, chef said: Set to 1? Interesting. That might take a while. I guess I'm trying to understand what the workload is here - if running the detection threads in parallel speeds up the entire process - then what is causing the 'bottleneck' ? I'm just testing on a 80 'episode' emby test instance - Fingerprinting took 5 mins, 3 secs - the resulting dB is tiny with 80 rows. Detection took 15 mins, 33 secs .. CPU was 6-12% (i7 6700K) during the detection. Media on local SATA Physical disk. Emby on an SSD. ps - I think I've also found a potential bug in the process - If there is no 'Season X' folder - then the detection appears to hang ? This was my mistake when setting up my test instance - ie used \tv-test\showname\showfile.mkv instead of \tv-test\showname\season 1\showfile.mkv. Of course emby worked ok with that, but looking at the logs - it was listing the show in the SkipIntro log as \tv-test ... it had no mention of \tv-test\showname\ - all other shows used the full folder name. Unfortunately, I deleted the logs and started fresh to get it to work again (after adding a season folder in the show) - but I should be able to re-create easily enough. -- Let me get some stats on how the parallel processes impact things - I'm going to put it back to 5 for starters .. 1
chef 3810 Posted August 25, 2021 Posted August 25, 2021 @rbjtech Check this out. This version creates a singleton of the "TitleSequenceDetectionManager". This might speed things up because we wouldn't be 'new-ing' up an instance each time during the task. We will just grab the instance and use it. It would most likely keep memory usage down as well because there wouldn't be several instances of it waiting for garbage collection. The only other thing I could try (which might speed things up...) is to make all the math calculations asynchronous... But they kind of already are multithreaded, so it might not change anything. IntroSkip_v2.zip @samuelqwe I have something that will take more then one episode (1-2 && 3-4, then 5-6 && 7-8) and compare them. But this means that we would be running 4 series at a time, with 4 episodes at time. Accessing 16 items at once for the calculations. It really pushed the CPU, but it was fast with the same results. My dev machine only has an i5.
rbjtech 5284 Posted August 25, 2021 Posted August 25, 2021 Thanks Chef - will try that version in a sec. So on the previous versions - my findings are : if I try and run more than 1 (I tried 2..) for finger printing - the HDD goes ballistic and hits 100% IO - resulting it it actually becoming slower to complete the task. I cancelled it after 10 mins (vs 5 to complete previously). However, detection is much better with 5 (I only have 5 shows to test with..) - this is still interesting though as watching the CPU graph, it peaked at 60% and then stepped down 10% as each show completed (I presume), multiple times as the number of show episodes differ - then levelled out at 10% for another minute or so. Total time 6 mins vs the previous 15 minutes. Logs show the parallelism here as opposed to sequential list. So - couple of things for me to try a) put the test shows on the SSD and re-test. If this speeds up substantially, then this is going to be an issue - as all my media is on HDD as I suspect is most peoples. Haven't tried on a NAS yet - but suspect that will swamp the I/O pretty quick... b) I'm going to graph the CPU and match with the detection logs. c) I'm going to run procmon to see what is being read/written to - maybe there are some efficiencies to be had somewhere for the fingerprinting ... d) I'll try the new binary once I've concluded the above - I don't want to keep moving the goalposts 1
rbjtech 5284 Posted August 25, 2021 Posted August 25, 2021 So it's pretty easy to see why the I/O is getting hammered ! This was just running with 2 threads, and the reads are flipping between the two files with what looks to be 32K chunks ? Thus the HDD is seeking each time. Is there any way to read larger file chunks / increase the buffer size on the ffmpeg reads perhaps ? 1
chef 3810 Posted August 25, 2021 Posted August 25, 2021 1 hour ago, rbjtech said: Thanks Chef - will try that version in a sec. So on the previous versions - my findings are : if I try and run more than 1 (I tried 2..) for finger printing - the HDD goes ballistic and hits 100% IO - resulting it it actually becoming slower to complete the task. I cancelled it after 10 mins (vs 5 to complete previously). However, detection is much better with 5 (I only have 5 shows to test with..) - this is still interesting though as watching the CPU graph, it peaked at 60% and then stepped down 10% as each show completed (I presume), multiple times as the number of show episodes differ - then levelled out at 10% for another minute or so. Total time 6 mins vs the previous 15 minutes. Logs show the parallelism here as opposed to sequential list. So - couple of things for me to try a) put the test shows on the SSD and re-test. If this speeds up substantially, then this is going to be an issue - as all my media is on HDD as I suspect is most peoples. Haven't tried on a NAS yet - but suspect that will swamp the I/O pretty quick... b) I'm going to graph the CPU and match with the detection logs. c) I'm going to run procmon to see what is being read/written to - maybe there are some efficiencies to be had somewhere for the fingerprinting ... d) I'll try the new binary once I've concluded the above - I don't want to keep moving the goalposts that is interesting! I wonder if it is because we are requesting a parallel task and then only giving it one thing to do. I'll have to read some of MS documentation of the Task Parallel Library and see if setting a MaxDegreeOfParallism to 1 has side effects.
chef 3810 Posted August 25, 2021 Posted August 25, 2021 (edited) 57 minutes ago, rbjtech said: So it's pretty easy to see why the I/O is getting hammered ! This was just running with 2 threads, and the reads are flipping between the two files with what looks to be 32K chunks ? Thus the HDD is seeking each time. Is there any way to read larger file chunks / increase the buffer size on the ffmpeg reads perhaps ? Instead of reading each bin after it has been processed and saved, we can wait and read a bunch of them at the end of the task. I believe this is possible. EDIT: plus we are opening and writing to the database each time. Edited August 25, 2021 by chef
rbjtech 5284 Posted August 25, 2021 Posted August 25, 2021 3 minutes ago, chef said: Instead of reading each bin after it has been processed and saved, we can wait and read a bunch of them at the end of the task. I believe this is possible. I don't believe it's the .bin that has the problem - that's saved on the SSD in my case, so separate I/O channel. I'm actually wondering if this is simply a case of the limitations of a mechanical HDD.. If I used 2 HDD's (two 'spindles') - half the shows one one, half on the other - I suspect, it would halve the fingerprinting time ... 1
chef 3810 Posted August 25, 2021 Posted August 25, 2021 Oh there is something that might be concerning. What is that buffer overflow in the log? They are never a good thing, especially on a web facing services. That is were people can hide bad code in requests.
rbjtech 5284 Posted August 25, 2021 Posted August 25, 2021 10 minutes ago, chef said: Oh there is something that might be concerning. What is that buffer overflow in the log? They are never a good thing, especially on a web facing services. That is were people can hide bad code in requests. There were a lot of them (filtered) - and not just writing the .bin's - all from ffmpeg though...
rbjtech 5284 Posted August 25, 2021 Posted August 25, 2021 So some quick results on moving the media to a SATA based SSD For a single thread, there is not a lot in it vs the HDD, but as soon as I add more threads, it's substantially quicker up to a point (3 threads), then it's no faster - I suspect I'm maxing the SATA I/O from the SSD at this point. Fingerprint Threads Time Taken 1 5min 1sec 2 2min 11sec 3 1min 20sec 4 1min 24 sec 5 1min 20sec So in summary, I *think* unless you have a way of running these processes on separate HDD's - then you will hit the disk I/O pretty quickly. I'm going to do a comparison on a 1Gig 'NAS' (file share) as well to see how that handles the I/O.
samuelqwe 317 Posted August 25, 2021 Posted August 25, 2021 2 hours ago, chef said: @samuelqwe I have something that will take more then one episode (1-2 && 3-4, then 5-6 && 7-8) and compare them. But this means that we would be running 4 series at a time, with 4 episodes at time. Accessing 16 items at once for the calculations. It really pushed the CPU, but it was fast with the same results. My dev machine only has an i5. So it’s faster? Is there anything that prevents you from only doing 2 episodes per series at a time, or is it just a choice to speed things up?
chef 3810 Posted August 25, 2021 Posted August 25, 2021 36 minutes ago, samuelqwe said: So it’s faster? Is there anything that prevents you from only doing 2 episodes per series at a time, or is it just a choice to speed things up? The CPU percent increases. One thread handles odd episodes and the other even. I don't think anything gets missed. I'm just running it again to make sure.
chef 3810 Posted August 25, 2021 Posted August 25, 2021 (edited) 1 hour ago, rbjtech said: So some quick results on moving the media to a SATA based SSD For a single thread, there is not a lot in it vs the HDD, but as soon as I add more threads, it's substantially quicker up to a point (3 threads), then it's no faster - I suspect I'm maxing the SATA I/O from the SSD at this point. Fingerprint Threads Time Taken 1 5min 1sec 2 2min 11sec 3 1min 20sec 4 1min 24 sec 5 1min 20sec So in summary, I *think* unless you have a way of running these processes on separate HDD's - then you will hit the disk I/O pretty quickly. I'm going to do a comparison on a 1Gig 'NAS' (file share) as well to see how that handles the I/O. Nice work. That totally makes sense. There is quite a bit of disk I/O in that task. Let me see if I can pipe the ffmpeg result to memory stream instead of writing to disk. I'll try this: https://stackoverflow.com/questions/19634749/can-i-use-ffmpeg-to-output-jpegs-to-a-memory-stream-instead-of-a-file Edited August 25, 2021 by chef 1
chef 3810 Posted August 25, 2021 Posted August 25, 2021 OH! I'm close to outputting to memory instead of file. Wow! Okay, this might take a second! 1
Sammy 790 Posted August 25, 2021 Posted August 25, 2021 Is Process Monitor something to add in to a Win10 installation?
Sammy 790 Posted August 25, 2021 Posted August 25, 2021 Feature Request: Please update the version number on each release and show the version number in the forum post so we know which version we are using.
Sammy 790 Posted August 25, 2021 Posted August 25, 2021 This process seems lengthy and there's no real way to know if it properly completed or not: Spinner spins and spins then stops but this dialog screen still shows. 1
rbjtech 5284 Posted August 25, 2021 Posted August 25, 2021 4 minutes ago, Sammy said: This process seems lengthy and there's no real way to know if it properly completed or not: Spinner spins and spins then stops but this dialog screen still shows. Yea there are a few little issues with the GUI - the same if you use the 'Remove Data' for that show - unless you click another show, the data still shows (even though it's gone). But these are just minor refresh issues I'm sure - the key issue (for me anyway) at the moment is performance testing. @Sammy I see you have selected 4 threads for Fingerprinting (from the screen grab above) - what type of storage is this using and what levels of activity do you see when it's running ?
Recommended Posts