goin3d 4 Posted May 11, 2025 Posted May 11, 2025 19 minutes ago, mickle026 said: No, I havent changed that, its still multicore, i have only changed the writing of the strms, and now the http downloader to support secure sockets and tls in the version sent via pm thanks! i'll keep trying to figure it out. for others running tests, are you experiencing the same in terms of increased run times, or is this just something about what i'm doing?
Jekaboom 4 Posted May 11, 2025 Posted May 11, 2025 (edited) 2 hours ago, goin3d said: thanks! i'll keep trying to figure it out. for others running tests, are you experiencing the same in terms of increased run times, or is this just something about what i'm doing? Yes, runtime increased by roughly factor x4. Recent results are now as per below. Do note that I run this on a very 'weak' virtual container and against a huge file (over 500k .strm files), so I dont mind how long it takes really - the results / writes with the latest version are much more comprehensive and accurate; i take this over a faster processing if that's what it takes.. 10.May.2025 - 13:05:11 Found 60704 folders in Output path. Scanning... 10.May.2025 - 13:05:12 Found 513234 files with extension .strm 10.May.2025 - 13:05:12 Low memory mode enabled. RAM threshold set to: 2990.70 MB 11.May.2025 - 17:38:15 Unknown M3U Items found: 0. 11.May.2025 - 19:41:38 Task Ended at: 11/05/2025 19:41:38 11.May.2025 - 19:41:38 Task Duration: 06:37:09.37 Edited May 11, 2025 by Jekaboom 1
mickle026 650 Posted May 11, 2025 Author Posted May 11, 2025 (edited) here is the latest with TLS handler for secure sockets when downloading I have also removed a check i put in a while ago that checks if an output strm exists in the queue (it does not check contents) so what will happen is where there are duplicates both will be sent to write, but the writer will decide based on config whether to write or not. I am wondering if this is the cause of the slow down, I had put this in while testing because of the few duplicates causing emby to rescan library I thought I had removed this already, sorry guys I hadn't , it was put there when checking why duplicates were causing emby to rescan your libraries. if (!Strms.Any(t => t.Item1 == Path.Combine(folderName, fileName))) I shall soon know by your comments if this was the reason for the slow down. VODtoSTRMscheduledTask.dll Edited May 11, 2025 by mickle026
sundevil67 28 Posted May 11, 2025 Posted May 11, 2025 35 minutes ago, mickle026 said: here is the latest with TLS handler for secure sockets when downloading I have also removed a check i put in a while ago that checks if an output strm exists in the queue (it does not check contents) so what will happen is where there are duplicates both will be sent to write, but the writer will decide based on config whether to write or not. I am wondering if this is the cause of the slow down, I had put this in while testing because of the few duplicates causing emby to rescan library I thought I had removed this already, sorry guys I hadn't , it was put there when checking why duplicates were causing emby to rescan your libraries. if (!Strms.Any(t => t.Item1 == Path.Combine(folderName, fileName))) I shall soon know by your comments if this was the reason for the slow down. VODtoSTRMscheduledTask.dll 966.5 kB · 3 downloads I'm at 25% after two days ... I just downloaded this latest file & restarted task
mickle026 650 Posted May 11, 2025 Author Posted May 11, 2025 (edited) 7 minutes ago, sundevil67 said: I'm at 25% after two days ... I just downloaded this latest file & restarted task on a zx spectrum? LOL - jeez! I know its not funny really. I am just testing the obfuscate m3u outputs, and if they are ok i will post that Edited May 11, 2025 by mickle026 damn dyslexia!
goin3d 4 Posted May 11, 2025 Posted May 11, 2025 ok, this is gonna sound crazy but here goes lol! the last version you posted i'm seeing smaller files run a little bit faster, but the larger files not really. The reason i asked about the cores earlier is a few versions ago i was used to seeing many of the CPU cores working when the plugin ran. Now i see this. I have no idea why though.
mickle026 650 Posted May 11, 2025 Author Posted May 11, 2025 9 minutes ago, goin3d said: ok, this is gonna sound crazy but here goes lol! the last version you posted i'm seeing smaller files run a little bit faster, but the larger files not really. The reason i asked about the cores earlier is a few versions ago i was used to seeing many of the CPU cores working when the plugin ran. Now i see this. I have no idea why though. What is in the top of your log? Like this example: 11.May.2025 - 21:36:32 Memory Allocated for this plugin: 4.00 GB 11.May.2025 - 21:36:33 CPU Cores: 8: Logical Processors (Max Threads): 16, Hyperthreading: True 11.May.2025 - 21:36:33 Using max 16 threads
mickle026 650 Posted May 11, 2025 Author Posted May 11, 2025 (edited) I can see what you mean... but not sure why yet .... Recording 2025-05-11 231804-cpu monitor.mp4 Edited May 11, 2025 by mickle026
goin3d 4 Posted May 11, 2025 Posted May 11, 2025 Machine 1: (native Windows) - not sure why 4gb is all that's allocated? 11.May.2025 - 16:23:00 Build Version: 1.0.0.176 11.May.2025 - 16:23:00 Memory Allocated for this plugin: 4.00 GB 11.May.2025 - 16:23:02 CPU Cores: 16: Logical Processors (Max Threads): 32, Hyperthreading: True 11.May.2025 - 16:23:02 Using max 32 threads 11.May.2025 - 18:19:47 Debugging: [scanM3UFileAsync] m3uContents Contains: 290683 Items. 11.May.2025 - 18:19:47 -----------------------[ Updating Output Folder ]------------------------------- 11.May.2025 - 18:19:47 All Updating tasks completed. 11.May.2025 - 18:21:43 Finished this import. 11.May.2025 - 18:21:43 Task Ended at: 5/11/2025 6:21:43 PM 11.May.2025 - 18:21:43 Task Duration: 01:58:40.83 Machine 2: (Docker) 11.May.2025 - 17:45:29 Build Version: 1.0.0.176 11.May.2025 - 17:45:29 Memory Allocated for this plugin: 56.00 GB 11.May.2025 - 17:45:29 CPU Cores: -1: Logical Processors (Max Threads): 32, Hyperthreading: True 11.May.2025 - 17:45:29 Using max 32 threads 11.May.2025 - 19:14:26 Debugging: [scanM3UFileAsync] m3uContents Contains: 290673 Items. 11.May.2025 - 19:14:26 -----------------------[ Updating Output Folder ]------------------------------- 11.May.2025 - 19:14:26 All Updating tasks completed. 11.May.2025 - 19:14:35 Finished this import. 11.May.2025 - 19:14:35 Task Ended at: 5/11/2025 7:14:35 PM 11.May.2025 - 19:14:35 Task Duration: 01:29:06.86 both configured: I can't even begin to understand why the Windows machine is a faster box, yet finished 30 minutes slower! Also don't know why machine 2 is showing -1 for CPU Cores.
mickle026 650 Posted May 12, 2025 Author Posted May 12, 2025 (edited) I am going to re-write the m3u processor code part, so please bare with me a while, so that it makes sure it uses all the available cpu's in the config. To check it, because windows monitor is rubbish I just cobbled together a cpu monitor app for windows, its attached. I wrote it for my machine so there might be layout issues on others, it has code to resize for how many cores, but its not tested anywhere else so i don't know resizing works..... use at your own risk, no liability accepted etc... CPUMonitor.zip This screenshot is from the m3u plugin using all cores, but i am not happy with it (m3u plugin) because there are errors now where there wasn't before. Edited May 12, 2025 by mickle026
sundevil67 28 Posted May 12, 2025 Posted May 12, 2025 On 5/11/2025 at 1:34 PM, mickle026 said: on a zx spectrum? LOL - jeez! I know its not funny really. I am just testing the obfuscate m3u outputs, and if they are ok i will post that Some more info in case it helps: it is crashing the Emby app now every time when it gets to around 15%. When I relaunch the server, the scheduled task area says I haven't run it for 2 weeks.
fbrassin 47 Posted May 13, 2025 Posted May 13, 2025 (edited) I realized that this: #EXTINF:-1 tvg-id="" tvg-name="KENGAN ASHURA S01 E01" tvg-logo="https://image.tmdb.org/t/p/w600_and_h900_bestv2/zFxRwM1hG7rmAkRC5VA1KbigRAF.jpg" group-title="Serie",KENGAN ASHURA S01 E01 http://XXXX/xxxxxx.mp4 Is saved as KAN ASHURA removing ENG in the middle. I didn't check anything about removing language in my config. Maybe is it a bug? What could i check? Edited May 13, 2025 by fbrassin
goin3d 4 Posted May 13, 2025 Posted May 13, 2025 7 hours ago, fbrassin said: I realized that this: #EXTINF:-1 tvg-id="" tvg-name="KENGAN ASHURA S01 E01" tvg-logo="https://image.tmdb.org/t/p/w600_and_h900_bestv2/zFxRwM1hG7rmAkRC5VA1KbigRAF.jpg" group-title="Serie",KENGAN ASHURA S01 E01 http://XXXX/xxxxxx.mp4 Is saved as KAN ASHURA removing ENG in the middle. I didn't check anything about removing language in my config. Maybe is it a bug? What could i check? check the contents of this box: I think you'll see ENG listed in the box. for example: Redux,ENG,HINDI,FRENCH,GERMAN,SPANISH this is going to remove ENG from any title. 1
mickle026 650 Posted May 13, 2025 Author Posted May 13, 2025 yeah, it wont remove ENG even with remove country codes, so it must be somewhere in your config to do so. Just so you know i've been trying to fix the m3u processing part to ensure its truly multithreaded, and I have done that but completely broken the output stage, and being an idiot i have gone and saved over my older code, so i have a big job now correcting it again. it will takes some time. However a big m3u with 1/2 million entries is processing in about 5 mins again. I will fix it, it will just take a while. 2
goin3d 4 Posted May 13, 2025 Posted May 13, 2025 19 minutes ago, mickle026 said: yeah, it wont remove ENG even with remove country codes, so it must be somewhere in your config to do so. Just so you know i've been trying to fix the m3u processing part to ensure its truly multithreaded, and I have done that but completely broken the output stage, and being an idiot i have gone and saved over my older code, so i have a big job now correcting it again. it will takes some time. However a big m3u with 1/2 million entries is processing in about 5 mins again. I will fix it, it will just take a while. sorry for all the trouble you're having to go through, but it's greatly appreciated!!!!!
mickle026 650 Posted May 13, 2025 Author Posted May 13, 2025 To be fair, i have learnt lots about multi-threading since going this route and actually how bad it actually is! It is very hard to control and the cpu more often than not gets into race conditions where it alters something it shouldn't, it losses variables, drops items because it lost track of where it is. Multithreading although it is very fast is actually very poor! If I send a task off to do something on a single thread , thats fine. If i send another task off on another thread that too is fine. However, if I spread the load of something across all cpu's, my goodness - this is bad! You have to do all sorts of monitoring to use it that way and it can still go wrong! So what I am attempting now is, get the cpu count, divide the m3u in tom parts by that many, then process separately each part on a separate thread like doing separate jobs, then join them at the end. The m3u processing so far is working like a dream - that is working well!, the joining then at the end is not. I have to get that to behave. Some of my previous code is not compatible with this method, I mean it works but misbehaves so I am getting the m3u processing to say there are for example 50,000 strms, but then its failing at the output and only writing 1,000. So like I said before where I am at is fixing it. And at least I now have a cpu monitor tool thats better than windows11's!
goin3d 4 Posted May 13, 2025 Posted May 13, 2025 you should make the cpu monitor it's own emby plugin LOL
Apotropaic 41 Posted May 13, 2025 Posted May 13, 2025 Glad that you're making progress, just sad that none of us can really help you in any meaningful way!! I've changed so much of my setup over the past week that I can no longer get anything working, tried both older and newer versions. Just seen your post and had the brainwave to install emby on my main windows machine which has more CPU and memory than my emby server just to get my strms exported!
mickle026 650 Posted May 14, 2025 Author Posted May 14, 2025 5 hours ago, goin3d said: you should make the cpu monitor it's own emby plugin LOL Here you go, no idea if it works on linux or mac (the code is there for it but I dont have them so i cant check or debug it), but this is running on windows 11 - it uses powershell in the background on windows 11 to get the data. I needed timeout, so I had a go..... CPUMonitor.dll Its a demo (first) build, but it works 1
Apotropaic 41 Posted May 14, 2025 Posted May 14, 2025 The CPU monitor kinda works on Linux (Ubuntu 24.04), it loads up but only shows one CPU. I'm in a VM with 3 cores but 6 threads overall. But I'm not too sure what it is actually showing, doesn't seem to be a particular core, or an average or even the sum of the 6. The % does change but it seems to stay between 4-6% even when I can see the CPU's getting battered using 'htop'. Anyway hope that helps but don't spend unnecessary time to fix this
mickle026 650 Posted May 14, 2025 Author Posted May 14, 2025 22 minutes ago, jaspalgill said: The CPU monitor kinda works on Linux (Ubuntu 24.04), it loads up but only shows one CPU. I'm in a VM with 3 cores but 6 threads overall. But I'm not too sure what it is actually showing, doesn't seem to be a particular core, or an average or even the sum of the 6. The % does change but it seems to stay between 4-6% even when I can see the CPU's getting battered using 'htop'. Anyway hope that helps but don't spend unnecessary time to fix this Ah, that's from /proc/stat, I don't have linux so a quick look up on bing gave me that. I did another look up after your post where you mention htop and found two options for getting cpu usadge I changed it to mpstat or htop, took 5 mins - not really wasted time its all learning which mpstat && mpstat -P ALL 1 1 or htop -d 1 -b | head -20 I found these commands, and same outputs so i changed the code to suit, it tries mpstat, if that fails it tries htop - it should get data like this Core 0: 15.34% Core 1: 23.12% Core 2: 18.76% Core 31: 10.95% then json encode that and send that to the webbrowser, which in turn reads it and shows the bars, if it dosent work can you copy your htop output and paste it here. It will help also with the m3u plugin. Thanks CPUMonitor.dll
mickle026 650 Posted May 14, 2025 Author Posted May 14, 2025 I will start a new thread with this soon, so it doesn't fill this one.
Apotropaic 41 Posted May 14, 2025 Posted May 14, 2025 Not sure everyone else will be happy you learning random linux stuff when you could be working on the M3U plugin!! The new version is doing something as I can see the page formatting/line spacing change randomly, but it doesn't display anything. This the output from the mpstat command. The htop command fails with the following message. htop: invalid option -- 'b' The -b switch may for the 'top' command, if I use that I get the following.
mickle026 650 Posted May 14, 2025 Author Posted May 14, 2025 55 minutes ago, mickle026 said: Ah, that's from /proc/stat, I don't have linux so a quick look up on bing gave me that. I did another look up after your post where you mention htop and found two options for getting cpu usadge I changed it to mpstat or htop, took 5 mins - not really wasted time its all learning which mpstat && mpstat -P ALL 1 1 or htop -d 1 -b | head -20 I found these commands, and same outputs so i changed the code to suit, it tries mpstat, if that fails it tries htop - it should get data like this Core 0: 15.34% Core 1: 23.12% Core 2: 18.76% Core 31: 10.95% then json encode that and send that to the webbrowser, which in turn reads it and shows the bars, if it dosent work can you copy your htop output and paste it here. It will help also with the m3u plugin. Thanks CPUMonitor.dll 59.5 kB · 1 download I'm a bit peeved with how bad multi threading actually is, the deeper you did the worse it is. Running different things single core together on Separate cores is fine, ie one thing on one core and another on another core, but true multi threading a single task across cores is extremely poorly supported. I need a break to get my thoughts together, when you walk away you get the idea eventually that's been hindering you while working hard on something. Thus distracting myself for a short time. Any chance you can copy and paste those images in text format?
Apotropaic 41 Posted May 14, 2025 Posted May 14, 2025 I think you've come across why the PS3 was so hard to write games for, even though modern graphics cards seem to do it well but I guess their workloads are consistent and well defined. Maybe just split the workload into alphabet chunks like I think you mentioned before and we just have to accept some of our cores will get used more than others, the playlists are always going to be weighted by the language used to name things which can't be controlled. I imagine in commercial applications they would give each item a reference/hash and that way you can ensure it's an equal workload, but then that would waste more CPU cycles doing all that....anyway enough about my ramblings, if it was that simple I'd be writing a plugin! /usr/bin/mpstat Linux 6.8.0-59-generic (asajjventress) 14/05/25 _x86_64_ (6 CPU) 13:52:43 CPU %usr %nice %sys %iowait %irq %soft %steal %guest %gnice %idle 13:52:44 all 1.17 0.00 0.17 0.17 0.00 0.00 0.00 0.00 0.00 98.50 13:52:44 0 2.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 98.00 13:52:44 1 1.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 99.00 13:52:44 2 2.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 98.00 13:52:44 3 0.00 0.00 1.00 0.00 0.00 0.00 0.00 0.00 0.00 99.00 13:52:44 4 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 100.00 13:52:44 5 2.00 0.00 0.00 1.00 0.00 0.00 0.00 0.00 0.00 97.00 Average: CPU %usr %nice %sys %iowait %irq %soft %steal %guest %gnice %idle Average: all 1.17 0.00 0.17 0.17 0.00 0.00 0.00 0.00 0.00 98.50 Average: 0 2.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 98.00 Average: 1 1.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 99.00 Average: 2 2.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 98.00 Average: 3 0.00 0.00 1.00 0.00 0.00 0.00 0.00 0.00 0.00 99.00 Average: 4 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 100.00 Average: 5 2.00 0.00 0.00 1.00 0.00 0.00 0.00 0.00 0.00 97.00 x@asajjventress:/home/emby$ top -d 1 -b | head -20 top - 13:53:44 up 5:05, 2 users, load average: 0.14, 0.06, 0.12 Tasks: 154 total, 1 running, 153 sleeping, 0 stopped, 0 zombie %Cpu(s): 0.0 us, 0.0 sy, 0.0 ni,100.0 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st MiB Mem : 7214.8 total, 1282.0 free, 4084.5 used, 2080.3 buff/cache MiB Swap: 2048.0 total, 1929.3 free, 118.7 used. 3130.2 avail Mem PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 4759 emby 20 0 6349216 980272 97140 S 20.0 13.3 6:51.85 EmbySer+ 1 root 20 0 22116 11236 8548 S 0.0 0.2 0:02.33 systemd 2 root 20 0 0 0 0 S 0.0 0.0 0:00.00 kthreadd 3 root 20 0 0 0 0 S 0.0 0.0 0:00.00 pool_wo+ 4 root 0 -20 0 0 0 I 0.0 0.0 0:00.00 kworker+ 5 root 0 -20 0 0 0 I 0.0 0.0 0:00.00 kworker+ 6 root 0 -20 0 0 0 I 0.0 0.0 0:00.00 kworker+ 7 root 0 -20 0 0 0 I 0.0 0.0 0:00.00 kworker+ 9 root 0 -20 0 0 0 I 0.0 0.0 0:00.00 kworker+ 12 root 0 -20 0 0 0 I 0.0 0.0 0:00.00 kworker+ 13 root 20 0 0 0 0 I 0.0 0.0 0:00.00 rcu_tas+ 14 root 20 0 0 0 0 I 0.0 0.0 0:00.00 rcu_tas+ 15 root 20 0 0 0 0 I 0.0 0.0 0:00.00 rcu_tas+
Recommended Posts
Create an account or sign in to comment
You need to be a member in order to leave a comment
Create an account
Sign up for a new account in our community. It's easy!
Register a new accountSign in
Already have an account? Sign in here.
Sign In Now