goin3d 4 Posted May 5, 2025 Posted May 5, 2025 thanks for looking into it! that's interesting! i hadn't thought to check the very first run.
mickle026 650 Posted May 8, 2025 Author Posted May 8, 2025 On 05/05/2025 at 18:44, goin3d said: thanks for looking into it! that's interesting! i hadn't thought to check the very first run. Just so you know, I am still looking into this. lots and lots of test runs later and I have narrowed it down. With your m3u and settings I am outputting consistently now . Test runs varied 56,788 items are added 55,939 are written each time 849 are not written I am still testing, but it appears that when multithreading (16 logical cpu's, ie 16 threads at once), asking the filesystem "does that file exist?" can cross talk to another thread and answer another threads same question for a different file. Thus seemingly randomising the output. Anyway the culprit is cross threading because of using all the cores assigned. Its near the end of such a big list that the errors start occurring. One fix is: Adding all the files to a task list then executing the task list works, but it is much slower so I am still researching a solution what should actually write is 56,788 files 203 are duplicates ------------------------------ 56,585 will write on a single core, or from a compiled task list. That is the 56,788 minus the duplicates. I am currently testing different write methods across the cores to see what is most reliable. It is just taking some time... 2
goin3d 4 Posted May 8, 2025 Posted May 8, 2025 2 minutes ago, mickle026 said: Just so you know, I am still looking into this. lots and lots of test runs later and I have narrowed it down. With your m3u and settings I am outputting consistently now . Test runs varied 56,788 items are added 55,939 are written each time 849 are not written I am still testing, but it appears that when multithreading (16 logical cpu's, ie 16 threads at once), asking the filesystem "does that file exist?" can cross talk to another thread and answer another threads same question for a different file. Thus seemingly randomising the output. Anyway the culprit is cross threading because of using all the cores assigned. Its near the end of such a big list that the errors start occurring. One fix is: Adding all the files to a task list then executing the task list works, but it is much slower so I am still researching a solution what should actually write is 56,788 files 203 are duplicates ------------------------------ 56,585 will write on a single core, or from a compiled task list. That is the 56,788 minus the duplicates. I am currently testing different write methods across the cores to see what is most reliable. It is just taking some time... Thank you for the update and thank you for the time you've put into this!!!!
sundevil67 28 Posted May 8, 2025 Posted May 8, 2025 still no worky for me .. goes through lengthy process, seems to be working, but never writes any files ScheduledTask-[M3UToStrm]_06.May.2025-14.54.43.txt.zip
mickle026 650 Posted May 9, 2025 Author Posted May 9, 2025 15 hours ago, sundevil67 said: still no worky for me .. goes through lengthy process, seems to be working, but never writes any files ScheduledTask-[M3UToStrm]_06.May.2025-14.54.43.txt.zip 1.95 MB · 2 downloads Its very strange that there is only your install not writing any output, can you send me your config file? you can download it from the server next to the bottom "save settings" button?
mickle026 650 Posted May 9, 2025 Author Posted May 9, 2025 This is to test only! This is working but I have not finished so the IO logging at the moment is in the normal log for the strms, it is using a completely different system to output the strm files. I am only sharing it so you can test it is outputting everything VODtoSTRMscheduledTask.dll
goin3d 4 Posted May 9, 2025 Posted May 9, 2025 I'm running some fresh test now. Understand this is all testing atm, but it looks like my first run is going to take a couple hours, vs. 12 or so minutes before. Will post results when i have them!
mickle026 650 Posted May 9, 2025 Author Posted May 9, 2025 55 minutes ago, goin3d said: I'm running some fresh test now. Understand this is all testing atm, but it looks like my first run is going to take a couple hours, vs. 12 or so minutes before. Will post results when i have them! Nothing has changed other than the writing, so only logging should make a difference. It takes 25 mins on my pc with the m3u you sent and logging on.
goin3d 4 Posted May 9, 2025 Posted May 9, 2025 More details. i'm trying two files and running them on two different machines. 1st file is the same source that you have. it is relatively small at under 65k items. Last version (the old method) the file took around 2 minutes to run, but we know it was missing stuff. Latest version took 10 minutes to run, and so far i don't see any misses/rewrites, when running the same file over and over. What i do notice is the first 55% of the scheduled task where the input file is being read is what takes much longer. (That may just be by design in how this new way processes). Overall the CPU utilization is way down from the pervious build. The 2nd file i'm using is much larger at around 300k items. Previously i think that took around 15 or so minutes. That one is still going at the 1 hour mark so we'll see what that result is.
Apotropaic 41 Posted May 9, 2025 Posted May 9, 2025 20 hours ago, sundevil67 said: still no worky for me .. goes through lengthy process, seems to be working, but never writes any files ScheduledTask-[M3UToStrm]_06.May.2025-14.54.43.txt.zip 1.95 MB · 4 downloads Hi, are you sure this is this the end of your log as it seems as if it's still running. On my system it shows similar messages while it's still processing but the actual system disk writes seem to be batched up towards the end of the plugin process. This is what I see when it is finally finished. 09.May.2025 - 14:00:31 Finished this import. 09.May.2025 - 14:00:31 Task Ended at: 09/05/2025 14:00:31 09.May.2025 - 14:00:31 Task Duration: 03:45:09.96 09.May.2025 - 14:00:31 Finished.
mickle026 650 Posted May 9, 2025 Author Posted May 9, 2025 34 minutes ago, goin3d said: More details. i'm trying two files and running them on two different machines. 1st file is the same source that you have. it is relatively small at under 65k items. Last version (the old method) the file took around 2 minutes to run, but we know it was missing stuff. Latest version took 10 minutes to run, and so far i don't see any misses/rewrites, when running the same file over and over. What i do notice is the first 55% of the scheduled task where the input file is being read is what takes much longer. (That may just be by design in how this new way processes). Overall the CPU utilization is way down from the pervious build. The 2nd file i'm using is much larger at around 300k items. Previously i think that took around 15 or so minutes. That one is still going at the 1 hour mark so we'll see what that result is. Nothing at all has changed in the way it reads the m3u file, it should take the same time as before and still does on my machine. It's weird how other setups seem to behave so different. All I have changed is the output files are written after the m3u is processed. So from about the 75% - 100% point. Basically the decision over what to write, overwrite or delete and how they are written. The progress bar hasn't even been thought about during these changes, but it might still work in 5% increments. The queue isn't being used for the strm files at the moment, they are being written as the decisions happen rather than all afterwards. I haven't altered the posters download or the removing of obsolete files or anything else. I just haven't had time. I've been almost living at my workplace.
fbrassin 47 Posted May 9, 2025 Posted May 9, 2025 I have a question. I would like to start Scan library After plugin finishes. Do you think i can set It in the script and put It in the plugin config? In this case Is there a terminal command to run It? Tnx
goin3d 4 Posted May 9, 2025 Posted May 9, 2025 not sure about the plugin part but i use the Emby Scripter X plugin to run scripts after a scheduled task complete. If you wanted a script to update all emby libraries you could: #!/bin/bash # Emby server configuration EMBY_URL="http://xxx.xxx.xxx.xxx:8096" API_KEY="EMBY API KEY" # Trigger full Emby library scan curl -s -X POST "$EMBY_URL/emby/Library/Refresh" \ -H "X-Emby-Token: $API_KEY" echo "Emby library refresh triggered at $(date)" 1
mickle026 650 Posted May 9, 2025 Author Posted May 9, 2025 (edited) 2 hours ago, goin3d said: not sure about the plugin part but i use the Emby Scripter X plugin to run scripts after a scheduled task complete. If you wanted a script to update all emby libraries you could: #!/bin/bash # Emby server configuration EMBY_URL="http://xxx.xxx.xxx.xxx:8096" API_KEY="EMBY API KEY" # Trigger full Emby library scan curl -s -X POST "$EMBY_URL/emby/Library/Refresh" \ -H "X-Emby-Token: $API_KEY" echo "Emby library refresh triggered at $(date)" I have added this into the plugin - it needs an api-key though, even running it from the server itself. Scanning All libraries triggers the progressbar in the dashboard - triggers a library refresh the same as goig3d does with the curl command But scanning the single path does not - triggers a library update (path only) 2025-05-10 00:06:21.277 Info Server: http/1.1 Response 204 to ::1. Time: 2ms. POST http://localhost:8096/emby/Library/Media/Updated?api_key=xxxxxxxxx with a post body This will only updated the path if the path is an Emby Library! Also note this will store the Api Key in plain text in your config files, so be careful about posting them. (not in the log though) This is what it will write to the log, so that is safe from being exposed in there. 09.May.2025 - 23:52:43 Finished this import. 09.May.2025 - 23:52:43 Task Ended at: 09/05/2025 23:52:43 09.May.2025 - 23:52:43 Task Duration: 00:00:01.60 09.May.2025 - 23:52:44 Triggering Full Library Scan Update 09.May.2025 - 23:52:44 Library refresh triggered successfully! 09.May.2025 - 23:52:44 Task completed. 09.May.2025 - 23:52:44 CreateMovieStrmsFromM3UTaskStarter method completed. 09.May.2025 - 23:52:44 Finished. VODtoSTRMscheduledTask.dll I have a week off work now, although I have other things to do I will be able to do some more on this. Edited May 9, 2025 by mickle026 1
goin3d 4 Posted May 10, 2025 Posted May 10, 2025 did you put the library update ability in the version with your new processing method?
mickle026 650 Posted May 10, 2025 Author Posted May 10, 2025 (edited) 51 minutes ago, goin3d said: did you put the library update ability in the version with your new processing method? In the one posted above your last post (2 above this one) build 1.0.0.175 I already knew how to do the updating, so I just added it in because it seemed as if it was something you guys wanted based on your last few posts about triggering scripts to update. 51 minutes ago, goin3d said: new processing method only the file output writing is new and a couple of log entries at the start of the log, I don't know why its running slower on @jaspalgillsetup than it was before, if anything it should be a little faster as it starts writing earlier than it did before and the write tasks fire off on different threads and are not waited for completion before triggering the next. 10.May.2025 - 00:27:29 Write TVShow strms: True 10.May.2025 - 00:27:29 Write Movie strms: True 10.May.2025 - 00:27:29 Write LiveTV strms: False These are at the top of the log, so I could see if @sundevil67 had no output selected or was misunderstanding something in the config. Edited May 10, 2025 by mickle026 1
Apotropaic 41 Posted May 10, 2025 Posted May 10, 2025 I don't mind the speed right now, the plugin is working and reliable enough so when I trigger it overnight I know it will have done its thing! I have a feeling the speed is down to memory, on my other previous system (Ubuntu 22.04) it used to allocate all available memory I had, on Ubuntu 24.04 for me at least it seems overly conservative. I now have more memory available to emby/plugin but for some reason it is using less than before. I'm still on version .170 as I'm playing around with HLS-Proxy and re-working my libraries, once that's done I'll do some testing with the latest plugin version. Again thanks for all the effort you put into this @mickle026.
fbrassin 47 Posted May 10, 2025 Posted May 10, 2025 With lastest version i think this option is not working It seems both files Movies-Obfuscated.m3u and Movies.m3u are the same.
goin3d 4 Posted May 11, 2025 Posted May 11, 2025 I've added a 3rd file to my run to test out and i've been having trouble loading it. If i download the M3U locally it runs, but if i use the web address it fails to load: 10.May.2025 - 23:32:49 Include [Contains]: 10.May.2025 - 23:32:49 Include [StartsWith]: 10.May.2025 - 23:32:49 Use URL for tvgtype detection: True 10.May.2025 - 23:32:49 Task Started at: 5/10/2025 11:32:49 PM 10.May.2025 - 23:32:49 ----------------------------------------------------------------------- 10.May.2025 - 23:32:49 Create Movie Stream Files From M3U Scheduled Task Triggered. 10.May.2025 - 23:32:49 ----------------------------------------------------------------------- 10.May.2025 - 23:32:49 Input File 1: https://xxx/get.php?username=xxx&password=xxx&type=m3u_plus&output=mpegts 10.May.2025 - 23:32:49 Loading M3U file. 10.May.2025 - 23:32:49 Attempting to download file: https://xxx.xxx/get.php?username=xxxx&password=xxxxx&type=m3u_plus&output=mpegts 10.May.2025 - 23:32:49 Failed to download file: https://xxxx.xxx/get.php?username=xxxx&password=xxxxx&type=m3u_plus&output=mpegts. Error: Name or service not known (xx.xx:443) 10.May.2025 - 23:32:49 No lines found in the M3U file. 10.May.2025 - 23:32:49 Debugging: IsAoolloTV=False 10.May.2025 - 23:32:49 No Input File 2 10.May.2025 - 23:32:49 No Input File 3 10.May.2025 - 23:32:49 Total Input Files: 1 10.May.2025 - 23:32:49 Total m3u Items: 0 10.May.2025 - 23:32:49 Loading m3u File failed, exiting. 10.May.2025 - 23:32:49 Task Ended at: 5/10/2025 11:32:49 PM 10.May.2025 - 23:32:49 Task Duration: 00:00:00.03 In looking at the M3Us i've been running this one has a 2nd line in it that the others don't (in bold below). Could that be causing it to fail? It's really weird that using the URL fails, but using that same URL to download the file locally and then run it works. #EXTM3U #EXT-X-SESSION-DATA:DATA-ID="com.xui.1_5_13" #EXTINF:-1 tvg-id="somename" tvg-name="somename" tvg-logo="https://abc.123:443/images/LjMxooJdLoUjs_tBLxgrV0mr491X6WYzBJN9ILOmpaO-l9-Z28XxT-gT2DScS1c-.png" group-title="4K / UHD",somename https://xxx.xx:443/xxx/xxxx/abc123
mickle026 650 Posted May 11, 2025 Author Posted May 11, 2025 (edited) 3 hours ago, goin3d said: Error: Name or service not known This means that the name or url was not contactable, In other words, your computer tried to reach a website or server, but it didn’t recognize the name or couldn’t find the corresponding address. Can you ping that address and port? Typo maybe? The timeout for downloading is set at 10 minutes per file port 443 is a secure port for encrypting the data set back from the server - maybe ssl or tls. It could also be that specific headers are required to download from that site/port and the plugin might not be using them If you can capture the headers from the web browser F12 console that will help me check that, response and request to do this, press F12 before loading the page or reload it when the console is open. When the page has loaded or the browser starts to download the file, click the link to the file in the console and copy the headers on the right pane For your security, this data will be better sent via pm rather than posting it here. This is an example (this site) { "responseHeaders": { "headers": [ { "name": "Cache-Control", "value": "no-cache=\"Set-Cookie\", max-age=900, public, s-maxage=900, stale-while-revalidate, stale-if-error" }, { "name": "Connection", "value": "keep-alive" }, { "name": "Content-Encoding", "value": "gzip" }, { "name": "Content-Length", "value": "17409" }, { "name": "Content-Security-Policy", "value": "frame-ancestors 'self'" }, { "name": "Content-Security-Policy", "value": "upgrade-insecure-requests;" }, { "name": "Content-Type", "value": "text/html;charset=UTF-8" }, { "name": "Date", "value": "Sun, 11 May 2025 07:38:30 GMT" }, { "name": "Expires", "value": "Sun, 11 May 2025 07:53:30 GMT" }, { "name": "Last-Modified", "value": "Sun, 11 May 2025 07:38:30 GMT" }, { "name": "Referrer-Policy", "value": "strict-origin-when-cross-origin" }, { "name": "Server", "value": "nginx" }, { "name": "Vary", "value": "Cookie,Accept-Encoding" }, { "name": "X-Content-Security-Policy", "value": "frame-ancestors 'self'" }, { "name": "X-Frame-Options", "value": "sameorigin" }, { "name": "X-IPS-LoggedIn", "value": "0" }, { "name": "X-XSS-Protection", "value": "0" } ] } } { "requestHeaders": { "headers": [ { "name": "Accept", "value": "text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8" }, { "name": "Accept-Encoding", "value": "gzip, deflate, br, zstd" }, { "name": "Accept-Language", "value": "en-US,en;q=0.5" }, { "name": "Connection", "value": "keep-alive" }, { "name": "Cookie", "value": "ips4_ipsTimezone=Europe/London; ips4_hasJS=true; ips4_IPSSessionFront=ck68l09d6rbg5n9ou93kq8qadg" }, { "name": "Host", "value": "emby.media" }, { "name": "If-Modified-Since", "value": "Sun, 11 May 2025 07:37:42 GMT" }, { "name": "Priority", "value": "u=0, i" }, { "name": "Sec-Fetch-Dest", "value": "document" }, { "name": "Sec-Fetch-Mode", "value": "navigate" }, { "name": "Sec-Fetch-Site", "value": "none" }, { "name": "Sec-Fetch-User", "value": "?1" }, { "name": "Upgrade-Insecure-Requests", "value": "1" }, { "name": "User-Agent", "value": "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:138.0) Gecko/20100101 Firefox/138.0" } ] } } 3 hours ago, goin3d said: #EXT-X-SESSION-DATA:DATA-ID="com.xui.1_5_13" This line will not make it fail, the m3u once downloaded is "cleaned" of entries like this, so that all that is in the file when it loads is the extinf line and the urls private string[] CleanLines(string[] lines, string currentlog, IO.FileLogging log, PluginConfiguration config, CancellationToken cancellationToken) { // Check for cancellation if (cancellationToken.IsCancellationRequested) { log.LogToMyFile( currentlog, DateTime.Now.ToString("dd.MMM.yyy - HH':'mm':'ss") + " Task was cancelled.", 2 ).ConfigureAwait(false); cancellationToken.ThrowIfCancellationRequested(); } return lines.Where(line => { string trimmedLine = line?.Trim(); HashSet<string> EqualsWholeStrings = new HashSet<string> { "#EXTINF", "#EXTM3U", "#EXTINF:-1", "#EXTINF:0" }; HashSet<string> StartsWithStrings = new HashSet<string>(StringComparer.OrdinalIgnoreCase) { "#EXTM3U", "#EXTGRP:", "rem", "#EXT-X", "#EXTENC", "#EXT-X-STREAM-INF", "#EXT-X-MEDIA", "#EXT-X-PLAYLIST", "#PLAYLIST", "#EXTGRP", "#EXTALB", "#EXTART", "#EXTGENRE", "#EXTM3A", "#EXTBYT", "#EXTBIN", "#EXTIMG" }; // Check if the line is in the set of whole strings if (EqualsWholeStrings.Contains(trimmedLine)) return false; // Check if the line starts with any of the specified strings if (StartsWithStrings.Any(s => trimmedLine.StartsWith(s, StringComparison.OrdinalIgnoreCase))) return false; // Remove empty or whitespace lines if (string.IsNullOrWhiteSpace(trimmedLine)) return false; // Retain the line if it passes all conditions return true; }).ToArray(); } Edited May 11, 2025 by mickle026
goin3d 4 Posted May 11, 2025 Posted May 11, 2025 (edited) Thanks! I’ll try that. Edited May 11, 2025 by goin3d
mickle026 650 Posted May 11, 2025 Author Posted May 11, 2025 (edited) Although the plugin does upgrade secure requests, it might need a security protocol such as TLS explicitly defined. I have come across this before in another app, but not needed to switch on/off on demand. I should be able to tell from the headers... I sent a DLL via pm for you to test Edited May 11, 2025 by mickle026
goin3d 4 Posted May 11, 2025 Posted May 11, 2025 (edited) Thanks! that fixed it up for me! Doing a full run now of all 3 of my connections. Not sure how long it will take but we'll see what happens! In the latest versions is the reading of streams now single core? can't remember if i imagined that or you had mentioned it before? Edited May 11, 2025 by goin3d
mickle026 650 Posted May 11, 2025 Author Posted May 11, 2025 11 minutes ago, goin3d said: Thanks! that fixed it up for me! Doing a full run now of all 3 of my connections. Not sure how long it will take but we'll see what happens! In the latest versions is the reading of streams now single core? can't remember if i imagined that or you had mentioned it before? No, I havent changed that, its still multicore, i have only changed the writing of the strms, and now the http downloader to support secure sockets and tls in the version sent via pm
mickle026 650 Posted May 11, 2025 Author Posted May 11, 2025 On 10/05/2025 at 13:07, fbrassin said: With lastest version i think this option is not working It seems both files Movies-Obfuscated.m3u and Movies.m3u are the same. I have took a look at that, it was only working if the url had /tvshow/ in it, I will release an update as soon as i cant test the changes. My server is busy at the moment so i cannot restart it to load the new DLL yet. 1
Recommended Posts
Create an account or sign in to comment
You need to be a member in order to leave a comment
Create an account
Sign up for a new account in our community. It's easy!
Register a new accountSign in
Already have an account? Sign in here.
Sign In Now