Plugin: Export VODS from .m3u - split m3u livetv, tvshows, movies (Going BETA!!)

[mickle026 650]

Please test this one and see if the re-writes are cured.

 

VODtoSTRMscheduledTask.dll

[goin3d 4]

it's much faster on the subsequent runs now!  still see a lot of rewrites though.  Here's one example:  

Edited April 29, 2025 by GrimReaper
Attachments removed

[fbrassin 47]

After first run i had many rewrites.

Now i'm running a second time to see if something changes.

[mickle026 650]

7 hours ago, fbrassin said:

After first run i had many rewrites.

Now i'm running a second time to see if something changes.

first run probably would cause re-writes, and the second run?

[fbrassin 47]

I'm sorry to tell you that also after second run i have many rewrites.

I would say the same rewrites as the first run.

[fbrassin 47]

Moreover, with this new version after running the plugin emby becames unresponding.

It happened after the first run, but i thought it was casually, but it happened also after second and third time a run this latest version.

The third time i run it on an empty dir.

[Apotropaic 41]

1 hour ago, fbrassin said:

Moreover, with this new version after running the plugin emby becames unresponding.

It happened after the first run, but i thought it was casually, but it happened also after second and third time a run this latest version.

The third time i run it on an empty dir.

I'm getting the same issue, I'm presuming @mickle026hasn't enabled the low memory function so it's running wild

The memory usage seems to jump when it's at the 'Updating Output Folder' stage. It might just be the way memory management works in linux systems when compared to windows, as well as a function of how much memory your system has and the size of your M3U file. I'm sure if we all had 32gb we would be fine!

My system is in a loop where the memory management (kswapd0) is using 100% of the CPU just moving memory around, which is starving emby of CPU, which may explain why on some systems the scans take so long. I'll let this try and finish, then give my VM a bit more memory and see what happens.

Update: After 20 minutes emby is getting around 33-35% CPU as kswapd0 is now hovering around 97-100% but the emby GUI is still unresponsive. Also, I'm only running the first log events debug setting. No additional debugging.

[mickle026 650]

updated output settings to use memory & cpu limiter when the limit option is selected

within this function, I simply set to use 80% cores (or nearest value) instead of wait for resources for now as it is less complicated and will work on all systems
 

29.Apr.2025 - 12:51:16 Debugging: Setting Filesystem Update to 12 of 16 cores/threads to prevent system freeze.
29.Apr.2025 - 12:51:16 Debugging: StrmsToWriteBag.Count=0
29.Apr.2025 - 12:51:16 Starting File Writer: 0 Items to write/update.
29.Apr.2025 - 12:51:16 All write tasks completed.
29.Apr.2025 - 12:51:16 Debugging: Fetch Images: False.
29.Apr.2025 - 12:51:16 Debugging: Remove Obsolete Files: True.
29.Apr.2025 - 12:51:16 Debugging: Files to check: 2353.
29.Apr.2025 - 12:51:16 Debugging: Remove Obsolete Files - Progress: 4.04%, Memory used: 34 MB.
29.Apr.2025 - 12:51:16 Debugging: Remove Obsolete Files - Progress: 9.08%, Memory used: 34 MB.
29.Apr.2025 - 12:51:16 Debugging: Remove Obsolete Files - Progress: 14.12%, Memory used: 34 MB.
29.Apr.2025 - 12:51:16 Debugging: Remove Obsolete Files - Progress: 19.16%, Memory used: 34 MB.
29.Apr.2025 - 12:51:16 Debugging: Remove Obsolete Files - Progress: 24.20%, Memory used: 34 MB.
29.Apr.2025 - 12:51:16 Debugging: Remove Obsolete Files - Progress: 29.28%, Memory used: 34 MB.
29.Apr.2025 - 12:51:16 Debugging: Remove Obsolete Files - Progress: 34.32%, Memory used: 34 MB.
29.Apr.2025 - 12:51:16 Debugging: Remove Obsolete Files - Progress: 39.36%, Memory used: 34 MB.
29.Apr.2025 - 12:51:16 Debugging: Remove Obsolete Files - Progress: 44.40%, Memory used: 34 MB.
29.Apr.2025 - 12:51:16 Debugging: Remove Obsolete Files - Progress: 49.44%, Memory used: 34 MB.
29.Apr.2025 - 12:51:16 Debugging: Remove Obsolete Files - Progress: 54.48%, Memory used: 34 MB.
29.Apr.2025 - 12:51:16 Debugging: Remove Obsolete Files - Progress: 59.56%, Memory used: 34 MB.
29.Apr.2025 - 12:51:16 Debugging: Remove Obsolete Files - Progress: 64.60%, Memory used: 34 MB.
29.Apr.2025 - 12:51:17 Debugging: Remove Obsolete Files - Progress: 69.64%, Memory used: 34 MB.
29.Apr.2025 - 12:51:17 Debugging: Remove Obsolete Files - Progress: 74.68%, Memory used: 34 MB.
29.Apr.2025 - 12:51:17 Debugging: Remove Obsolete Files - Progress: 79.72%, Memory used: 34 MB.
29.Apr.2025 - 12:51:17 Debugging: Remove Obsolete Files - Progress: 84.96%, Memory used: 34 MB.
29.Apr.2025 - 12:51:17 Debugging: Remove Obsolete Files - Progress: 90.00%, Memory used: 34 MB.
29.Apr.2025 - 12:51:17 Debugging: Remove Obsolete Files - Progress: 95.04%, Memory used: 34 MB.
29.Apr.2025 - 12:51:17 Starting FileRemover: 0 Items to delete.
29.Apr.2025 - 12:51:17 Checking for folder deletion Tasks.
29.Apr.2025 - 12:51:17 Skipped root folder: D:\VODS\ (contains .strm files or subfolders containing them).
29.Apr.2025 - 12:51:17 All delete tasks completed.
29.Apr.2025 - 12:51:17 All Updating tasks completed.
29.Apr.2025 - 12:51:17 Finished this import.

 

VODtoSTRMscheduledTask.dll

 

 

[sundevil67 28]

I'm still getting tuck around 15%

[Apotropaic 41]

5 hours ago, mickle026 said:

updated output settings to use memory & cpu limiter when the limit option is selected

within this function, I simply set to use 80% cores (or nearest value) instead of wait for resources for now as it is less complicated and will work on all systems
 

Something in the .167 build is crashing Emby, I've tried it a few times now and sometimes the emby process dies after 20-30 minutes or sometimes the server hangs. Note: this is for the first initial run with empty directories. I don't have a test setup so I'll wait until tonight to do another run with a debug turned on.

This is probably why I should setup a test environment!

[mickle026 650]

20 minutes ago, sundevil67 said:

I'm still getting tuck around 15%

Are you logging it?

I still don't know if its downloading all the m3u parts, you have never let me know?

10 minutes ago, jaspalgill said:

Something in the .167 build is crashing Emby, I've tried it a few times now and sometimes the emby process dies after 20-30 minutes or sometimes the server hangs. Note: this is for the first initial run with empty directories. I don't have a test setup so I'll wait until tonight to do another run with a debug turned on.

This is probably why I should setup a test environment!

What is it doing when it crashes, where does it get to?

[Apotropaic 41]

Just now, mickle026 said:

What is it doing when it crashes, where does it get to?

I only have the 'Log Operations' ticked right now, it gets to this stage within 3 minutes of the start. It's at this point the CPU/memory usage jumps and the crashes happen around 20 minutes after. I've just started another run but with Verbose & Debug on.

29.Apr.2025 - 16:42:47 -----------------------[ Scanning Current Folders ]----------------------------

29.Apr.2025 - 16:42:47 Memory used: 792 MB

29.Apr.2025 - 16:42:47 Scanning Output Directory for content: /home/emby/Testing/Test3a/Library.

29.Apr.2025 - 16:42:47 Debugging: Attempt 1: scannedcount = -1, strmfilescount = 0

29.Apr.2025 - 16:42:47 Found 0 folders in Output path. Scanning...

29.Apr.2025 - 16:42:48 The output directory is empty.

29.Apr.2025 - 16:42:48 ----------------------[ Reading M3U File Contents ]----------------------------

29.Apr.2025 - 16:45:39 Saved '/home/emby/Testing/Test3a/M3U/LiveTV/LiveTV.m3u'

29.Apr.2025 - 16:45:39 Saved '/home/emby/Testing/Test3a/M3U/Movies/Movies.m3u'

29.Apr.2025 - 16:45:40 Saved '/home/emby/Testing/Test3a/M3U/TV Shows/TV Shows.m3u'

29.Apr.2025 - 16:45:40 Unknown M3U Items found: 0.

29.Apr.2025 - 16:45:40 Debugging: M3U file processing completed.

29.Apr.2025 - 16:45:40 Debugging: [scanM3UFileAsync] m3uContents Contains: 317049 Items.

29.Apr.2025 - 16:45:40 -----------------------[ Updating Output Folder ]-------------------------------

[Apotropaic 41]

I started the task at 18:36 and at 19:07 it stopped logging anything new, I checked the sever at 21:00 and it was unresponsive. I had the following debug settings.

Task started at 29.Apr.2025-18:36:04.

29.Apr.2025 - 19:07:05 Debugging: [AddToContentListAsync] Memory used: 799 MB

29.Apr.2025 - 19:07:05 [AddToContentListAsync] Adding to m3uContents - Count = 317056

29.Apr.2025 - 19:07:05 ======================================

29.Apr.2025 - 19:07:05 Saved '/home/emby/Testing/Test3a/M3U/LiveTV/LiveTV.m3u'

29.Apr.2025 - 19:07:05 Live TV M3U Items found: 10332.

29.Apr.2025 - 19:07:05 Saved '/home/emby/Testing/Test3a/M3U/Movies/Movies.m3u'

29.Apr.2025 - 19:07:05 Movies M3U Items found: 35630.

29.Apr.2025 - 19:07:05 Saved '/home/emby/Testing/Test3a/M3U/TV Shows/TV Shows.m3u'

29.Apr.2025 - 19:07:05 TV Show M3U Items found: 271094.

29.Apr.2025 - 19:07:05 Unknown M3U Items found: 0.

29.Apr.2025 - 19:07:05 ======================================

29.Apr.2025 - 19:07:05 Debugging: Memory used: 799 MB

29.Apr.2025 - 19:07:05 Debugging: Time taken to process M3U file: 00:30:32

29.Apr.2025 - 19:07:05 Debugging: M3U file processing completed.

29.Apr.2025 - 19:07:05 Debugging: [scanM3UFileAsync] m3uContents Contains: 317056 Items.

29.Apr.2025 - 19:07:05 -----------------------[ Updating Output Folder ]-------------------------------

Edited April 29, 2025 by jaspalgill

[mickle026 650]

3 hours ago, jaspalgill said:

I started the task at 18:36 and at 19:07 it stopped logging anything new, I checked the sever at 21:00 and it was unresponsive. I had the following debug settings.

Task started at 29.Apr.2025-18:36:04.

29.Apr.2025 - 19:07:05 Debugging: [AddToContentListAsync] Memory used: 799 MB

29.Apr.2025 - 19:07:05 [AddToContentListAsync] Adding to m3uContents - Count = 317056

29.Apr.2025 - 19:07:05 ======================================

29.Apr.2025 - 19:07:05 Saved '/home/emby/Testing/Test3a/M3U/LiveTV/LiveTV.m3u'

29.Apr.2025 - 19:07:05 Live TV M3U Items found: 10332.

29.Apr.2025 - 19:07:05 Saved '/home/emby/Testing/Test3a/M3U/Movies/Movies.m3u'

29.Apr.2025 - 19:07:05 Movies M3U Items found: 35630.

29.Apr.2025 - 19:07:05 Saved '/home/emby/Testing/Test3a/M3U/TV Shows/TV Shows.m3u'

29.Apr.2025 - 19:07:05 TV Show M3U Items found: 271094.

29.Apr.2025 - 19:07:05 Unknown M3U Items found: 0.

29.Apr.2025 - 19:07:05 ======================================

29.Apr.2025 - 19:07:05 Debugging: Memory used: 799 MB

29.Apr.2025 - 19:07:05 Debugging: Time taken to process M3U file: 00:30:32

29.Apr.2025 - 19:07:05 Debugging: M3U file processing completed.

29.Apr.2025 - 19:07:05 Debugging: [scanM3UFileAsync] m3uContents Contains: 317056 Items.

29.Apr.2025 - 19:07:05 -----------------------[ Updating Output Folder ]-------------------------------

Can you run again with IO logging on and inspect that to see if file writes are failing so we might have a clue why it is stopping?  That might possibly have a clue in it.

[sundevil67 28]

7 hours ago, mickle026 said:

Are you logging it?

I still don't know if its downloading all the m3u parts, you have never let me know?

What is it doing when it crashes, where does it get to?

I went back to just trying a single M3U link, as I haven't been able to run the plugin successfully since I tried that first time to do more than one. I thought I mentioned it in our PMs; if not sorry. When I run it, and it hangs, then stop it, it says "stopping" forever, at least until I restart the server a bunch of times. 

It gets to about 15%-ish most of the time, but it varies. 

ScheduledTask-[M3UToStrm]29.Apr.2025 - 12.52.21.txt.zip

[mickle026 650]

4 hours ago, sundevil67 said:

I went back to just trying a single M3U link, as I haven't been able to run the plugin successfully since I tried that first time to do more than one. I thought I mentioned it in our PMs; if not sorry. When I run it, and it hangs, then stop it, it says "stopping" forever, at least until I restart the server a bunch of times. 

It gets to about 15%-ish most of the time, but it varies. 

ScheduledTask-[M3UToStrm]29.Apr.2025 - 12.52.21.txt.zip 1.72 MB · 1 download

Cool to see that it is downloading all the m3u files!

29.Apr.2025 - 12:52:22 Input File 1: https://****************/1
29.Apr.2025 - 12:52:22 Input File 1 is an Multi URL - attempting to get them all.
29.Apr.2025 - 12:52:23 Input File 1: https://****************/1
29.Apr.2025 - 12:52:23 Imput m3u Lines: 0
29.Apr.2025 - 12:52:24 Loading M3U file.
29.Apr.2025 - 12:52:24 Attempting to download file: https://****************/1
29.Apr.2025 - 12:52:25 Download Progress: 10%
29.Apr.2025 - 12:52:25 Download Progress: 20%
29.Apr.2025 - 12:52:25 Download Progress: 30%
29.Apr.2025 - 12:52:25 Download Progress: 40%
29.Apr.2025 - 12:52:25 Download Progress: 50%
29.Apr.2025 - 12:52:25 Download Progress: 60%
29.Apr.2025 - 12:52:25 Download Progress: 70%
29.Apr.2025 - 12:52:25 Download Progress: 80%
29.Apr.2025 - 12:52:25 Download Progress: 90%
29.Apr.2025 - 12:52:25 Download Progress: 100%
29.Apr.2025 - 12:52:25 Downloading & Loading m3u File Total execution time: 1770 ms
29.Apr.2025 - 12:52:25 Debugging: Input File 1 returned 10000 new lines.
29.Apr.2025 - 12:52:25 Debugging: Current Input m3u Lines: 10000
29.Apr.2025 - 12:52:27 Input File 2: https://****************/2
29.Apr.2025 - 12:52:27 Imput m3u Lines: 10000
29.Apr.2025 - 12:52:27 Loading M3U file.
29.Apr.2025 - 12:52:27 Attempting to download file: https://****************/2
29.Apr.2025 - 12:52:29 Download Progress: 10%
29.Apr.2025 - 12:52:29 Download Progress: 20%
29.Apr.2025 - 12:52:29 Download Progress: 30%
29.Apr.2025 - 12:52:29 Download Progress: 40%
29.Apr.2025 - 12:52:29 Download Progress: 50%
29.Apr.2025 - 12:52:29 Download Progress: 60%
29.Apr.2025 - 12:52:29 Download Progress: 70%
29.Apr.2025 - 12:52:29 Download Progress: 80%
29.Apr.2025 - 12:52:29 Download Progress: 90%
29.Apr.2025 - 12:52:29 Download Progress: 100%
29.Apr.2025 - 12:52:29 Downloading & Loading m3u File Total execution time: 1944 ms
29.Apr.2025 - 12:52:29 Debugging: Input File 2 returned 10000 new lines.
29.Apr.2025 - 12:52:29 Debugging: Current Input m3u Lines: 20000
29.Apr.2025 - 12:52:30 Input File 3: https://****************/3
29.Apr.2025 - 12:52:30 Imput m3u Lines: 20000
29.Apr.2025 - 12:52:30 Loading M3U file.
29.Apr.2025 - 12:52:30 Attempting to download file: https://****************/3
29.Apr.2025 - 12:52:31 Download Progress: 10%
29.Apr.2025 - 12:52:31 Download Progress: 20%
29.Apr.2025 - 12:52:31 Download Progress: 30%
29.Apr.2025 - 12:52:31 Download Progress: 40%
29.Apr.2025 - 12:52:31 Download Progress: 50%
29.Apr.2025 - 12:52:31 Download Progress: 60%
29.Apr.2025 - 12:52:31 Download Progress: 70%
29.Apr.2025 - 12:52:32 Download Progress: 80%
29.Apr.2025 - 12:52:32 Download Progress: 90%
29.Apr.2025 - 12:52:32 Download Progress: 100%
29.Apr.2025 - 12:52:32 Downloading & Loading m3u File Total execution time: 1271 ms
29.Apr.2025 - 12:52:32 Debugging: Input File 3 returned 10000 new lines.
29.Apr.2025 - 12:52:32 Debugging: Current Input m3u Lines: 30000

Did it stop/hang at this point, or is this log truncated (cut) to show me it downloading? 

Does it stop here every-time?

Are you running emby in a container?

29.Apr.2025 - 17:57:11 =============== Item [9108/140278] ===== m3u read: 6.49%, ReadTime:05:03:06 ==============
29.Apr.2025 - 17:57:11 Debugging: Sanitize InputString: 'Big Nate (2022)'

There is still an issue with your m3u because  'Big Nate (2022)' has many entries all, the same.  The episode and season are not detected and I think it might be adding tv-shows as live-tv.
This should not stop it processing though ....

Edited April 30, 2025 by mickle026

[BigIrn 10]

Wow this took off… awesome!

[mickle026 650]

Some corrections for identifying apollo tv - tv shows

30.Apr.2025 - 06:12:28 Debugging AddToContentListAsync: folderName=D:\VODS\sundevil67\TV Shows\The Hunting Party (2025)\tt21906554\Season 01, fileName=The Hunting Party (2025) [tt21906554] S01E05.strm, fileContents=https://providerURL/username/password/tvshow/tt21906554/1/5


Changed the order of updating file output to delete obsolete files before writing new files

VODtoSTRMscheduledTask.dll

[BigIrn 10]

Not sure what I did wrong. ver 167. It is not writing to an empty new folder (c:vods).

ScheduledTask-[M3UToStrm]_29.Apr.2025-22.38.13_Part[6].zip

Edited April 30, 2025 by BigIrn

[mickle026 650]

3 hours ago, BigIrn said:

Not sure what I did wrong. ver 167. It is not writing to an empty new folder (c:vods).

ScheduledTask-[M3UToStrm]_29.Apr.2025-22.38.13_Part[6].zip 54.63 MB · 1 download

****Item Ignored*** (Not in Includes)

Everything is filtered out by your Includes, so you are looking for only this/these item(s) and do not have any.  What is your config here?

I think you have replace spaces with underscores ON, so you probably need to add "EN_,EN_|,US_,US_|" etc to includes, the same variants with no spaces but underscores
Example:
EN ,EN -,EN|,EN_,US ,US|,US_,UK,|EN|,EN_|,US_|

I would need to check though, but no time right now.

Edited April 30, 2025 by mickle026

[BigIrn 10]

In your code can there be a way to ignore a url string based on filters before it looks up the meta data?

[mickle026 650]

20 minutes ago, BigIrn said:

In your code can there be a way to ignore a url string based on filters before it looks up the meta data?

can you give me an example?

I am not sure what you mean by this question so an example would help.

[BigIrn 10]

11 minutes ago, mickle026 said:

can you give me an example?

I am not sure what you mean by this question so an example would help.

What is the order of operation? Does it pull metadata first before it determines whether to keep or ignore the stream? Why not exclude first before sending an api call if it will ultimately not be used? I hope that makes sense. 

[mickle026 650]

3 hours ago, BigIrn said:

What is the order of operation? Does it pull metadata first before it determines whether to keep or ignore the stream? Why not exclude first before sending an api call if it will ultimately not be used? I hope that makes sense. 

So basically you would like the underscores applying after the includes/ignores?

[BigIrn 10]

4 hours ago, mickle026 said:

So basically you would like the underscores applying after the includes/ignores?

Yeah. Exclude and retain the strms that are going to be outputted AFTER all of the filtering and THEN do all the file/string changes/manipulation, especially any api calls if we use the TMDB. I don't know if this is possible but just a thought. I'm running the latest version and it's at 54% and we are at the 10 hour mark. I'm trying to convert 1,153,232 possible urls and I have all the logging turned on so I know it will take a long time.