Plugin: Export VODS from .m3u - split m3u livetv, tvshows, movies (Going BETA!!)

[BigIrn 10]

46 minutes ago, gillmacca01 said:

This was run on my laptop. I currently work away from home all week, and don't have access to the root of my QNAP to update the dll.

Maybe I will give it one more try, but on my QNAP at the weekend 

I’m using a ssd for storage and a 1gb connection from home. Are you remoting in or piped to your machine? I’ve seen people complain bc it was slow but the file was moving from source/remote machine vpn/home storage. The file going back and forth instead of direct. 

[fbrassin 44]

Also for me took so much.

I started yestarday ar 6 p.m. and ended at about 5 a.m. this morning.

I see no tv show saved.

In tv show folder i have only movies not recognised  as movie, but no tv show.

I don't know whats wrong.

Tv show.m3u created has series inside it.

[gillmacca01 179]

3 hours ago, BigIrn said:

I’m using a ssd for storage and a 1gb connection from home. Are you remoting in or piped to your machine? I’ve seen people complain bc it was slow but the file was moving from source/remote machine vpn/home storage. The file going back and forth instead of direct. 

I have Emby installed on my laptop for testing, so apart from retrieving the M3U everything else is done on the laptop

[gillmacca01 179]

4 hours ago, mickle026 said:

Testing async log writing now, and its a huge increase in speed here on this Ryzen 7 cpu.

749,999+ items processed 17 minutes! Its using all 16 cores of my cpu but only using at 26% usage.

The files have not been written yet, that's a slow part.

But the log is a lot more out of order.

 

05.Mar.2025 - 05:05:27 -----------------------------------------------------------------------
05.Mar.2025 - 05:05:27 Create Movie Stream Files From M3U Scheduled Task Triggered.
05.Mar.2025 - 05:05:27 -----------------------------------------------------------------------
05.Mar.2025 - 05:05:27 Input File 1: E:\tv_channels_b3e6755ebfe2.m3u
05.Mar.2025 - 05:05:27 -----------------------[ Scanning Current Folders ]----------------------------
05.Mar.2025 - 05:05:27 ----------------------[ Reading M3U File Contents ]----------------------------
05.Mar.2025 - 05:05:27 --------------------------------------------------------------------------------
05.Mar.2025 - 05:05:27 Checking to see if input path is an URL.
05.Mar.2025 - 05:05:28 Attempting to read input file from local.....
05.Mar.2025 - 05:05:28 Valid m3u linecount: 2181304
.
.
.
05.Mar.2025 - 05:10:51 Item [233083] Detected Season: 5, Episode: 8, Setting to Type: TV Shows
05.Mar.2025 - 05:10:51 Item [233084] Detected Season: 5, Episode: 9, Setting to Type: TV Shows
05.Mar.2025 - 05:10:51 Item [233084] [VOD] Adding to internal M3UContent List folderName: TV Shows\EN - Blue Bloods (2010) (US)
05.Mar.2025 - 05:10:49 Item [232378] [VOD] Adding to internal M3UContent List tvgType: TV Shows
.
.
.
05.Mar.2025 - 05:11:35 Item [245907] [VOD] Adding to internal M3UContent List fileName: EN - Ally McBeal (1997) (US) S02 E20.strm
05.Mar.2025 - 05:11:33 Item [245503] [VOD] Adding to internal M3UContent List tvgType: TV Shows
05.Mar.2025 - 05:11:33 Item [245500] [VOD] Adding to internal M3UContent List folderName: TV Shows\EN - The Tick (2016) (US)
05.Mar.2025 - 05:11:33 Item [245395] Detected Season: 2, Episode: 1, Setting to Type: TV Shows
05.Mar.2025 - 05:11:32 Item [245261] [VOD] Adding to internal M3UContent List fileName: EN - Last Week Tonight with John Oliver (2014) (US) S09 E03.strm
05.Mar.2025 - 05:11:34 Item [245707] [VOD] Adding to internal M3UContent List tvgType: TV Shows
.
.
.
05.Mar.2025 - 05:13:56 Item [287339] Detected Season: 4, Episode: 1, Setting to Type: TV Shows
05.Mar.2025 - 05:13:56 Item [287339] [VOD] Adding to internal M3UContent List folderName: TV Shows\EN - Breeders (US)
05.Mar.2025 - 05:13:56 Item [287340] [VOD] Adding to internal M3UContent List folderName: TV Shows\EN - Breeders (US)
05.Mar.2025 - 05:13:56 Checking: #EXTINF:-1,EN - Breeders (US) S04 E04
05.Mar.2025 - 05:13:55 Processing [287060]..Item Name: EN - Supergirl (US) S06 E06
.
.
.
05.Mar.2025 - 05:22:09 Processing [749999]..Item Name: R - Hesse Sevom حس سوم S01 E02

Just by looking at this I can see there are speed improvements to be made.  At the moment I don't know by how much , I will let this run and see how long it takes.

I am not trying to convince you, FYI there is a python script that does similar in the windows tools forum, so there are other options.  

I am going to finish this regardless of who uses it and who doesn't.  The very fact that it can be triggered automatically at a set timeframe and left after setup is what I need for myself.
If it was triggered once a week and took a few hours then so be it, but I am pretty sure I can speed it up some by dropping things it doesn't need like trying to read the tvg-id.
There are still some improvements that can be made.  It is at the moment a prototype if you like...

I am going to bed now, just got in from a night shift.

That's why I want to use this. I can just set everything up in Emby and away it goes, rather than having to config a python script

[mickle026 573]

bare with me, still testing

Changing from synchronous to asynchronous has thrown up some unexpected IO issues that don't occur in synchronous mode,  I am now running emby and the plugin as an attached process in the VS IDE, to try and trace what the hell is going on,  this might be related to @fbrassins issue, currently I am not sure.

Before the code alteration, this one writes D:\VODS\TV Shows\A+ - Gutsy\A+ - Gutsy S01 E08.strm but this one throws the exception D:\VODS\TV Shows\A+ - Dear...\A+ - Dear... S01 E01.strm The difference seems to but the full stops.  There should be no issue at all with the full stops.
However instead of the error being handled by the error handler, in the async method the plugin just exists with no warnings...  That's no good, that needs sorting out.

So where I am is I am trying to trap the exact reason for this, then I can act accordingly such as maybe replacing the full stops with something else or removing them if required.

I will keep you updated.

 

[mickle026 573]

Fixed the IO errors

Running this test m3u with just logging on (not verbose) - Not downloading images either

116 MB (122,407,234 bytes) on my pc.

Started at   12:27:16
Finished at 12:50:47
Took 23.31 minutes in total, including outputting the filtered files and directories.

Without the ignores, it would have took longer.

Going to try yours now @gillmacca01 with just logging on (not verbose) - Not downloading images either
Same ignore filter

132 MB (138,657,792 bytes) on my pc

Started at   15:18:47
Finished at 15:48:54
Took 30.07 mins in total, including outputting filtered files and directories.

Output a lot more files than the previous m3u, so that is probably the time difference

Although a lot of the operations are asynchronous - I am now going to attempt to convert even more so they don't run on the main CPU thread.  We will see how it goes!

This on my PC though so its not your laptop & conditions will vary.  Even so though down to 30 mins is a huge leap.

 

[mickle026 573]

With more altered, it took a little longer???  This is not running emby standalone, this is running it through the VS Debugger as An attached Process...

Started 17:27:20
Finished 18:14:54 
47 mins approximately. 

However the log is a lot more in order

Here is the current build

VODtoSTRMscheduledTask.dll

Not completely sure if its removing old items from the folder - I still need to check that.  Off to work again now but 30 - 47 mins is a big improvement on over 3hrs.

I have no idea what this would be on your laptop.

 

Information Update

05.13 am 06/03/2025
This version is not updating the output folder, and removing old items , I will post a fix today
When I removed the m3u filetype/URL checks, I had removed a line of code integral to the output folder caching, I have replaced that, I am just running a full verbose log mode to check it, then later after a few more checks I will post it.

Edited March 6 by mickle026

[mickle026 573]

So I set an option where you can now choose to use the LOG in Multi-Thread or not, basically choosing not to wait for the log to be written before continuing with the next operation.
I am testing what I can , and what I cant multi thread here.  If it is multithreaded, some operations will finish before others so the log will likely go out of order.

I am pretty new to multithreading so I am finding a few caveats, where it doesn't behave as I would expect so bare with me, testing takes time. I am not fully sure I have done everything I can yet.
I have managed to get @gillmacca01 m3u processing down from 7+ hrs down to 4 mins 13 seconds !!!
(Using my Ignore list of comma separated words) and outputting 172,785 Files, 7,743 Folders

Here is the difference:

132 MB (138,657,792 bytes) 
Output directory already containing files and folders so cached list to compare against built as well.

Full logging really slows this down !!

Full logging on ( including verbose ) - not multithreaded logging
Started: 05:30:32
Still Running: 12:21:37
Over 7 hrs and still running - not outputting files/folders yet  Cancelled it because that is a long time to wait to get results.
Single threaded log writing is not good!

Logging but not verbose - multi threaded logging on
Started:12:27:34
Finished: 12:52:44
Time Take: 25.01 minutes

I  have deleted the existing VOD folder and re-run this test

Started: 14:55:50
Finished: 15:07:54
Time Take: 12.04 minutes

No Logging at all, ran from the Manual Trigger "Play" button in scheduled tasks, so that the time taken is logged there.

4 minutes 13 Seconds

 

So you can tell here that its the logging is a massive impact on performance.

Test what you want with whatever logging you need to get the desired results you want, then turn it off is my advice - until you need to debug something.

 

Here it is attached

VODtoSTRMscheduledTask.dll

 

Edited March 6 by mickle026
dyslexic typos being fixed

[mickle026 573]

I noticed a tvg-name parsing error when the name contained a comma, resulting in file folder and filenames being incorrect in those cases

#EXTINF:-1 tvg-id="" tvg-name="EN - Devotion, a Story of Love and Desire (2022) S01 E01" tvg-logo="http://image.tmdb.org/t/p/w185/2RdEzEl8TrLeJpTOsKdfsky64y7.jpg" group-title="|EN| LATEST RELEASES",EN - Devotion, a Story of Love and Desire (2022) S01 E01

a Story of Love and Desire (2022) tvg-logo=httpimage.tmdb.orgtpw1852RdEzEl8TrLeJpTOsKdfsky64y7.jpg group-title=EN LATEST RELEASES,EN - Devotion, a Story of Love and Desire (2022)

I will rebuilt it after testing other edits and post it tomorrow

What I wan to know for now, is - is it quick enough?  The rest will be sorted out at the next post.

Edited March 6 by mickle026

[fbrassin 44]

Woow, i'll try as soon as i can to see if new one is faster also for me.

[gillmacca01 179]

Just started last build on my QNAP. Won't be as fast as yours as I only have 4 threads, but within a minute, it had already created the m3u files, and strm's have already started to appear in the VOD folder, so a massive performance boost - this is with logging off 

[mickle026 573]

I have fixed the directory parsing mentioned  in my last post where the name contained a comma, so you will not get spurious directories now.

I have made even more things async, i.e. spun more operations - especially I/O into other threads, so the more logical processors, cores and threads you have the bigger the performance gains.

I am pretty convinced that its now at the best performance I can get from it, from where it was a few days ago to now is a HUGE difference!

This is yours @gillmacca01with this ignore filter, no logging.  1 minute 22 seconds
135mb m3u input file

07.Mar.2025 - 15:33:28 Task Started at: 07/03/2025 15:33:28
07.Mar.2025 - 15:34:50 Task Ended at: 07/03/2025 15:34:50
07.Mar.2025 - 15:34:50 Task Duration: 00:01:22.34

Current output with filter
166,043 strm Files, 5,841 Folders

 

AF|,AF -,US|,US -,IT|,IT -,FR|,FR - ,PT|,PT -,BE|,BE -,XU|,XU -,HU|,HU -,ES|,ES -,NO|,NO -,GR|,GR -,SC|,SC -,MT|MT -,SE|,SE -,DK|,DK -,DE|,DE -,DE (BLURAY),TR|,TR -,RU|,RU -,JP|,JP -,IL|,IL -,BG|,BG -,KR|,KR -,KU|,KU -,IN|,IN -,AR|,AR -,AR-ANM -,AR-KID -,PPV,RX ,####,4K,AJ -,AL -,AR-,BR -,CN -,EX -,FI -,IN-,IN -,IR -,IR-,LA -,NF -,MT -,NL -,PL -,RO -,TOP -,AJ -,IS -,PK -,VIU -,AR -,QC -,TOD -,SH -,SE-,SC-,(KR),(TR),(GR),(DE),(FR),(IR),(RU),LAT -,LA LATINO,NL NETFLIX,,NL HBO,EXYU ,NL FILMS,NL 24,,NL FILMS,NL MUZIEK,NL REGIONAAL

 

This is with the directory already created, it took approximately 15 mins altogether to create it from scratch on the first run.

 

I think this can now be considered a beta version, now that it is working correctly.

 

VODtoSTRMscheduledTask.dll

 

Here is this dll version.

I just need to try to get the progress bar to work properly in the scheduled task, I have it kind of working, but that's not so important.

Any do you think it needs an GUI improvements ?

 

Let me know what you guys think and what it needs  
Any bugs you find, let me know  

 

 

Thanks

 

 

Edited March 7 by mickle026

[gillmacca01 179]

I am going to restart again with the build, and using your ignore list, but will remove US from it, and using MultiThread logging only.

The 3 override fields are currently empty

[mickle026 573]

1 hour ago, gillmacca01 said:

I am going to restart again with the build, and using your ignore list, but will remove US from it, and using MultiThread logging only.

The 3 override fields are currently empty

Let ne know how it goes!

I am not convinced its deleting remove items properly, so next I will investigate that.

[gillmacca01 179]

Just now, mickle026 said:

Let ne know how it goes!

I am not convinced its deleting remove items properly, so next I will investigate that.

When I said I was restarting, I meant from scratch. I deleted the m3u and VOD files to try to get a more accurate time period

[mickle026 573]

if you are on windows 11, delete the VOD folder from the "CMD" window, but be careful, rmdir cannot restore if you do it wrong.

Example:

rmdir /q /s "D:\VODS" 
[Enter]

This with take a couple of minutes, compared to windows taking forever....

I have found the error in the directory deletion code and corrected it.
I have fixed logging verbose mode so that it writes everything, even if Log Operation is unticked. (they were 2 separate Log writing commands, verbose should be both)
A few GUI description tweaks.

Removed these "--------------------------------------------------------------------------------" lines from the logging, they were originally there to make it easier for me to read and debug.
They are not needed anymore and taking up CPU cycles and making the log bigger than it needs to be.

You might have already noticed I have also put started and completed timestamps in the log.  If logging is off they should still create a log, but just with the timestamps and nothing else.

 

I am currently testing in in full verbose mode, as we know this is much slower so it'll take a bit longer.

 

I think this plugin is getting to be a decent addition.  It still works with my own VOD m3u, and now seems like it will work with almost any.

[gillmacca01 179]

It hasn't finished for me yet, but so far it has been running for 2hrs 20 mins

[mickle026 573]

Yes,

Logging is a good bit slower than without it,  verbose logging is a lot slower still.  Without multithreading its horrendously slow.

Even if the process is finished, its all wrapped in a task that emby can monitor (I have no control over the scheduled task only triggering and progress bar), so it will wait until everything is done before reporting back to emby that it is complete.
So that means even if the file is fully processed, all the OS commands are issued, any outstanding IO writes or deletes (queue) will make the task wait until they are done including log writes.
I can skip this and force it, but what happens then is the task runs for 0 seconds, and all processes are running in the background, so the reporting is wrong.

[gillmacca01 179]

It has now been running for over 14 hrs and still hasn't finished 

The tv shows folder in VOD only has 18 folders, so I believe there must be an awful lot more to appear. If I'm right, then this is going to take well over 24 hrs

Even taking into account i have slower hardware, less threads, it still shouldn't take this long, when i can add the whole m3u into live tv and have it finish in less than 2 hrs

Edited March 8 by gillmacca01

[mickle026 573]

4 hours ago, gillmacca01 said:

It has now been running for over 14 hrs and still hasn't finished 

The tv shows folder in VOD only has 18 folders, so I believe there must be an awful lot more to appear. If I'm right, then this is going to take well over 24 hrs

Even taking into account i have slower hardware, less threads, it still shouldn't take this long, when i can add the whole m3u into live tv and have it finish in less than 2 hrs

It is definitely the logging,

The next dll has a very different method of logging,
instead of writing at every action as it happens which it seems is causing thread locking while the CPU waits for a thread allocation permit (called a semaphore).  (This was beyond me, until I spent more than a few hours reading about it, I have been up half the night!)
So now I have spent a few hours more figuring out a way to get around this issue.
This new version has a rewritten logging system, that stacks the log requests, then writes in batches.  so instead of writing in 1kb chunks, it writes in 300kb+ chunks.  Much faster logging.  It reduces the IO operations for logging 300% so reduces the threads requested massively and reduces disk access, and therefore bottle neck created by the log requests.

However logging might still be slower than not logging, and then again maybe not by much. (testing that)
I am hoping (fingers crossed) that the new system will have a minimal impact, but running some tests.

With the new logging system:

standard logging: Task Duration: 00:30:54 for me when the VOD folder didn't exist using your m3u file and my ignore list. and new to this version I stacked the output into YEAR ordered folders like this: D:\VODS\TV Shows\2011\NL HBO MAX\Game of Thrones (2011)\Season 08\Game of Thrones (2011) S08 E06.strm 

I am just checking with no logging & then I will check with verbose.

I have fixed the check and update (I believe) & I am just testing that now. (I was checking folders but not folder content in previous versions)
I don't think it will be as fast as just a few minutes, because of IO operations here, but if I can get it to average 30 mins then I am happy.
I am hopeful that the new logging method will reduce everything's timings to be in that range.

The new version also has a number of additions.

Be aware that I am still testing this version
 

 

VODtoSTRMscheduledTask.dll

[gillmacca01 179]

Testing now. I have unticked all logging options, so nothing should be logged (apart from start time and maybe end time)

[mickle026 573]

No logging on at all - 5 minutes 49 seconds.

 

I will start the verbose test now, post results later as I am off to work....

[fbrassin 44]

I've just tried with all disabled.

I mean no log, no poster, no api.

 

08.mar.2025 - 13:43:02 Task Started at: 08/03/2025 13:43:02
08.mar.2025 - 13:56:15 Task Ended at: 08/03/2025 13:56:15
08.mar.2025 - 13:56:15 Task Duration: 00:13:13.47

Not too bad

Edited March 8 by fbrassin

[gillmacca01 179]

I'm still wondering why it's working for other people, but when I try, it takes hours - I've still never managed to complete one full run 

Restarting again. only things that have been entered in the plugin is, the url link, and ignore words (used mickle026 list, plus added another 15 ignore words). Override fields are all empty, no API keys entered, no grouping, no fetching images and no logging

Started at 14:01, 14:06 m3u files are created, so that's 5 minutes gone, before it evens starts creating strm files

 

Edited March 8 by gillmacca01

[gillmacca01 179]

Found a problem which might explain some of the issues (either a bug or I have configured the plug-in wrong.

I have added EN BIBLICAL in the ignore field, but the folder and the strm files are still being created

I have attached my configuration file (removed m3u link). Can someone confirm I have set the ignore up correctly

Here is a screenshot of the folder for EN BIBLICAL that has been created in VODS

I'm going to stop it running for now, until i get feedback

VODtoSTRMscheduledTask.xml

Edited March 8 by gillmacca01