Plugin: Export VODS from .m3u - split m3u livetv, tvshows, movies (Going BETA!!)

[gillmacca01 211]

ignore that post for now,. just checked the m3u file created by the plugin, and for biblical it looks like this:

#EXTINF:-1 tvg-id="" tvg-name="EN - War Room (2015)" tvg-logo="https://www.themoviedb.org/t/p/original/vMe2TzwlcohcjlFV9GRKjoipOl7.jpg" group-title="|EN| BIBLICAL",EN - War Room (2015)

When the folder is created the folder is called EN BIBLICAL (missing | each end of EN), which is what i entered into ignore list. Going to change it to |EN| BIBLICAL and see what happens

[mickle026 650]

43 minutes ago, gillmacca01 said:

ignore that post for now,. just checked the m3u file created by the plugin, and for biblical it looks like this:

#EXTINF:-1 tvg-id="" tvg-name="EN - War Room (2015)" tvg-logo="https://www.themoviedb.org/t/p/original/vMe2TzwlcohcjlFV9GRKjoipOl7.jpg" group-title="|EN| BIBLICAL",EN - War Room (2015)

When the folder is created the folder is called EN BIBLICAL (missing | each end of EN), which is what i entered into ignore list. Going to change it to |EN| BIBLICAL and see what happens

To be fair I have also nioticed a couple of items not being ignored, and I have that to investigate.

My main efforts have been to try and speed this process up and have put off fine tuning to do that.

I still have a couple more ideas that might work for speeding it up that I want to try.

The pipe used extensively n your m3u is an illegal character when creating folders and file names, that's why it is removed at output.

I'm at work at there moment so this is not 100%, but I think it's ignored on input with the pipe in ignore.

 

This is something to investigate, but what I want to know right now is:

Is it working, and how long is it taking?

 

[fbrassin 47]

After my previous try that lasted very little i put some new words to ignore and some new movie to force to movie and run it again.

It started 2 hours ago.

I don't know what it is doing as i truned off log.

I see it has deleted all movies and all series in the dir and for now only rebuilt 7 movie folders with .strm files into those dir.

Now it seems not doing anything.

How can i know if plugin is running?

[gillmacca01 211]

so what I am doing now is, I have downloaded the m3u to my laptop, and any folder that is being created that I do not want, I am finding it in the M3U and adding the correct group--title to the ignore list - hopefully this should reduce it a lot

Edited March 8, 2025 by gillmacca01

[fbrassin 47]

I tried to restart emby and run plug again it as been super fast.

All seems to be ok.

Now i'll do some more try

 

08.mar.2025 - 16:37:42 Finished this import.
08.mar.2025 - 16:37:42 Task Ended at: 08/03/2025 16:37:42
08.mar.2025 - 16:37:42 Task Duration: 00:03:18.46

 

[mickle026 650]

42 minutes ago, fbrassin said:

After my previous try that lasted very little i put some new words to ignore and some new movie to force to movie and run it again.

It started 2 hours ago.

I don't know what it is doing as i truned off log.

I see it has deleted all movies and all series in the dir and for now only rebuilt 7 movie folders with .strm files into those dir.

Now it seems not doing anything.

How can i know if plugin is running?

The best way is either to run it from the manual trigger > button in scheduled tasks as there will be a progress bar when it is running rather than the manual trigger button in the plugin gui.

They both trigger it the same way, but the scheduled task wraps it in a task that monitors it.

The progress bar isn't incrementing properly yet but it does move and the task completes it tells you how long it took.

[mickle026 650]

32 minutes ago, gillmacca01 said:

so what I am doing now is, I have downloaded the m3u to my laptop, and any folder that is being created that I do not want, I am finding it in the M3U and adding the correct group--title to the ignore list - hopefully this should reduce it a lot

So how long is it taking to do what it's doing on average?

I know it's not accurate yet, I will get to that, but how long?

[sftech13 25]

4 hours ago, mickle026 said:

It is definitely the logging,

The next dll has a very different method of logging,
instead of writing at every action as it happens which it seems is causing thread locking while the CPU waits for a thread allocation permit (called a semaphore).  (This was beyond me, until I spent more than a few hours reading about it, I have been up half the night!)
So now I have spent a few hours more figuring out a way to get around this issue.
This new version has a rewritten logging system, that stacks the log requests, then writes in batches.  so instead of writing in 1kb chunks, it writes in 300kb+ chunks.  Much faster logging.  It reduces the IO operations for logging 300% so reduces the threads requested massively and reduces disk access, and therefore bottle neck created by the log requests.

However logging might still be slower than not logging, and then again maybe not by much. (testing that)
I am hoping (fingers crossed) that the new system will have a minimal impact, but running some tests.

With the new logging system:

standard logging: Task Duration: 00:30:54 for me when the VOD folder didn't exist using your m3u file and my ignore list. and new to this version I stacked the output into YEAR ordered folders like this: D:\VODS\TV Shows\2011\NL HBO MAX\Game of Thrones (2011)\Season 08\Game of Thrones (2011) S08 E06.strm 

I am just checking with no logging & then I will check with verbose.

I have fixed the check and update (I believe) & I am just testing that now. (I was checking folders but not folder content in previous versions)
I don't think it will be as fast as just a few minutes, because of IO operations here, but if I can get it to average 30 mins then I am happy.
I am hopeful that the new logging method will reduce everything's timings to be in that range.

The new version also has a number of additions.

Be aware that I am still testing this version
 

 

VODtoSTRMscheduledTask.dll 588 kB · 2 downloads

I haven't tried the plugin but found good speed results in python when parsing the the m3u in batches using aiofiles, it splits the m3u into blocks for processing and I have seen dramatic speed increase with large m3us. 
 

[gillmacca01 211]

it would take 5 minutes before creating m3u and VOD. files, but 10 hrs later it was still creating VODs, but i think it was due to the ignore words. I ran it about 20 mins ago, and it said it was completed in about 8 mins (which is more inline with what you have been getting). Just need to run a couple more tests, and ignore a few more things to confirm

[mickle026 650]

I need to write the process down as there is obvious confusion here.

I cannot sneak more than a couple of mins from work at the moment to do it, but:

1st thing is cache is built of files and folders that exist in memory.  This is for later clean and update operations.

When the m3u is read, it is completely read into memory before any output file operations happen. The ignore list is processed at the import and the m3u lines are not added to the list in memory.

So in memory is a modified m3u file, that is filtered by the ignore list.

Then this is sorted into categories,live TV, TV show and movies.

I don't remember off my head which is next, but

The vod strm files are created and the m3u split files are created.

Then the directory update and clean up happens, including checking if links in the strm files themselves have changed.  If they are they are updated.

That's a rough guide of how the plugin runs, and that's why the vods have a delay before being written.

[gillmacca01 211]

 I've have just deleted the m3u files and vods created by the plugin. Ran the plugin from the scheduled task, still took 5 minutes to start creating files, but completed in 8 mins 22 secs - I am more than happy with that duration

I do have another issue though (i'm sure your not suprised). not all the vod's are titled correctly

[sftech13 25]

44 minutes ago, gillmacca01 said:

 I've have just deleted the m3u files and vods created by the plugin. Ran the plugin from the scheduled task, still took 5 minutes to start creating files, but completed in 8 mins 22 secs - I am more than happy with that duration

I do have another issue though (i'm sure your not suprised). not all the vod's are titled correctly

@mickle026have you considered writing just the folder of the show/movie instead of the group then show? Or is it as designed? 

I was on the fence of this also and decided to go without writing the group to the folder structure. Curious if you thought about it and why you decided this route. I decided to go the show/movie route with a parent directory to act like a library folder.

 

Just curious 

I'm going to load this up this week and try it out. 

Edited March 8, 2025 by sftech13

[mickle026 650]

1 hour ago, gillmacca01 said:

 I've have just deleted the m3u files and vods created by the plugin. Ran the plugin from the scheduled task, still took 5 minutes to start creating files, but completed in 8 mins 22 secs - I am more than happy with that duration

I do have another issue though (i'm sure your not suprised). not all the vod's are titled correctly

Is this on the qnap?

This looks like a long filename substitution, like win95 used to do.

Another hurdle LOL  

As @sftech13mentioned I may have to include an option to leave out the group name and thus make the directory name shorter.  The limit on the qnap must be shorter than my Netgear nas.

Adding this as an option is not a problem.

 

 

Found this

https://forum.qnap.com/viewtopic.php?t=141919

Which sheds a bit of light on windows > qnap network behaviour,  can you enter your qnap webui and check if the filenames are correct there?  See if it's windows or the qnap that's doing this.

 

Edited March 8, 2025 by mickle026

[gillmacca01 211]

5 minutes ago, mickle026 said:

Is this on the qnap?

This looks like a long filename substitution, like win95 used to do.

Another hurdle LOL  

As @sftech13mentioned I may have to include an option to leave out the group name and thus make the directory name shorter.  The limit on the qnap must be shorter than my Netgear nas.

Adding this as an option is not a problem.

Yes, it's on my QNAP

[mickle026 650]

Edited the post above, read the qnap forum page in the link and check yours, see if it's windows networking or the qnap.

[gillmacca01 211]

Using qfile (QNAP File Explorer app) and in the QNAP's filestation, the files are titled correctly 

[mickle026 650]

1 hour ago, gillmacca01 said:

Using qfile (QNAP File Explorer app) and in the QNAP's filestation, the files are titled correctly 

So this is neither the plugin or the qnap, it's how windows views long filenames on network drives.  That's a registry setting needs toggling, I can't remember offhand which but you can Google it.

I can add an option though as I said before to make the directory path shorter.

[gillmacca01 211]

No point adding an option just for me, unless you think it could happen to others.

The main thing is that it displays correctly in Emby

Edited March 8, 2025 by gillmacca01

[mickle026 650]

I have updated my output file writing code to try and use the same method as I did with the logging.
This has been a royal pain in the *ss !! but i still managed to do it within this evening/night.

I now have a new class and reusable function that can be re-used in all of my future projects, so that's a bonus.
I am now detecting the system architecture, cpu and memory to calculate the max parallel writes I can do without causing thread locking or cpu maxed out conditions.

In my system that is 16 writes per single request. I can now write 16 files at a time, that's a 1600% increase in efficiency.  As my writes were not taking too long that's not a huge increase here, but some will see it as a big difference on their systems.

Added a few more entries to the minimal log (no log options), so that you can see where yours is slow.

09.Mar.2025 - 02:44:00 Create Movie strm files From M3U
09.Mar.2025 - 02:44:00 Max Degree of File Write Parallelism (Parallel Writes): 16
09.Mar.2025 - 02:44:00 Task Started at: 09/03/2025 02:44:00
09.Mar.2025 - 02:44:00 Scanning Output Directoy for content: D:\VODS.
09.Mar.2025 - 02:44:01 Decoding input m3u file.
09.Mar.2025 - 02:44:20 Updating OutputFolder (Creating/Deleting strms & folders)
09.Mar.2025 - 02:48:17 Task Ended at: 09/03/2025 02:48:17
09.Mar.2025 - 02:48:17 Task Duration: 00:04:16.14

 

Added the option for you @gillmacca01 to not use group-title folders

 

This will shorten directory lengths and hence the long path names, everything will be dropped directly in the TV Show or Movie folders.

another test run, almost exactly the same.  4 mins 16 seconds , just 0.06 milliseconds in it! LOL

I think this is probably the best I can get, unless I think of something else.

09.Mar.2025 - 03:24:21 Create Movie strm files From M3U
09.Mar.2025 - 03:24:21 Max Degree of File Write Parallelism (Parallel Writes): 16
09.Mar.2025 - 03:24:21 Task Started at: 09/03/2025 03:24:21
09.Mar.2025 - 03:24:21 Scanning Output Directoy for content: D:\VODS.
09.Mar.2025 - 03:24:22 Decoding input m3u file.
09.Mar.2025 - 03:24:41 Updating OutputFolder (Creating/Deleting strms & folders)
09.Mar.2025 - 03:28:37 Task Ended at: 09/03/2025 03:28:37
09.Mar.2025 - 03:28:37 Task Duration: 00:04:16.20

 

I also fixed empty () parathesis on show output where shows had two years in the name (2000-2003) etc.
 

VODtoSTRMscheduledTask.dll

Edited March 9, 2025 by mickle026

[fbrassin 47]

I run the plugin yesterday night with api enable:

TMDB one.

 Finished this import.
08.mar.2025 - 22:53:13 Task Ended at: 08/03/2025 22:53:13
08.mar.2025 - 22:53:13 Task Duration: 06:00:27.33

But i don't see any refereces [imdb-tt0120855] at the end of the file name.

It is enough to put api key or something else has to be set? I don't see any check box.

[mickle026 650]

Difference now with full verbose logging

1 hr 31 minutes 26 seconds

This is where I am seeing a massive gain in performance!

 

33 minutes ago, fbrassin said:

I run the plugin yesterday night with api enable:

TMDB one.

 Finished this import.
08.mar.2025 - 22:53:13 Task Ended at: 08/03/2025 22:53:13
08.mar.2025 - 22:53:13 Task Duration: 06:00:27.33

But i don't see any refereces [imdb-tt0120855] at the end of the file name.

It is enough to put api key or something else has to be set? I don't see any check box.

I haven't tested this yet , maybe there is something wrong with my logic and its not fetching them properly.
Yes I should add a checkbox to enable and disable, rather than using if the key exists to enable it

I have done the programming logic, but not checked it yet.
I will try to check it today, update it if necessary, check it again...

 

[gillmacca01 211]

The option to not create a group-title works, I just haven't decided if I'm going to use it yet. Depends on how I'm going to add the VODs to Emby

[mickle026 650]

I am currently at work, work , work..... No access to my computer here..

However, I have discovered during all this effort how to do other things in a multi thread way.  I have learned a huge amount about multi threading that I didn't know you could do.

I have already modified my existing directory cache builder to do this using all cpu cores.  I have 16 cores so this is now 16 times faster here.  This read what is already there in the output folder.

This will be in the next build.

I will first be looking how to do this for everything else that I can do it with and make this as efficient as possible.

There is already a check box for update and delete, so there is already a kind of don't remove , just add feature.  This plugins might be different from @sftech13's but I am not sure.  So if you want to add to the directory, and not remove you can uncheck that but currently if you uncheck that it might not update links in the strm files , I will need to check that.

I open the existing strm files and check the links to the media file haven't changed on the next import.  If they have changed I update them.

I will have to check if I overwrite anyway or not, not sure now, can't check at the moment.  I might just be writing what's not there or needs updating.

I have a day off tomorrow, hopefully I will get time to do some more on this.

I actually think I can ."maybe" get this down to under a minute with @gillmacca01's huge m3u file.

This is so far advanced to where I was a week ago already!

Mine is only a few hundred kb, its finished as you click start, lol  

[mickle026 650]

5 hours ago, gillmacca01 said:

The option to not create a group-title works, I just haven't decided if I'm going to use it yet. Depends on how I'm going to add the VODs to Emby

This is why I like options, it's your choice  

[BigIrn 10]

Hey guys I can’t figure out how to determine what is VOD. I’m using Strong8k. Any suggestions to help identify or some sort of epg?