Plugin: Export VODS from .m3u - split m3u livetv, tvshows, movies (Going BETA!!)

[fbrassin 47]

Yes, i disabled real time monitor and emby check

[mickle026 650]

1 hour ago, jaspalgill said:

Was that the first initial run, as in the folder was empty so all the items are being created the first time?

1 hour and 26 minutes is a long time, I have 311,000 items (265,086 being extracted into strms) in my M3U and it took my system 00:04:50.27.

My second scan is where I have the problem.

I am running emby inside a VM but it is on a SSD.

I’m sure you may have already discounted any real time monitoring enabled in your libraries? I’ve also disabled the automatic media library scheduled scan as well.

It all seems to be around the check, update and clean operations.

In earlier versions, it worked but wasn't doing it properly and was missing items.

I re wrote the code to work properly.  However with it being multi threading it can get ahead of itself, and try to delete a file that was already deleted on another thread by another core.  This causes an io exception and thus a delay.  An exception causes a stack trace and each one can add a few seconds. On slower systems this seems to be a lot worse, so it was going into a loop of rechecking.  That is what the last dll tries to address.

There is a huge amount to explain here about how this all works and the way the differences between machines , not only speed but how they work can be dramatic.  I am not going to go into that.

However, what we do know is that when it didn't work properly, it was fast.  When I fixed it, so the cleanup was happening it became slow.

So what I need to do is think hard how I can do this without clashes in the io subsystem.

I have a few ideas, so I will think on it a bit more.

[fbrassin 47]

Tnx @mickle026, your explanation it's clear.

Just to let you know i tried two more runs outside emby and all when right

These are the durations

Second run: Task Duration: 04:25:45.02

third run: Task Duration: 04:09:24.69

These are ok for me as i run it at night.

The problems for me begin when i write inside emby.

[Apotropaic 41]

@mickle026- thanks for the explanation as well, I’m far from being a developer but good to get an insight into what’s going on.

I did a run without the delete option and 311,000 items completed in 5 minutes. Only taking one more minute over the initial scan and export.

@fbrassin- when you say inside/outside of emby, are you just meaning you are doing a test outside of your media library locations?

Where are your library file locations?

For example my emby is installed in /var/lib/emby.

Where as my library is pointing to /emby/library/TV etc.

I’m doing my testing outside of my main library location, so it’s not getting imported into emby. When I get home tonight I will run it on my main library without the delete option just to see if I can replicate my 5 minute scan time there.

[mickle026 650]

I have been brainstorming and had an epiphany, i have come up with what I think is a very good idea and solution to this file system problem.  I am hoping it will solve a lot of issues at once.

I have to code it yet, so will let you know how it works when I have done some tests.

 

@jaspalgilli wouldn't really class myself as a "good" developer yet.  I am still learning, albeit I have improved vastly over the last year, and improving still, learning new things and grasping it very quickly now.

I have only been doing multi threading a few weeks now and have quite a good grasp of it now .

[BigIrn 10]

unhandled_a620fb78-8bf8-4ec6-9b8c-fb8211845ffc.txtNot sure what this error means. I won't scan the movies from the Streaming folder to Emby GUI movie folder.

[mickle026 650]

15 minutes ago, BigIrn said:

unhandled_a620fb78-8bf8-4ec6-9b8c-fb8211845ffc.txtNot sure what this error means. I won't scan the movies from the Streaming folder to Emby GUI movie folder.

It means a directory it was attempting to use is no longer there.
This is one of the things I am trying to address, I have had an idea (see post above) - I just need to finish coding it.

[goin3d 4]

thank you so much for this plugin!  this is exactly what i was hoping for and should allow me to get rid of doing this outside of emby through docker.  so far i've tested this on various streams and everything runs very well on my setup performance wise.  I did have a question around the directories that are being created.  I'm seeing instances where some 3 character TV Shows are having words filtered out of their directory names but i can't figure out why.  for example the show 'FBI' is showing up as.  i checked and didn't see any differences in that series in the source data.  The same happens for another 3 character series "DMZ".  Could you point me in the right direction on where i should start looking for the cause?  Thanks! 

[mickle026 650]

3 minutes ago, goin3d said:

thank you so much for this plugin!  this is exactly what i was hoping for and should allow me to get rid of doing this outside of emby through docker.  so far i've tested this on various streams and everything runs very well on my setup performance wise.  I did have a question around the directories that are being created.  I'm seeing instances where some 3 character TV Shows are having words filtered out of their directory names but i can't figure out why.  for example the show 'FBI' is showing up as.  i checked and didn't see any differences in that series in the source data.  The same happens for another 3 character series "DMZ".  Could you point me in the right direction on where i should start looking for the cause?  Thanks! 

Run it with full logging on and it will show you where the string was manipulated like this. Post that part of the log (the lines)

Then I can advise better

[goin3d 4]

11 minutes ago, mickle026 said:

Run it with full logging on and it will show you where the string was manipulated like this. Post that part of the log (the lines)

Then I can advise better

Thanks!  This is what i see:

16.Apr.2025 - 12:41:43 sanitized groupTitle [ Path.Combine() folderName]: 
16.Apr.2025 - 12:41:43 Removed Country code: Input: 'FBI S01 E10', Output: 'FS01 E10'
16.Apr.2025 - 12:41:43 channelName: FS01 E10

 

I do have the "remove preceding country code option selected" but not sure why that would remove the "BI' from the middle

[mickle026 650]

On 15/04/2025 at 14:07, mickle026 said:

Sorry, I was playing with trying not to remove Italian articles in the strings from the beginning of titles (as well as country codes) and i didn't revert the test code before the last build "il, lo, la, i, gli, le"

Ah, yes another victim of my messing with the code - i will revert the changes in the next build, just be patient until then  

for @fbrassinI recommend turn OFF remove country codes, and Just remove "IT - " with custom removal strings, to achieve the same result but keep the articles (for now)

Edited April 16, 2025 by mickle026

[goin3d 4]

3 minutes ago, mickle026 said:

Ah, yes another victim of my messing with the code - i will revert the changes in the next build, just be patient until then  

Thank you so much!!!

[fbrassin 47]

19 minutes ago, mickle026 said:

Ah, yes another victim of my messing with the code - i will revert the changes in the next build, just be patient until then  

for @fbrassinI recommend turn OFF remove country codes, and Just remove "IT - " with custom removal strings, to achieve the same result but keep the articles (for now)

Yes, i turned it off as i don't need it.

Tnx

[fbrassin 47]

3 hours ago, jaspalgill said:

 

@fbrassin- when you say inside/outside of emby, are you just meaning you are doing a test outside of your media library locations?

Where are your library file locations?

For example my emby is installed in /var/lib/emby.

Where as my library is pointing to /emby/library/TV etc.

I’m doing my testing outside of my main library location, so it’s not getting imported into emby. When I get home tonight I will run it on my main library without the delete option just to see if I can replicate my 5 minute scan time there.

I mean outside emby library.

Emby library point to /media/strm and now plugin is working to /media/m3u.

For now 'till it will work after creating /media/m3u with plugin i manually copy it to /media/strm without overwrite.

Edited April 16, 2025 by fbrassin

[goin3d 4]

@mickle026apologies if this was covered elsewhere.  While processing if I would like to skip an entire group and everything under it from processing, for example under Series:  "Belgium Kids (FR)" would i just add the group name here, or am i totally off base?

 

[mickle026 650]

Right, here we go:

I have dumped/removed the whole process I was using to update, remove and clean the files/folders.
I have completely rewritten this, and using a completely different method, so this cannot really compare to the previous method.

Everything is now done in memory and the filesystem isn't even touched until a file needs to be written or deleted, and then only those that need to be changed or deleted are modified.

This is a MASSIVE change to how it was being done.

I had been thinking about this for a long time how i could do this, then yesterday I had a moment of inspiration.

I already read the files when I check the directories, so I know what is there and what link they contain.
I already read the m3u data, so I already know what files I am going to write and what links they have.

I can check them in memory without even touching the drive! - why did this not even occur to me until yesterday???

So this build I am doing this method, then only writing or deleting what is different.

This should massively reduce IO operations and thus increase speed dramatically.

I have also reverted the country code back to its original

@fbrassinI hope this solves all of your issue's! - i await cautiously.......



 

VODtoSTRMscheduledTask.dll

Edited April 16, 2025 by mickle026

[mickle026 650]

4 minutes ago, goin3d said:

@mickle026apologies if this was covered elsewhere.  While processing if I would like to skip an entire group and everything under it from processing, for example under Series:  "Belgium Kids (FR)" would i just add the group name here, or am i totally off base?

 

You can put that in either ignore box.
The first one ignore it only if its at the start of the line, the second one ignores it anywhere in the data

So if you ignore words (contains) "(FR)"  Then anything - including subtitled or dubbed that contained  (FR)anywhere in the string would get ignored, but if you put "Belgium Kids (FR)" Then only this group would get ignored.

Its pretty flexible

I suggest you create a few smaller m3u files and test the output in another directory until you can achieve what you desire

[Apotropaic 41]

5 hours ago, mickle026 said:

@jaspalgilli wouldn't really class myself as a "good" developer yet.  I am still learning, albeit I have improved vastly over the last year, and improving still, learning new things and grasping it very quickly now.

I have only been doing multi threading a few weeks now and have quite a good grasp of it now .

Hi @mickle026don't put yourself down, I can only wish I had both the skills and the time to contribute to the community like you do. It's bad enough being at work and have people critique your work and ask for stupid enhancements!

I'm running a full test now...

I think the CPU usage is going through the roof and not following what I set in the plugin, I had set it to 50%, which the logs show.

16.Apr.2025 - 21:02:02 CPU Cores: 2: Logical Processors (Max Threads): 4, Hyperthreading: True

16.Apr.2025 - 21:02:02 Using max 2 threads

 

However the server ground to a halt and I can see all 4 of my 'cores' being smashed. It killed the stream I was watching and the dashboard was unusable so I killed the process.

 

I'll do another test by turning the delete option off later just to make sure everything else is working as expected.

Edited April 16, 2025 by jaspalgill
Testing update

[mickle026 650]

59 minutes ago, jaspalgill said:

Hi @mickle026don't put yourself down, I can only wish I had both the skills and the time to contribute to the community like you do. It's bad enough being at work and have people critique your work and ask for stupid enhancements!

I'm running a full test now...

I think the CPU usage is going through the roof and not following what I set in the plugin, I had set it to 50%, which the logs show.

16.Apr.2025 - 21:02:02 CPU Cores: 2: Logical Processors (Max Threads): 4, Hyperthreading: True

16.Apr.2025 - 21:02:02 Using max 2 threads

 

However the server ground to a halt and I can see all 4 of my 'cores' being smashed. It killed the stream I was watching and the dashboard was unusable so I killed the process.

 

I'll do another test by turning the delete option off later just to make sure everything else is working as expected.

Ah sorry, there is no cpu limiter in the new code yet - its for testing only.

The new part is the only part that doesn't have the limiter - unless I screwed something up

Edited April 16, 2025 by mickle026

[fbrassin 47]

I've just launch the new version.

I hope i can give a feedback soon, but i'm not sure as tomorrow morning i am leaving to Florence for 3 days.

I will try to tell you if it worked.

I tried to put folder inside emby library this time. Finger crossed.

Anyway tnx for all the job you are doing.

[mickle026 650]

OK, I have added cpu limiter.

50% CPU (8 of 16 cores)
 

16.Apr.2025 - 23:56:44 Saved 'D:\splitm3ufiles\TV Shows\TV Shows.m3u'
16.Apr.2025 - 23:56:44 TV Show M3U Items found: 5000.
16.Apr.2025 - 23:56:44 Unknown M3U Items found: 0.
16.Apr.2025 - 23:56:44 ======================================
16.Apr.2025 - 23:56:44 Debugging: Memory used: 112 MB
16.Apr.2025 - 23:56:44 Debugging: Time taken to process M3U file: 00:01:19
16.Apr.2025 - 23:56:44 Debugging: M3U file processing completed.
16.Apr.2025 - 23:56:44 Debugging: [scanM3UFileAsync] m3uContents Contains: 5000 Items.
16.Apr.2025 - 23:56:44 -----------------------[ Updating Output Folder ]-------------------------------
16.Apr.2025 - 23:56:44 Using; 8 cores/threads.
16.Apr.2025 - 23:56:44 Starting File Writer: 0 Items to write/update.
16.Apr.2025 - 23:56:44 Starting FileRemover: 0 Items to delete.
16.Apr.2025 - 23:56:44 Checking for folder deletion Tasks.
16.Apr.2025 - 23:56:45 Skipped root folder: D:\VODS3 (contains .strm files or subfolders containing them).
16.Apr.2025 - 23:56:45 All delete tasks completed.
16.Apr.2025 - 23:56:45 Finished this import.
16.Apr.2025 - 23:56:45 Task Ended at: 16/04/2025 23:56:45
16.Apr.2025 - 23:56:45 Task Duration: 00:01:23.47
16.Apr.2025 - 23:56:45 Task completed.
16.Apr.2025 - 23:56:45 CreateMovieStrmsFromM3UTaskStarter method completed.
16.Apr.2025 - 23:56:45 Finished.

No changes detected on second scan, so nothing updated and nothing deleted
 

VODtoSTRMscheduledTask.dll

Edited April 16, 2025 by mickle026

[Apotropaic 41]

1 hour ago, fbrassin said:

I've just launch the new version.

I hope i can give a feedback soon, but i'm not sure as tomorrow morning i am leaving to Florence for 3 days.

I will try to tell you if it worked.

I tried to put folder inside emby library this time. Finger crossed.

Anyway tnx for all the job you are doing.

I have this strange Deja vu with your issue of library location but I can’t remember if it’s this forum or another I’m remembering!

Do you have any TV Show or Movie download management software monitoring the emby library folders? File timestamp changes could be triggering something within those tools. Maybe using the Kodi plugin, which may do something similar although I think that runs periodically on a schedule.

[mickle026 650]

11 minutes ago, jaspalgill said:

I have this strange Deja vu with your issue of library location but I can’t remember if it’s this forum or another I’m remembering!

Do you have any TV Show or Movie download management software monitoring the emby library folders? File timestamp changes could be triggering something within those tools. Maybe using the Kodi plugin, which may do something similar although I think that runs periodically on a schedule.

It appears to me that a filesystem change is kicking something CPU intensive off,  Emby can do that with real-time monitor just because of so many concurrent updates, but is it emby?
Remember you have to restart Emby when toggling the real-time monitor as it doesn't switch off until you do.

[goin3d 4]

3 hours ago, mickle026 said:

You can put that in either ignore box.
The first one ignore it only if its at the start of the line, the second one ignores it anywhere in the data

So if you ignore words (contains) "(FR)"  Then anything - including subtitled or dubbed that contained  (FR)anywhere in the string would get ignored, but if you put "Belgium Kids (FR)" Then only this group would get ignored.

Its pretty flexible

I suggest you create a few smaller m3u files and test the output in another directory until you can achieve what you desire

thanks!  conversely if i add something to the "Include Words (start with)" would it ignore everything else?  for example lets say you had a file with 50 VOD categories and you only wanted 2 of them.  would you include the two or have to exclude the other 48? 

[mickle026 650]

13 minutes ago, goin3d said:

thanks!  conversely if i add something to the "Include Words (start with)" would it ignore everything else?  for example lets say you had a file with 50 VOD categories and you only wanted 2 of them.  would you include the two or have to exclude the other 48? 

Depends what you want,
Put the 2 of them in include and it will ignore the rest (it will ignore live tv and movies too though)
Put something like:
"EN - ","Action VOD" in includes
and
"XXX" in excludes
in includes and it will include all "EN - ", so all live tv, movies etc, but then only include "Action VOD" and it will ignore all "XXX"

Have a play on a smaller m3u file, copy some of what you want and some of what you dont want into a new txt file and run it on another output folder

 

further down in the options you can choose only to output movies, only live tv, only tv shows, or a combination, its up to you!

Edited April 16, 2025 by mickle026