Plugin: Export VODS from .m3u - split m3u livetv, tvshows, movies (Going BETA!!)

[mickle026 650]

2 minutes ago, goin3d said:

Ty!!  i hope the data helps.  When you run a file for the 2nd time what really happens when you don't have remove obsolete files checked?  does the plugin still compare the current run to what's on disk to determine what to write?  trying to figure out why the run times and cpu usage increase 10x from the first run with an empty output folder to the subsequent runs.

I will check and get back to you.  I am at work right now (night shift) so it will likely be later after I had some sleep.

I am actively on another project now, but won't abandon this one, I will keep it up to date.

[sundevil67 28]

53 minutes ago, mickle026 said:

Do you have one URL that downloads a master file with all the other URLs in it, or are they all separate?.  It sounds like a very odd setup....

Agreed... and it took probably 30 minutes altogether to squeeze it out of the support rep, and I think I was providing him with as much guidance as he was me. He ultimately had no idea why it is set up that way, or why the "main" URL they provide in the public Emby setup guide directs you to an M3U file that excludes everything before 2023 & a select group of recent/popular shows.

He finally clued me in to that other URL for which you could just change that number at the end, up to around 30, to find previous years.I tried to use that main URL again as a test, since it'd worked before, but I still can't get the plugin to start/run at all now.

 

[mickle026 650]

44 minutes ago, sundevil67 said:

Agreed... and it took probably 30 minutes altogether to squeeze it out of the support rep, and I think I was providing him with as much guidance as he was me. He ultimately had no idea why it is set up that way, or why the "main" URL they provide in the public Emby setup guide directs you to an M3U file that excludes everything before 2023 & a select group of recent/popular shows.

He finally clued me in to that other URL for which you could just change that number at the end, up to around 30, to find previous years.I tried to use that main URL again as a test, since it'd worked before, but I still can't get the plugin to start/run at all now.

 

Any chance you can send me the files?

You can find & replace all in most text editors, to replace the domain url and the username/password.  I just need the format, i don't need anything else.
use personal message if you are able to send them

Edited April 23, 2025 by mickle026

[mickle026 650]

9 hours ago, goin3d said:

No settings have changed, no strings added, etc.  just running the same file 3 times in a row.  This is a file where the first run takes about a minute.  the 2nd and 3rd runs take around 6 minutes.  that doesn't sound bad, but as i run larger files the two example logs below grow and grow.  It looks like with each consecutive run some of the exact same items show up over and over again to be written to disk.  For these examples i do not have remove obsolete files checked.  If i do check it the run takes longer but i also see removals in the log over and over, very similar to the additions i attached.  I hope that helps!

FileWriteDebugLogging [M3UToStrm]22.Apr.2025 3rd Run.txt 74.02 kB · 0 downloads FileWriteDebugLogging [M3UToStrm]22.Apr.2025 2nd Run.txt 74.88 kB · 0 downloads

There is nothing in these logs that show the reason why.  The reasons are in the extra verbose log, because the delete function just carries out a request list of files to delete, its the m3u processor that decides if they are to be deleted or not and the reason is written as a  debugging log entry under extra verbose settings.

[fbrassin 47]

@mickle026

I've just made a try.

I downloaded m3u file and run plugin twice reading that downloaded file. 

Even if the original file is the same, the plugin recreated .strm files the second time i run plugin.

If you tell me what you need (m3u file, logs, config?) so i can send them to you.

 

Edited April 24, 2025 by fbrassin

[fbrassin 47]

@mickle026

i sent you logs and my config file.

For the third time plugin overwrote strm files even if the original m3u is the same, this causes emby to rescan library.

I hope it can help. Tell me if you need anything else.

Bulid version is 1.0.0.159, i don't know if it is the latest

[goin3d 4]

On 4/24/2025 at 4:31 AM, fbrassin said:

@mickle026

i sent you logs and my config file.

For the third time plugin overwrote strm files even if the original m3u is the same, this causes emby to rescan library.

I hope it can help. Tell me if you need anything else.

Bulid version is 1.0.0.159, i don't know if it is the latest

From your logs could you tell if it was the same shows/movies being overwritten each time or did it change?

[fbrassin 47]

I cannot Say for each movies and series, but from what i checked It seems they are the same ones to be rewritten.

[mickle026 650]

After many, many builds and getting nowhere but slowing down the plugin,

I had found no difference causing the re-writes.

Then I had a thought , can these systems use different ascii/utf8 characters that look the same visually, but are different codes in the system.  I have used a weird conversion that I have never used before.  It took me a while to figure out how to do it, but so far, current tests seem to work - no re-writes yet!

scannedItem.FolderName?.Trim('\r', '\n', ' ').ToLowerInvariant().Normalize(System.Text.NormalizationForm.FormC)

I will test on a larger m3u and post those findings

 but here is the DLL, back up to being fast again!
 

VODtoSTRMscheduledTask.dll

[goin3d 4]

I'm in the process of trying the new build.  with the previous version using my large test file (85mb) a fresh run on an empty directory would take around 6 minutes.  The 2nd or 3rd run would take about an hour but have a lot of re-writes.  So far with this new version with no changes to settings (i didn't turn on any of the new character options you added yet) the first run took 21 minutes.  the 2nd run is still going but the log has been sitting at this spot for about 35 minutes.  During this Updating Output Folder phase is the time consuming activity looking at what's already there and trying to determine what to write and what not to write?  Would the changes you made in this last version impact the time that would take?

26.Apr.2025 - 19:31:45 Saved '/config/STRM/m3u/Movies/Movies.m3u'
26.Apr.2025 - 19:31:45 Saved '/config/STRM/m3u/TV Shows/TV Shows.m3u'
26.Apr.2025 - 19:31:45 Unknown M3U Items found: 0.
26.Apr.2025 - 19:31:45 Debugging: M3U file processing completed.
26.Apr.2025 - 19:31:45 Debugging: [scanM3UFileAsync] m3uContents Contains: 333555 Items.
26.Apr.2025 - 19:31:45 -----------------------[ Updating Output Folder ]-------------------------------

[mickle026 650]

13 minutes ago, goin3d said:

I'm in the process of trying the new build.  with the previous version using my large test file (85mb) a fresh run on an empty directory would take around 6 minutes.  The 2nd or 3rd run would take about an hour but have a lot of re-writes.  So far with this new version with no changes to settings (i didn't turn on any of the new character options you added yet) the first run took 21 minutes.  the 2nd run is still going but the log has been sitting at this spot for about 35 minutes.  During this Updating Output Folder phase is the time consuming activity looking at what's already there and trying to determine what to write and what not to write?  Would the changes you made in this last version impact the time that would take?

26.Apr.2025 - 19:31:45 Saved '/config/STRM/m3u/Movies/Movies.m3u'
26.Apr.2025 - 19:31:45 Saved '/config/STRM/m3u/TV Shows/TV Shows.m3u'
26.Apr.2025 - 19:31:45 Unknown M3U Items found: 0.
26.Apr.2025 - 19:31:45 Debugging: M3U file processing completed.
26.Apr.2025 - 19:31:45 Debugging: [scanM3UFileAsync] m3uContents Contains: 333555 Items.
26.Apr.2025 - 19:31:45 -----------------------[ Updating Output Folder ]-------------------------------

yes, not really impacting the time it takes, (perhaps the extra text manipulation has added some time - I don't know yet) but updating the log is different. instead of writing them as it finds them, it is collecting them all, and writing them all together (there will be a pause here now) so that I can debug it easier and have a full list of what is being written twice, its much easier to then look up and then compare them.  This is only 5% of the progress bar so it might seem like it stopped when it has not.

Edited April 27, 2025 by mickle026

[goin3d 4]

soon as I hit post it finished lol.  I had no rewrites for movies (whoohooooooooo), but hundreds for tv shows still.  the overall time was good at 50 minutes though!  Couple questions:

1.  Would you be willing to run the same data I am? 

2.  Unrelated to the rewrites:   if I'm setup like this screenshot and run two different files and two different times I thought what should happen is nothing gets deleted but the output directory would retain the strm files from both runs?  What i'm seeing is If i run two different M3Us in two different runs, the output of the first is deleted.  Should that occur?     

[fbrassin 47]

With this latest versione the time is good as it lasted only 00:29:28.26, but i still have many rewrites.

Don't you think the cause of rewrite could be that some movies has more then one entry in the m3u file?

Do you need the logs?

 

[mickle026 650]

I am still trying to figure out the reason for the re-writes.

The debugging is telling me that the don't exist on disk - which after all other debugging can only mean they were not read on the directory scan.

I am still investigating

** Edit - everything IS read , not found the culprit yet  

Edited April 27, 2025 by mickle026

[fbrassin 47]

1 hour ago, mickle026 said:

I am still trying to figure out the reason for the re-writes.

The debugging is telling me that the don't exist on disk - which after all other debugging can only mean they were not read on the directory scan.

The question is WHY? - no errors were thrown.

I am still investigating

I'm not a coder and i don't know anything, maybe i say nonsense, but what about writing i file with all the entry written and use it to scan?

[mickle026 650]

1 hour ago, fbrassin said:

I'm not a coder and i don't know anything, maybe i say nonsense, but what about writing i file with all the entry written and use it to scan?

I do that already internally.  The are written to a list in memory and compared in memory.

What is happening is that the computer is saying something is different between them, but when I inspect them , they are the same.
So because the computer is saying they are not the same, its resulting in the plugin re-writing them because it believes they don't exist.

I am still investigating, they are mostly the same files every time so there must be a pattern that I am just not seeing.

Edited April 27, 2025 by mickle026

[goin3d 4]

Not sure if this logic will help you or not but I stumbled on a workaround for my needs that is very fast and produces no rewrites.  I’m using two directories.  The first directory is where the plugin outputs its files the first time it’s run.  Once the run is finished a script copies the files to a second final directory and deletes all the files in the first directory.  The script is set to copy but not touch any file that exists.  (cp -r -n /folder1 /folder2). Every repeat run the same process happens with the result being only net new files are copied to the final folder.  The great part is the plugin is always running on an empty output folder which is very fast.  Running the copy and delete part via script is much faster than the current plugin trying to figure it out.  All in all the process now takes around 20 minutes and has no rewrites vs the 60 minutes I had previously.  Would that workflow help work through the rewrites for you?

[fbrassin 47]

2 minutes ago, goin3d said:

Not sure if this logic will help you or not but I stumbled on a workaround for my needs that is very fast and produces no rewrites.  I’m using two directories.  The first directory is where the plugin outputs its files the first time it’s run.  Once the run is finished a script copies the files to a second final directory and deletes all the files in the first directory.  The script is set to copy but not touch any file that exists.  (cp -r -n /folder1 /folder2). Every repeat run the same process happens with the result being only net new files are copied to the final folder.  The great part is the plugin is always running on an empty output folder which is very fast.  Running the copy and delete part via script is much faster than the current plugin trying to figure it out.  All in all the process now takes around 20 minutes and has no rewrites vs the 60 minutes I had previously.  Would that workflow help work through the rewrites for you?

The only problem with this Is that doesn't delete no more present strm files in m3u. And strm files Will not update. Right?

[mickle026 650]

I have found where it is happening - that's the good news  

At the moment I am testing a few different methods to see which works without causing this and also trying out different logic to see which is fastest.

[goin3d 4]

8 minutes ago, fbrassin said:

The only problem with this Is that doesn't delete no more present strm files in m3u. And strm files Will not update. Right?

This is correct!  For my m3us I never used that functionality as I don’t see items being removed.

side note, a few post up I mentioned that cleanup/delete function is running regardless if it’s turned on or not.  For example if I run two sources in 1 run and then the next run I only run 1 source all of the 2nd source files are deleted even with the cleanup options turned off.  So for me in the short term I really wanted to avoid anything like that happening

[mickle026 650]

5 minutes ago, goin3d said:

I mentioned that cleanup/delete function is running regardless if it’s turned on or not.

You are right, I have just put that option back in so it will only work with the checkbox option. from the next build.
I don't remember disabling this option, maybe when I rewrote the whole file writer code I possibly didn't put it in ?? 

[fbrassin 47]

51 minutes ago, mickle026 said:

I have found where it is happening - that's the good news  

 

What was it?

[mickle026 650]

Current status is it is working except when there are 2 entries  for the same thing. One is causing the other to raise a deletion task.
I'll get there  

#EXTINF:-1 tvg-id="" tvg-name="Topolino - La Casa del divertimento S01 E01" tvg-logo="https://image.tmdb.org/t/p/w600_and_h900_bestv2/jOSPfbjtyZz8Vpt8rF16FUlSUVq.jpg" group-title="Serie Disney+",Topolino - La Casa del divertimento S01 E01
http://m3u_url/series/username/password/401002.mkv

#EXTINF:-1 tvg-id="" tvg-name="Topolino - La casa del divertimento S01 E01" tvg-logo="https://image.tmdb.org/t/p/w600_and_h900_bestv2/7NCVaHkcTLfG6pMoLOa6SKsbKtQ.jpg" group-title="Serie Disney+",Topolino - La casa del divertimento S01 E01
http://m3u_url/series/username/password/339700.mp4

Its the same episode but both the stream url and the logo url are different

 

It was an OR instead of an AND in the code, so it compiled without issue.  logic error - my fault.

 

testing with extra verbose logging on, so its taking a while

Edited April 27, 2025 by mickle026

[Apotropaic 41]

3 hours ago, mickle026 said:

Current status is it is working except when there are 2 entries  for the same thing. One is causing the other to raise a deletion task.
I'll get there  

#EXTINF:-1 tvg-id="" tvg-name="Topolino - La Casa del divertimento S01 E01" tvg-logo="https://image.tmdb.org/t/p/w600_and_h900_bestv2/jOSPfbjtyZz8Vpt8rF16FUlSUVq.jpg" group-title="Serie Disney+",Topolino - La Casa del divertimento S01 E01
http://m3u_url/series/username/password/401002.mkv

#EXTINF:-1 tvg-id="" tvg-name="Topolino - La casa del divertimento S01 E01" tvg-logo="https://image.tmdb.org/t/p/w600_and_h900_bestv2/7NCVaHkcTLfG6pMoLOa6SKsbKtQ.jpg" group-title="Serie Disney+",Topolino - La casa del divertimento S01 E01
http://m3u_url/series/username/password/339700.mp4

Its the same episode but both the stream url and the logo url are different

 

It was an OR instead of an AND in the code, so it compiled without issue.  logic error - my fault.

 

testing with extra verbose logging on, so its taking a while

Is this a true duplicate within the same Group Title? I have many duplicates of the same show but in different Group Titles, mostly it will be for different resolutions (4k, HD, SD etc.).

I save my outputs into Group Title folders so I get to choose which I import into my library, emby is good that it will merge the duplicates together and let me choose which version to play per episode.

[mickle026 650]

5 hours ago, jaspalgill said:

Is this a true duplicate within the same Group Title? I have many duplicates of the same show but in different Group Titles, mostly it will be for different resolutions (4k, HD, SD etc.).

I save my outputs into Group Title folders so I get to choose which I import into my library, emby is good that it will merge the duplicates together and let me choose which version to play per episode.

this is the same episode and same series folder, therefore the same write path, its just the content of the strm file & logo url that is different.  The current logic decides that the second one doesn't match the drive so calls for a delete.  The file to delete is the same path as the first one, so it will delete it.

I have since found another issue
D:\VODS\fbrassin-test2\TV Shows\Serie Anime\Animals.\Season 01\Animals. S01 E01.strm << output from m3u
D:\VODS\fbrassin-test2\TV Shows\Serie Anime\Animals\Season 01\Animals. S01 E01.strm << actually written to the drive

so next scan, they don't match because of a trailing full stop "." on the group title. So now I am going to trim off a full stop if its at the end of the string

Still testing, I hope to have it resolved soon.

Edited April 28, 2025 by mickle026