python script to convert m3u file to strm files with emby folder structure (free)

[t1m 8]

On 3/16/2022 at 4:49 PM, requa3r0 said:

@t1m

You should also have a look at https://github.com/erdesigns-eu/M3U2STRM

That scprit does a perfect job.

Especially if you teddy up the m3u a bit for IPTVservice specific flukes and errors.

If you are on git-hub you can have a look at my code/ script where I do quite a bit of house cleaning before i feed the m3u to the strm converter.

This process is important to avoid lost titles and messy out-put.

I can also just send you a PM with a download link.

It wound be fun to have an another programmers opinion on the approach ;O)

I have seen it, used it, and loved it...  but it is not cross platform that is why i created the python version  I do like it however, I used it prior to making the python version.  I really didn't expect the python version to get so much interest especially so long after I wrote it.  It was really designed for my one specific use case, but it's cool to see people still using it.  

My GitHub is https://github.com/silence48  

[t1m 8]

10 minutes ago, robrhedrick said:

I just updated the code and added in a provider variable. Right now I only know what the format is for apollo, but others could be added in if needed.

I also created a compare function so existing stuff doesn't get "re-written" with a new timestamp.

https://github.com/hooray4me/m3u2strm

 

just FYI if this is a fork from mine ( https://github.com/silence48/m3u2strm ) I wanted to let you know it worked with apollo and iptorrents but ipt used some other third party for their streams.  Anyways I love all the updates guys keep them coming! my emby users will love you for it too  my VOD libraries are used much more heavily than the 80tb of local media i have LOL

[hooray4me 52]

10 minutes ago, t1m said:

just FYI if this is a fork from mine ( https://github.com/silence48/m3u2strm ) I wanted to let you know it worked with apollo and iptorrents but ipt used some other third party for their streams.  Anyways I love all the updates guys keep them coming! my emby users will love you for it too  my VOD libraries are used much more heavily than the 80tb of local media i have LOL

I didn't know you were the original creator. I got the base script from someone else. Added u for credit in the readme.

And same here... VOD libraries are all my users use!

[afogleson 13]

1 hour ago, cayars said:

Forget to say something?

Sort of I was confused which one (python or java) you were talklng about then figured it out... 

[afogleson 13]

1 hour ago, t1m said:

I have seen it, used it, and loved it...  but it is not cross platform that is why i created the python version  I do like it however, I used it prior to making the python version.  I really didn't expect the python version to get so much interest especially so long after I wrote it.  It was really designed for my one specific use case, but it's cool to see people still using it.  

My GitHub is https://github.com/silence48  

I did very much the same thing as you. I had a java version for my own use primarily and have just added more things that I needed. most recently i added the ability to handle the apollo specific live events though those look a little funky in emby.. they work and parse great its just a matter of how to make them show up in some consumable format for users of emby. 

I just happened to have something very similar to what you wrote and.... yeah i cheated and piggybacked on your thread to put my java version out there

[t1m 8]

1 hour ago, afogleson said:

I did very much the same thing as you. I had a java version for my own use primarily and have just added more things that I needed. most recently i added the ability to handle the apollo specific live events though those look a little funky in emby.. they work and parse great its just a matter of how to make them show up in some consumable format for users of emby. 

I just happened to have something very similar to what you wrote and.... yeah i cheated and piggybacked on your thread to put my java version out there

Haha nice I know i had some regex parsers in the tools.py file to filter out most the live feeds that i didn't care to have, since i probably already had them in the dvr guide anyways.  I was thinking of making js or wasm version of this for fun but for my use the current script works fine lol runs the cron every day and i'm happy

Nice job on the java version! 

strm files is the #1 reason i use emby over plex 

[afogleson 13]

4 hours ago, t1m said:

strm files is the #1 reason i use emby over plex 

Oh without a doubt. i bought the lifetime license for plex a long while back but once i tried to make IPTV work with it nicely i gave up. And i grabbed emby and the lifetime license there and have never looked back. Every once in a while i look at the competition just to make sure there is not something that might be cool to try to create as a plugin or such but really emby does everything i need it to do 

[requa3r0 12]

On 29/03/2022 at 03:56, afogleson said:

Oh without a doubt. i bought the lifetime license for plex a long while back but once i tried to make IPTV work with it nicely i gave up. And i grabbed emby and the lifetime license there and have never looked back. Every once in a while i look at the competition just to make sure there is not something that might be cool to try to create as a plugin or such but really emby does everything i need it to do 

Same here.

Plex is totally dead..Emby rocks.

And...KODI is king...but don't tell any one here ;O)

[Carlo 4552]

Kodi and Emby are actually very different in functionality with some obvious similarities such as watching media.

They can complement each other very well if designed to do so especially if you purchase devices especially well suited for running Kodi on.
Kind of gives you the best of two worlds if done correctly.

[t1m 8]

4 hours ago, cayars said:

Kodi and Emby are actually very different in functionality with some obvious similarities such as watching media.

They can complement each other very well if designed to do so especially if you purchase devices especially well suited for running Kodi on.
Kind of gives you the best of two worlds if done correctly.

indeed this is true.  I use kodi as a front end for emby in my theatre room. it can be made to be so much more immersive and customized. 

[gillmacca01 180]

On 3/17/2022 at 9:35 PM, afogleson said:

I had a slight defect where i always looked for Season and episode information on TV shows. i fixed that  and parsed the current  url and i create the strm files and you can view them in emby if you create the library as "home videos". I just leave the options file empty its possible that would still work  anyway i have attached the version that works with Sports VOD below

CreateStreams-1.0.zip 71.13 kB · 6 downloads

Only just got round to looking at the sports VOD, and unfortunately, my knowledge in java is non existent.

I have managed to set the dir in the bat file, and a CMD window opens and then closes, but that is as far as I can get.

I'm hoping you, or someone else could lend a hand? 

[Carlo 4552]

Just curious, how do you deal with duplicate movies/shows from a provider if you already have it?

Same question if you had 2 or 3 providers as there would be quite a bit of overlap.

[gillmacca01 180]

I keep the VOD stuff in its own library, so the only overlap I should see is if I'm searching.

Would be great if there was a way to stop the strm being created if it already exists in the library 

[afogleson 13]

10 hours ago, gillmacca01 said:

Only just got round to looking at the sports VOD, and unfortunately, my knowledge in java is non existent.

I have managed to set the dir in the bat file, and a CMD window opens and then closes, but that is as far as I can get.

I'm hoping you, or someone else could lend a hand? 

So the zip is an application so you should be able to unzip it and then run CreateStreams.bat in the bin directory (or CreateStreams if on a *Nix/Mac system)

[afogleson 13]

4 minutes ago, gillmacca01 said:

I keep the VOD stuff in its own library, so the only overlap I should see is if I'm searching.

Would be great if there was a way to stop the strm being created if it already exists in the library 

Yes that is largely what the create Streams does. it would probably not be that hard to stop it from creating a strm file if it was already present. in fact im not sure that i dont already do that my own IPTV provider has something like 14K movies i would have to produce a .strm for every night and i cannot imagine i am doing that (although it is one reason i also added the start year and end year in order to not check old movies/streams

[afogleson 13]

3 minutes ago, afogleson said:

So the zip is an application so you should be able to unzip it and then run CreateStreams.bat in the bin directory (or CreateStreams if on a *Nix/Mac system)

ohh and actually if you want a command to run nightly.... here is my basic update call for movies every night

echo Handling Movies from IPTV
/PATHTO/CreateStreams-1.0/bin/CreateStreams -i=false -n=/PATH to Movies M3u/Movies.m3u  -o=/path to outputMovies/HD -s=2022 -e=3000 -u=true

 

-i is whether the -n attribute is a url

-n is either a path to an M3U or a url to an M3U see -i

-s is start year to limit

-e is end year

-u is to use groupname as a subfolder

you could also call CreateStreams.bat -help to see a message of the options

 

[afogleson 13]

9 minutes ago, afogleson said:

Yes that is largely what the create Streams does. it would probably not be that hard to stop it from creating a strm file if it was already present. in fact im not sure that i dont already do that my own IPTV provider has something like 14K movies i would have to produce a .strm for every night and i cannot imagine i am doing that (although it is one reason i also added the start year and end year in order to not check old movies/streams

and i just looked at the source and i do indeed check to see if the stream file is already there. its how i get statistics about how many new movies (or tv shows) we added.  so that is how i am handling duplicates :)

[hooray4me 52]

https://github.com/hooray4me/m3u2strm/blob/e79bdcf77d54c0613f79a46bfc0e44a0a63aa317/m3u2strm.py#L30 is where I call the compare function for existing folders and structures...

and here is the function that DOES the compare, fairly quickly

https://github.com/hooray4me/m3u2strm/blob/e79bdcf77d54c0613f79a46bfc0e44a0a63aa317/tools.py#L237

[hooray4me 52]

40 minutes ago, cayars said:

Just curious, how do you deal with duplicate movies/shows from a provider if you already have it?

Same question if you had 2 or 3 providers as there would be quite a bit of overlap.

I was thinking about the same thing... if were mixing real videos files vs strm files we could probably do a check before writing the strm file... and/or a check if we decide to download a movie to have local that already existing as a strm and remove the strm?

[afogleson 13]

1 hour ago, hooray4me said:

I was thinking about the same thing... if were mixing real videos files vs strm files we could probably do a check before writing the strm file... and/or a check if we decide to download a movie to have local that already existing as a strm and remove the strm?

Its an interesting thought. so in my case you have "options:" in the program that creates stream files. if you chose to use the group name you would definitely have a sub folder for every movies (my preference) and it would be relatively simple to say "does this folder have .*.strm file. in the case of not using that we would still be safe on .strm files since it creates the file with the same name. however if you ripped a BR or such to a single folder with all movies it would be difficult to handle unless the user was careful to use the same format as the stream creation would use. so you would have a video file with name.* where * is limited to known video file types (mkv, avi, mp4...) its much easier if you use a separate folder. MUCH easier. I may add this functionality to the java version with a couple options  so that if you are using (in my prgram) -u true then you could add some more options -d true.false for -delete   -t for type with either o or s (other or strm to indicate which type to delete or NOT add)

so then you dont have to worry about names... lets say i am in the folder The Longest Day ... and i have a stream i could create. now there are 2 options

1. if -d == true then we would look at -t and see if it was o or s

    a. if -t == 0 then we would create the strm and delete any video files that are not .strn and of course we would NOT create a .strm if there already was one but would still do the deletion

    b. if -t == s then we would not create the strm and we would ONLY delete the strm file if there is another video file present.

2. if -d ==false we just follow the same rules as today... dont create it if there is a .strm but do create it if there is not. largely i don't delete and really only have this case where i have a ripped UHD disk so i can have "low res" (1080P)  and UHD so i would leave them both generally. especially since a strm file is bytes and that is nothing compared to the gigabytes of a UHD rip

 

That should be pretty easy to deal with actually. I will probably want to refactor my code. it all works but its sort of hacked together so i probably want to create a more manageable (read maintainable) structure 

[hooray4me 52]

59 minutes ago, afogleson said:

Its an interesting thought. so in my case you have "options:" in the program that creates stream files. if you chose to use the group name you would definitely have a sub folder for every movies (my preference) and it would be relatively simple to say "does this folder have .*.strm file. in the case of not using that we would still be safe on .strm files since it creates the file with the same name. however if you ripped a BR or such to a single folder with all movies it would be difficult to handle unless the user was careful to use the same format as the stream creation would use. so you would have a video file with name.* where * is limited to known video file types (mkv, avi, mp4...) its much easier if you use a separate folder. MUCH easier. I may add this functionality to the java version with a couple options  so that if you are using (in my prgram) -u true then you could add some more options -d true.false for -delete   -t for type with either o or s (other or strm to indicate which type to delete or NOT add)

so then you dont have to worry about names... lets say i am in the folder The Longest Day ... and i have a stream i could create. now there are 2 options

1. if -d == true then we would look at -t and see if it was o or s

    a. if -t == 0 then we would create the strm and delete any video files that are not .strn and of course we would NOT create a .strm if there already was one but would still do the deletion

    b. if -t == s then we would not create the strm and we would ONLY delete the strm file if there is another video file present.

2. if -d ==false we just follow the same rules as today... dont create it if there is a .strm but do create it if there is not. largely i don't delete and really only have this case where i have a ripped UHD disk so i can have "low res" (1080P)  and UHD so i would leave them both generally. especially since a strm file is bytes and that is nothing compared to the gigabytes of a UHD rip

 

That should be pretty easy to deal with actually. I will probably want to refactor my code. it all works but its sort of hacked together so i probably want to create a more manageable (read maintainable) structure 

I'd really like to see an option for 1 icon for the movie, and then have a drop-down (defaulted to the highest quality available)... but say you're watching from spotty hotel internet.... you could drop-down to the strm version and not try to mess with the stream rate to force transcode the 4k version. Does that make sense?

[afogleson 13]

14 minutes ago, hooray4me said:

I'd really like to see an option for 1 icon for the movie, and then have a drop-down (defaulted to the highest quality available)... but say you're watching from spotty hotel internet.... you could drop-down to the strm version and not try to mess with the stream rate to force transcode the 4k version. Does that make sense?

+1... it does as you show work once you have clicked either one but it would be nice to only have 1

[Carlo 4552]

3 hours ago, gillmacca01 said:

I keep the VOD stuff in its own library, so the only overlap I should see is if I'm searching.

Would be great if there was a way to stop the strm being created if it already exists in the library 

 

3 hours ago, afogleson said:

ohh and actually if you want a command to run nightly.... here is my basic update call for movies every night

echo Handling Movies from IPTV
/PATHTO/CreateStreams-1.0/bin/CreateStreams -i=false -n=/PATH to Movies M3u/Movies.m3u  -o=/path to outputMovies/HD -s=2022 -e=3000 -u=true

 

-i is whether the -n attribute is a url

-n is either a path to an M3U or a url to an M3U see -i

-s is start year to limit

-e is end year

-u is to use groupname as a subfolder

you could also call CreateStreams.bat -help to see a message of the options

 

 

2 hours ago, hooray4me said:

https://github.com/hooray4me/m3u2strm/blob/e79bdcf77d54c0613f79a46bfc0e44a0a63aa317/m3u2strm.py#L30 is where I call the compare function for existing folders and structures...

and here is the function that DOES the compare, fairly quickly

https://github.com/hooray4me/m3u2strm/blob/e79bdcf77d54c0613f79a46bfc0e44a0a63aa317/tools.py#L237

I was curious to see if any of you tackled it in an elegant way.  Last year was I was playing with this I had 6 or 7 IPTV providers I could generate streams from. That's on top of 20K+ movies and 100K+ episodes in my libraries.

Ideally this could be found out using the Emby API with a call for each movie/episode but that would be a lot of hits. This is a one-off for my system but what I did was Generate the strm files including normal folder paths naming. 2 of the providers had pretty bad naming of movies so results were always questionable. So I'd generate everything based on the naming in the m3u files. Each provider had it's own local working space. I set this up so it didn't overwrite files that already exist.  That allows me to then pull only files with a newer timestamp for further processing.  Anything new would get ran through a file renamer util so regardless of source would have the same naming. I'd then load them by preference of provider.  First run would load everything.  I'd then scan it into Emby along with my other media.  Once done loading I'd shut down, grab a copy of the library.db and restart Emby.

With some SQL I  could then find dupes IDs and would write these out to a text file file.  The text file was actually a batch file that when ran would delete the strm files getting rid of dupes.  I wrote a little app that would remove folders that were empty to keep things a bit more tidy.  I'd just work down the line until I've loaded everything and deduped it.  A bit of work but it worked quite nicely.

Round 2.  Instead of processing files separately I setup a database and imported each m3u file including tv channels and vod content. With everything in one DB I made a small util that would test each channel and VOD channel and record the resolution, bitrates, audio languages and other similar info I could get.

With all providers and channels in the DB I had to handle renaming and identification of movies and shows in the db vs file system but wasn't hard. I could cleanup TV channel names as well using a standardized naming convention.  I could then filter out channels I have via OTA or cable. So now with 3 10 or 50 providers the job isn't really harder.

Besides the information I 've loaded from the m3u files I've also imported list of movies and episodes from my Emby system which I'd import roughly once a month.

 

This added additional features as well.  I could for example do everything as mentioned above with the export tailored to Emby Server use.  I could also ignore or forget I have any of my own content and just get a combined deduped list of all channels and vod back out as one m3u file which could be used conventionally from a phone or other device. My daughter for example does a lot of traveling and takes a small MiBox (vs Amazon Firestick) with her. Now anywhere with Internet she has a really nice collection of VOD as well as full channel list of TV channels by way of the Live Channels app (not available I don't think on Fire TV) which does it's own guide loading so one less thing to worry about.

Back to Emby...

Few of you would need anything as elaborate and probably have a single provider or two.  I've not tried this yet but imagine it would work well and that is using Chef's version of Auto-Organize to load the VOD content.  This should be able to filter out existing content so it doesn't get loaded. First pass might take a while to load but after the first export/import into Emby you can now process only new content  moving forward.  That assumes you setup your m3u2strm export to not overwrite existing files. You simply grab content with new file timestamps for processing which is easy to do on Windows and Linux.

Which can identify items that exist already.

Then as a backup you could also try the following plugin that combines "duplicate" versions of movies (caution if you want 4K and 1080 separate). Emby should handle merging TV shows on it's own.

So that's the different options I've played with. Having the custom database with the ability to validate and pick the best version (resolution, bitrate) does expand abilities a lot especilally if selection and export is automated like I had it.  Any one provider having issues or dead channels/vod would be found and not used.

The ideal way to do that is using a "redirection" plugin that Emby calls.  So you feed Emby "fake URLs pointing to your own app". The apps get the vod or channel number, dynamically pulls the best real URL to use and returns it to Emby as a redirect which Emby follows.   So the app needs to run but isn't involved in the transport of the thread.  It's just a traffic cop. Since it's testing channels throughout the day it can note when channels are down or find resolution changes, etc and deal with them real-time as Emby is about to use the stream.

With a slight modification the backend program could poll Emby via API to see what channels/streams are in use and act on that info.  If for example you can only get 2 streams per subscription you could have 2 or 3 subs from the same provider with different username/passwords. The backend would know 2 streams are in use from the provider so any new use would use the stream (username,passcode) from the 2 sub...  Poor man's m3u bonding if you want to call it that. But unlike XTeVe it's not sitting in the middle processing the stream just doing some quick traffic direction at the start of stream use which basically uses no additional system resources in a meaningful way.

Carlo

 

[gillmacca01 180]

11 hours ago, afogleson said:

So the zip is an application so you should be able to unzip it and then run CreateStreams.bat in the bin directory (or CreateStreams if on a *Nix/Mac system)

Apollogroup have 3 m3u's for sports VOD, where do I put these files, and what do I change in the bat file to point to these files?

[afogleson 13]

2 hours ago, gillmacca01 said:

Apollogroup have 3 m3u's for sports VOD, where do I put these files, and what do I change in the bat file to point to these files?

With apollo group keep in mind that files in my create streams can actually be one of 2 things

1. and individual M3U file

2 a file with a list of urls (one per line) to and m3U file

so lets say we have https://apollo/file1.m3u,  https://apollo/file2.m3u and https://apollo/file3.m3u
 

I would create a file on my file system that looks like so:

VOD.urls
 

https://apollo/file1.m3u
https://apollo/file2.m3u
https://apollo/file3.m3u

and then I would just point createStreams at that file.

I do this with Apollos TV groups since they are up to something like 20 now