Kodi MediaImport Project

[LongMan 89]

Quote

Would it support hardware acceleration as the official builds do?

It should be the same as the official builds but with MediaImport as an added feature.

Edited March 21, 2021 by LongMan

[TeamB 2352]

7 hours ago, quickmic said:

Just curious, cause I did some research in the past about that matter but not spent too much time...

Do you know is it possible to combine all those API call into one single command?

setProperties, setInfo, setCast, setArt

I mean, filling a dict with ALL info and perform a single API setProperties call?

 

No it is not possible, that is what I was asking about above, you have all the info, I wonder if it would help if you could just set it all in one call or when constructing the ListItem. I suggested json as a way of passing structured data as I don't know how much overhead the ipc bridge adds for complex objects.

 

Edited March 21, 2021 by TeamB

[Montellese 11]

56 minutes ago, TeamB said:

No it is not possible, that is what I was asking about above, you have all the info, I wonder if it would help if you could just set it all in one call or when constructing the ListItem. I suggested json as a way of passing structured data as I don't know how much overhead the ipc bridge adds for complex objects.

 

I'll have to look into that. With the current set of available data types which can be transfered between Python and Kodi core this is not possible. So there are two options:

1. Extend the data types to support arbitrary dictionaries. Right now it's only possible to use dictionaries with key strings and either a string or a vector of string value.
2. Pass the JSON data as a string and parse it again in Kodi core.

Both have the disadvantage that the API is not very well defined because the API definition cannot enforce a specific layout of the dictionary or the JSON string.

I have one question concerning your performance measurements: What properties are you setting using the ListItem.setProperties() method? That call is taking extremely long...

[TeamB 2352]

First of all just let me say I stacked that test a little, I x8 the number of properties I added in that test just to pump up the data and calls, I did this for all tests so the results are comparable, I just wanted biger numbers to compare, in reality the setProperties is probably lower than setCast in total time.

I am adding all sorts of stuff as properties, some of it may be legacy items but some was required for skins.

I create the properties map here:
https://github.com/faush01/plugin.video.embycon/blob/f8e01ae92394fea5a201f9c606ec8e1ddb61cca7/resources/lib/item_functions.py#L455

Add a bunch of stuff to the map and then call setProperties with it on line 622.

I agree, would be best to pass in a fully populated python object to the ListItem constructor.
Or even have a system where you set all the data on a ListItem and then call COMMIT to turn the ListItem into a Kodi visible object.

In fact the python native -> Kodi COMMIT approach would be awesome as you could build a bunch of these ListItem native python objects and then pickle them for later use if you wanted and then when you needed them later you could unpickel them and COMMIT to turn into a Kodi object and display them. At the moment I have to pickle and store a bunch of custom store python objects to build my local cache, would be easier if I could just create the local ListItem python objects and serialize/save them without all the extra overhead of re populating the ListItem.

[Montellese 11]

15 minutes ago, TeamB said:

I agree, would be best to pass in a fully populated python object to the ListItem constructor.
Or even have a system where you set all the data on a ListItem and then call COMMIT to turn the ListItem into a Kodi visible object.

In fact the python native -> Kodi COMMIT approach would be awesome as you could build a bunch of these ListItem native python objects and then pickle them for later use if you wanted and then when you needed them later you could unpickel them and COMMIT to turn into a Kodi object and display them. At the moment I have to pickle and store a bunch of custom store python objects to build my local cache, would be easier if I could just create the local ListItem python objects and serialize/save them without all the extra overhead of re populating the ListItem.

That sounds like it would require a custom pure python class which would then create the xbmcgui.ListItem object when calling the COMMIT method. I'm not sure if Kodi already provides custom pure python code / scripts / libraries. If not it could also be added as a script.module.xyz add-on. But it would still require an effective way to turn the pure python object into a xbmcgui.ListItem object which is currently missing.

I actually ran into the same problem concerning using pickle when I wanted to add some caching for faster debugging.

[TeamB 2352]

54 minutes ago, Montellese said:

But it would still require an effective way to turn the pure python object into a xbmcgui.ListItem object which is currently missing.

yes, and I am not sure what the most efficient way would be, JSON as a single text blob or a populated structured object passed python->kodi.

At least this way the pure Python object would be responsible for the JSON structure so that would help with as it would know what format the JSON needed to be. But then you have the overhead of __dict__ of the object to the JSON blob vs passing something like the __dict__ of the object directly through to Kodi.

 

[LongMan 89]

@alscaldasTurns out building Rockchip RK3399 is a bit more involved than Generic and Ras Pi 4. It actually builds for 10 different boards. Anyway here you go. Have fun.

 

LibreELEC-RK3399.arm-10.0-devel-20210321180917-c5d7822-rockpro64.img.gz

Edited March 22, 2021 by LongMan

[alscaldas 0]

10 minutes ago, LongMan said:

@alscaldasTurns out building Rockchip RK3399 is a bit more involved than Generic and Ras Pi 4. It actually builds for 10 different boards. Anyway here you go. Have fun.

 

LibreELEC-RK3399.arm-10.0-devel-20210321180917-c5d7822-rockpro64.img.gz 126.98 MB · 0 downloads

@LongMan I really appreciate you taking the time to help me. I'll download and try it right now. Thank you very much!

[LongMan 89]

No problem. Let us know how it goes.

[Montellese 11]

8 hours ago, TeamB said:

yes, and I am not sure what the most efficient way would be, JSON as a single text blob or a populated structured object passed python->kodi.

At least this way the pure Python object would be responsible for the JSON structure so that would help with as it would know what format the JSON needed to be. But then you have the overhead of __dict__ of the object to the JSON blob vs passing something like the __dict__ of the object directly through to Kodi.

 

I ran some quick tests with a newly introduced ListItem.setInfoFromJson() against the existing ListItem.setInfo() and ListItem.setCast(). The former (new) method is 20 - 30% faster. More than half the time spent in the new ListItem.setInfoFromJson() is spent deserializing the JSON data so it might actually be worth investigating the custom dict --> C++ type map.

[Montellese 11]

4 hours ago, Montellese said:

I ran some quick tests with a newly introduced ListItem.setInfoFromJson() against the existing ListItem.setInfo() and ListItem.setCast(). The former (new) method is 20 - 30% faster. More than half the time spent in the new ListItem.setInfoFromJson() is spent deserializing the JSON data so it might actually be worth investigating the custom dict --> C++ type map.

I hacked together a quick and dirty version of a Python --> C++ typemap which takes a Python dict and turns it into a generic C++ data type (which we also use to store parsed JSON data). There are a few restrictions on the Python dict, i.e. it must contain string keys and the values cannot be tuples or sets since those aren't supported by JSON either. Based on that I wrote a new ListItem.setInfoFromDict() which is approximately 50 - 60% faster than the original ListItem.setInfo() + ListItem.setCast(). I'll have to discuss this approach in the team.

[TeamB 2352]

6 hours ago, Montellese said:

I hacked together a quick and dirty version of a Python --> C++ typemap which takes a Python dict and turns it into a generic C++ data type (which we also use to store parsed JSON data). There are a few restrictions on the Python dict, i.e. it must contain string keys and the values cannot be tuples or sets since those aren't supported by JSON either. Based on that I wrote a new ListItem.setInfoFromDict() which is approximately 50 - 60% faster than the original ListItem.setInfo() + ListItem.setCast(). I'll have to discuss this approach in the team.

Wow cool, 50-60% is a great improvement, this would be awesome to see in a release build, let me know what the team thinks.

[Montellese 11]

2 hours ago, TeamB said:

Wow cool, 50-60% is a great improvement, this would be awesome to see in a release build, let me know what the team thinks.

I created an RFC / WIP pull request at https://github.com/xbmc/xbmc/pull/19459 to discuss the topic.

[TeamB 2352]

1 hour ago, Montellese said:

I created an RFC / WIP pull request at https://github.com/xbmc/xbmc/pull/19459 to discuss the topic.

thanks for the heads up, I will keep an eye on it

Edited March 23, 2021 by TeamB

[TeamB 2352]

@Montellese I wonder if you could take this a step further and pass in all the data for multiple ListItems and return a list of ListItems.

[neik 837]

Does/Will LiveTV also work with the Media importer?

[LongMan 89]

MediaImport builds are just vanilla Kodi + the MediaImport feature. If Live TV works on vanilla Kodi, it will work in MediaImport. Just be clear, what is the source of the Live TV?

[neik 837]

51 minutes ago, LongMan said:

If Live TV works on vanilla Kodi, it will work in MediaImport.

It does directly in Kodi and using E4K also through Emby Server.

52 minutes ago, LongMan said:

Just be clear, what is the source of the Live TV?

A m3u file with public OTA channels.

[LongMan 89]

Just use IP TV Simple PVR Addon in Kodi and it should work. E4K Live TV channels are just dynamic links, it is not really PVR, there is no Kodi PVR client for Emby server. With the new PVR API in Kodi V19, maybe somebody will write one.

Just to be clear, MediaImport does not prevent you from using any Kodi video addon. You could still install E4K and use the Live TV in it

[quickmic 1342]

1 hour ago, LongMan said:

Just use IP TV Simple PVR Addon in Kodi and it should work. E4K Live TV channels are just dynamic links, it is not really PVR, there is no Kodi PVR client for Emby server. With the new PVR API in Kodi V19, maybe somebody will write one.

Just to be clear, MediaImport does not prevent you from using any Kodi video addon. You could still install E4K and use the Live TV in it

Injection of Live TV m3u and EPG to IP TV Simple from Emby server is on my todo list, but no high priority. Not sure if this is on your agenda. If yes we can collaborate on that.

I'm already using it on my Kodi Boxes as a manual approach.

 

[LongMan 89]

@quickmic,

When I realized that Emby didn't have a Kodi PVR Client, I made a TV Headend server and have been using it for a few years now. Even so It would be a good thing to have m3u and EPG data for PVR Simple client.

Edited March 24, 2021 by LongMan

[TeamB 2352]

On 3/23/2021 at 9:51 AM, Montellese said:

I created an RFC / WIP pull request at https://github.com/xbmc/xbmc/pull/19459 to discuss the topic.

I am following the thread on the PR (my user faush01), I was under the impression you would be able to sel ALL info for a ListItem in one call, from what I can see in your examples and tests in that thread you are specifically talking about media info. When I mentioned Art and Properties the response was that that info is ListItem specific and not media related.

I though the idea was to have the ability to pass ALL content in a dict to set up a LIstItem in one call or in the constructor. Is that not the case?

[Montellese 11]

7 hours ago, TeamB said:

I am following the thread on the PR (my user faush01), I was under the impression you would be able to sel ALL info for a ListItem in one call, from what I can see in your examples and tests in that thread you are specifically talking about media info. When I mentioned Art and Properties the response was that that info is ListItem specific and not media related.

I though the idea was to have the ability to pass ALL content in a dict to set up a LIstItem in one call or in the constructor. Is that not the case?

TBH it's too big to do in one go. I'm currently working on doing it properly for video data so I'm extending InfoTagVideo with all the custom setters and a setInfo() method. Furthermore I'm adding an optional videoInfo parameter to the ListItem constructor. Once that's working I'll look into a combined info dictionary in ListItem. And once that's working I'll have to repeat the first step for music, picture and game data. If I do everything at once it will be a huge mess.

 

Concerning properties it might be difficult to integrate those into the generic info dict because it would result in any "wrong" key to be added as a property without any warning in the logs. So if you would use "directors" instead of "director" you would end up with a ListItem property named "directors" and the "director" data of InfoTagVideo would be empty. A possibility might be to allow a "properties" key in the info dict to avoid this.

[TeamB 2352]

ok cool thanks for the work you are doing, any improvements to performance for addons are welcome.

 

[Montellese 11]

Yeah I can now retrieve and create ~1k movies as ListItems from my Emby server within ~5 seconds. It then takes another ~5 seconds to add them to the Kodi library.