Jump to content

Kodi Startup Full Sync


jurmb84

Recommended Posts

Angelblue05

We would like to make a setup wizard of some sort, but the options are very limited in terms of dialogs. It is something we want to do, I will add it to our trello board.

 

I still feel it would just be simpler to give the option to disable the start up sync. We could go as far as also adding the option to disable websocket events. 

Benefits would be: Could technically be viable for a MySQL setup where you have one main device updating the database with Emby for Kodi and others just being synced via MySQL while being logged into Emby for report progress, etc.

If you don't want to have a start up sync and want to control when content is being sync, then one can just keybind the manual sync to a button and gain full control over the syncing process.

 

I just don't really like complicating things. People are already terrified to use Kodi because they think is too complex. It doesn't help the cause if we then, in turn add complexity to Emby for Kodi setup. We made this add-on with the idea that everything would work with Kodi in a native way while implementing as many Emby features as possible. We try very hard to keep maximum compatibility with Kodi features. The main goal should be to make the experience as easy and seamless as possible, be it if you are an advanced Kodi user or not...

 

Anyway, this is just my opinion as a user. :) Maybe with a wizard we can find a middle-ground for multi profile setup, but I don't know how other devs feel about this. :)

  • Like 1
Link to comment
Share on other sites

denethor

I still feel it would just be simpler to give the option to disable the start up sync...

 

I just don't really like complicating things. People are already terrified to use Kodi because they think is too complex. It doesn't help the cause if we then, in turn add complexity to Emby for Kodi setup.

Totally agree on this. Thats why I said that this is only for advanced users in the first place. It' s really hard to implement for general public. Maybe with the emby 4.0 add-on 2.0 or even further. And besides nobody know what will happen to Kodi MySQL support in 1 yrs. Maybe they will drop it for some reason. This may result waste of your development efforts.

For starters manually disable addon sync look looks like the best way to go. Now I'm importing my myvideos.db to mySQL. It will take 1hrs since I'm importing the data table by table with CSV files and checking the results by my eyes :) After that I will report the funcionality results of Kodi. If it works than we can disccuss how to sync against mysql instead of a sqlite.

Edited by denethor
Link to comment
Share on other sites

xnappo

@@denethor - if you are interested in trying this I don't really think it would be all that hard.  I don't think any of the devs have the energy to embark on it ourselves, but we would help.  Of course we would want you to be in for the long haul for user support.

 

@ - if we just used a web socket, then we get the list of all additions, plays, deletes, modifies for free.  We just log the web socket then we have a list of objects modified since time X.

Otherwise I don't understand how you get a list of items deleted since time X (without doing a server query/compare, which is a slow way to do it in comparison).

 

xnappo

Edited by xnappo
  • Like 1
Link to comment
Share on other sites

denethor

@@denethor - if you are interested in trying this I don't really think it would be all that hard.

I thought the same way but not really, I am trying to deal with UTF-8 mass in myvideos93.movie table for almost 3hrs :)
Link to comment
Share on other sites

xnappo

Ahhh - welcome to the club!  The vast majority of our bugs are due to encoding!

  • Like 1
Link to comment
Share on other sites

denethor

Why does Kodi add-on care if the Kodi runs on a local or remote database?.

I think answer is in utils.py :(

def KodiSQL(type="video"):
        if type == "music":
        dbPath = getKodiMusicDBPath()
    elif type == "texture":
        dbPath = xbmc.translatePath("special://database/Textures13.db")
    else:
        dbPath = getKodiVideoDBPath()
        connection = sqlite3.connect(dbPath)
        return connection
def getKodiVideoDBPath():
    kodibuild = xbmc.getInfoLabel("System.BuildVersion")
    if kodibuild.startswith("13"):
        # Gotham
        dbVersion = "78"
    elif kodibuild.startswith("14"):
        # Helix
        dbVersion = "90"
    elif kodibuild.startswith("15"):
        # Isengard
        dbVersion = "93"
    else:
        # Not a compatible build
        xbmc.log("This Kodi version is incompatible. Current version: %s" % kodibuild)
    dbPath = xbmc.translatePath("special://profile/Database/MyVideos" + dbVersion + ".db")
   
    return dbPath  
Edited by denethor
Link to comment
Share on other sites

xnappo

I think it is actually fairly minor.  @@marcelveldt - do you remember the exact problem?  We were looking into supporting it at one point but then Marcel found an issue and at the time we had bigger fish to fry.

Link to comment
Share on other sites

xnappo

Agree - only if someone wants a little C# project to play with. I have no idea if the plugin would have access to the server messages of course.

Link to comment
Share on other sites

xnappo

How do you propose he handle deleted items?  I don't think the server currently keeps track of episodes you 'used to have'.

 

This is why I like the idea of using the web socket MESSAGES (regardless of whether it is actually the web socket) to do this.  You have a baseline, then can see all the changes that occurred since the baseline.  So a query to the server for messages since time X is what I would like.  I think we could implement this without Luke using another piece of code, but obviously having that in the server would make things easier.

Link to comment
Share on other sites

marcelveldt

I think it is actually fairly minor.  @@marcelveldt - do you remember the exact problem?  We were looking into supporting it at one point but then Marcel found an issue and at the time we had bigger fish to fry.

I tried to add the MySQL support once but it wasn't really that straight forward. The only way to get that one going was to create some database logic layer between which sorts the command for SQLite or MySQL.

 

That said... It really doesn't have much benefit of having MySQL with this addon, the local SQLite database is by far fast enough to handle the job. Also you'd loose remote playback options. The current approach is really flexible because you can pick up your Kodi client, drop it at the other end of the world and it would still play (and sync) your content from the Emby server.

 

The ONLY benefit of having it all in a centralized MySQL database would be that the initial startup sync isn't needed on all clients but that can also be solved as soon as we can get a list of all changed items since last update. Now we query all the items with the eTag property (which is already pretty fast) but if the startup sync could only retrieve a list of changed items that would be almost instant. On the other end, the startup sync sits silently in the background, you don't even know it's there.

 

If someone wants to put work on this I think it's best that the server has extended support for this, maybe an additional sync api endpoint for client apps that want to sync the media library.

Or a "man in the middle"  concept with a small serverside tool or Emby plugin to which the Emby addon will connect.

  • Like 1
Link to comment
Share on other sites

marcelveldt

Totally agree on this. Thats why I said that this is only for advanced users in the first place. It' s really hard to implement for general public. Maybe with the emby 4.0 add-on 2.0 or even further. And besides nobody know what will happen to Kodi MySQL support in 1 yrs. Maybe they will drop it for some reason. This may result waste of your development efforts.

For starters manually disable addon sync look looks like the best way to go. Now I'm importing my myvideos.db to mySQL. It will take 1hrs since I'm importing the data table by table with CSV files and checking the results by my eyes :) After that I will report the funcionality results of Kodi. If it works than we can disccuss how to sync against mysql instead of a sqlite.

sync to MySQL is generally a bad idea. First of all because it's an unsupported and undocumented feature from the Kodi devs. I already saw comments that it would disappear when they introduce their new database system (far in the future) but most important because it doesn't fit the purpose of this addon which is basically top use Kodi as a client, not only local but also remote. The benefit of having MySQL in a regular Kodi setup is that you can sync your watchstates, only have to import library on one machine etc. But... in our case it's the Emby server all doing that.

There is no point at all of having MySQL in combination with this addon, it will overcomplicate stuff.

 

But.. I get your point.. on a large database it seems a but overdone to do a full sync at every start/resume.

I think we need to focus on that (see my previous post) by getting a serverside plugin to catch up with the events on the server. Some may call it a queue system.

Link to comment
Share on other sites

marcelveldt

Haha, I've read this thread bottom down... Now I see the responses that we should build a serverside plugin or small C# util to build the queue :-)

 

Well, building s small C# util to cache/queue the websocket messages won't be that hard to build. Long live .NET. Guess it will take 1 day or so.

It's just that it would be a separate program that overcomplicates stuff. Would be way better if it can be a plugin into Emby itself or offcourse...

 

My ideas (in short) about this:

- queue-program has its own little database

- queue-program has some rest endpoints to connect to from the client

- every connected kodi client will have it's own queue

- set expiration to the queue, for example 1 week no connection with the client drops the queue (resulting in full sync on the client)

- at startup the client connects to the queue api and either gets the queue results or message that a full sync is needed

- The queue-program only stores the messages from the websocket* that are needed by the clients in its database, rest is ignored.

 

* I agree with xnappo that the websocket is the easiest way to get going with this. OR if it can be made as an plugin for Emby I'm sure there will be some internal methods to use to be aware of database changes.

Link to comment
Share on other sites

xnappo

The nice thing about just requesting the web socket messages since time x is that we already have the code on our side to handle that.  Not that it would be a huge deal to change to something else.  One more thing to add to the list (obvious I guess) is that the queue needs to clear on server restarts. 

 

I don't know much about the web socket - does the server know if the message was successfully received? 

Edited by xnappo
Link to comment
Share on other sites

denethor

...One more thing to add to the list (obvious I guess) is that the queue needs to clear on server restarts...

 

Just curious, why? Because people who use dev server are getting daily or more updates/restarts. if the servers clears the queue every time, queue will get useless if I understood clearly
Link to comment
Share on other sites

xnappo

Sure - we could still provide the data in the same format but not actually use the websocket to hack it.

 

Or grep the server log :D

Link to comment
Share on other sites

I think what you'd probably want to do is continue using web sockets when the app is open because it's very effective.

 

Then you could have an Emby server plugin with it's own api endpoint. The plugin would just monitor the server changes that it cares about, and then the kodi add-on startup, it can query your custom api to get the list of changes. So you can use the two in conjunction with each other.

 

If you choose to go this route, then once the plugin is complete we could even embed it into the server so that users wouldn't have to install anything. Unfortunately I cannot take on this dev work myself though because I am overloaded as it is, but I can help answer questions for anyone that would like to do this.

  • Like 2
Link to comment
Share on other sites

Angelblue05

Cool that it can be done technically, however I do not have the time nor the energy for this. If it is something people really want, someone will need to step up for this. :)

Link to comment
Share on other sites

xnappo

Yeah, there is no hurry.  I may play with it a little - but very very slowly :D

Link to comment
Share on other sites

jurmb84

I don't code in either C nor Python... but I am a coder... I wouldn't mind at least getting a look under the hood and see what I can come up with... I very well may ask alot of questions at first but don't mind getting my hands dirty... Could you post a few links for where I can get information on atleast where to get started... I appreciate it...

  • Like 1
Link to comment
Share on other sites

xnappo

Sure.  Here is the main Emby Git:

https://github.com/MediaBrowser/Emby

 

Here is some info about plugin development:

https://github.com/MediaBrowser/Emby/wiki/How-to-build-a-Server-Plugin

 

Here is the Emby API:

https://github.com/MediaBrowser/Emby/wiki

 

Here is the current Web Socket code (we would want an API call to this new plugin that gave us the same JSON data for added/deleted/modified based on a timestamp):

https://github.com/MediaBrowser/Emby.Kodi/blob/master/resources/lib/WebSocketClient.py

(see UserDataChanged and LibraryChanged)

 

Existing plugins (I think some of the notification plugins might be a good place to start)

https://github.com/MediaBrowser/Emby.Plugins

Edited by xnappo
  • Like 1
Link to comment
Share on other sites

jurmb84

@@xnappo Thank you for the links... I will start my homework this evening... Please excuse some of the noob questions as I move along...

 

Noob question #1:  What IDE do you use for coding... Would prefer to use what you all are using to be sure that when I have questions they can be more easily answered if they may pertain to the IDE used for coding...

 

More noob questions to folllow... :D

 

Thanx

Link to comment
Share on other sites

xnappo

Well, we are writing the addon in Python, so we are just using text editors.  @@Angelblue05 - you wrote a plugin once right?  Did you use Visual Studio?

 

Do not worry about asking questions.

Edited by xnappo
Link to comment
Share on other sites

xnappo

Yep - that is the way to go.  This is what I meant all along BTW - sorry I was being confusing by saying 'web socket' when I really just meant the same data/format as the web socket available by timestamp.

Edited by xnappo
Link to comment
Share on other sites

Guest
This topic is now closed to further replies.
×
×
  • Create New...