Jump to content

Kodi Startup Full Sync


jurmb84

Recommended Posts

jurmb84

I am currently using XBian with Kodi 14.2 Final and the latest Emby for Kodi on a Raspberry Pi 2... After doing much tweaking to the Remote files and using @@tired dad setup for http://emby.media/community/index.php?/topic/23065-how-i-solved-the-emby-and-kodi-master-lock-issue/ it is currently working flawlessly...  So first off thank you all so much for your hard work... It is definitely keeping the wife and kid aggro low... 

 

My current library has 4302 movies, 260 Boxsets, 247 Series with 14987 Episodes... So as you can imagine, the initial sync took about 6 hrs (on the RPi2)... I use parental features on Emby with 3 accts... 1 for me, 1 for my wife and 1 for the kids... Everytime I switch from one profile to another it has to do a "Full Emby Sync"... Although it flies through that on an x86 processor, it can take up to 5 or so minutes on the Raspberry Pi... I am not bitching just wanted to ask a question... Based on the fact that your plugin has a manual sync all function, after the main "Full Sync" has completed, does Emby have an incremental option to query only the changes that have occurred from a snapshot in time?  If it does, could you not collect the date/time of function start (STARTTIME) and use the date time of Last Sync (LASTSYNCTIME) to query Emby for all changes since LASTSYNCTIME... process those changes and then save STARTTIME to settings??  Just a thought so that it doesn't have to go through the entire process of syncing everytime kodi launches...

 

Again, Thanx for all you guys do...

  • Like 2
Link to comment
Share on other sites

xnappo

I am going to add an option to make the startup sync optional.  I started in on it last weekend but the logic really isn't set up for it very well to do this (as I want to have a good way to handle music and video separately) and I am having a think on how to handle it.

 

I don't think a timer is a good idea, because it will be really annoying if it just starts syncing while browsing - kinda thinking of just having an optional 'Ask before syncing' dialog come up.

 

xnappo

  • Like 2
Link to comment
Share on other sites

jurmb84

That sound excellent... Let me add onto that if I may... I would say (and correct me if i am wrong) there are 3 sync's that occur... Movies, Movie Sets, and TV... Would it be possible, while setting up video vs music, to add in options for the first 3 as well... I add movies about 1 to 2 times per month, but tv shows are added daily... My need to sync movies is next to nothing by comparison...

 

Just to be clear, cause I thought this was the way it worked, if Kodi is up and Emby adds a new TV Show, the currently logged in profile gets that episode/movie/music updated automatically correct?

 

Thank you for the clearifications!!!  Apprecaited as always!!!

Link to comment
Share on other sites

jurmb84

@@xnappo

 

 

I don't think a timer is a good idea, because it will be really annoying if it just starts syncing while browsing - kinda thinking of just having an optional 'Ask before syncing' dialog come up.

 

I wasn't talking about a timer based situation for pulling data in at a timed interval... I was talking about it for the following situation:

 

Full Sync done...

Kodi shutoff...

Kodi turned on (instead of full sync do incremental sync from last sync time and log new last sync time)

run as normal until next exit...

rinse and repeat...

 

Sometimes my brain thinks better than my fingers can type it... :D

Edited by jurmb84
Link to comment
Share on other sites

xnappo

That sound excellent... Let me add onto that if I may... I would say (and correct me if i am wrong) there are 3 sync's that occur... Movies, Movie Sets, and TV... Would it be possible, while setting up video vs music, to add in options for the first 3 as well... I add movies about 1 to 2 times per month, but tv shows are added daily... My need to sync movies is next to nothing by comparison...

 

Just to be clear, cause I thought this was the way it worked, if Kodi is up and Emby adds a new TV Show, the currently logged in profile gets that episode/movie/music updated automatically correct?

 

Thank you for the clearifications!!!  Apprecaited as always!!!

 

I think that is a good idea.  I will look into doing it that way.

 

And yes, if the sync is only needed for things that happen while Kodi is closed.  When it is open the new items are added via a message from the server.

Link to comment
Share on other sites

xnappo

@@xnappo

 

 

I wasn't talking about a timer based situation for pulling data in at a timed interval... I was talking about it for the following situation:

 

Full Sync done...

Kodi shutoff...

Kodi turned on (instead of full sync do incremental sync from last sync time and log new last sync time)

run as normal until next exit...

rinse and repeat...

 

Sometimes my brain thinks better than my fingers can type it... :D

Yeah, there is no query to the server for 'get me all the items added/removed/changed since time X' unfortunately.  We started discussing adding this feature, but the removed items become nasty pretty quickly.

 

xnappo

Edited by xnappo
  • Like 1
Link to comment
Share on other sites

jurmb84

oh snap... completely forgot about removed items... I agree... that gets ugly quick...

 

Thank you for looking into this... It is appreciated!!!

Link to comment
Share on other sites

Angelblue05

I still think it would cause more problem in the long run - people would probably end up needing to reset resync their library way more, due discrepancies slipping in. It's not just item count, but also any type of playstate too right? As long as it's optional..... >.>

Link to comment
Share on other sites

xnappo

We could, but episodes are a far larger percentage of the time, and quite likely to get out of sync just going by total count.  And yes - playstates :D

 

Don't get me wrong - I am still not very satisfied with the system as is.

 

Would be nice if we could queue up 'missed' changes on the server...  Some mechanism to register a websocket receiver and have the server hold on to messages?

 

xnappo

Edited by xnappo
Link to comment
Share on other sites

jurmb84

@@xnappo

 

Would be nice if we could queue up 'missed' changes on the server...  Some mechanism to register a websocket receiver and have the server hold on to messages?

 

WOW... that would be bad to da bone... I work in the medical industry and do the same thing with HL7 messages that are needed to keep systems in sync...

 

I would say that the queued items couldn't be cleared out however due to the possibility of having more than one Kodi instance... i.e. a setup containing 2 tv's and 2 kodi's...

Link to comment
Share on other sites

xnappo

The only way I see something like this working is by making a server plugin.

Yes, it would.

Link to comment
Share on other sites

Angelblue05

so who wants to write a server plugin :D

 

I'm pretty happy working on the add-on... and I don't know C#....yet...

  • Like 1
Link to comment
Share on other sites

xnappo

Items that have changed since X don't do it - you have the delete problem again.

 

The plugin could RECEIVE all web socket events itself though, and then it would have deletes as well as everything else that has changed since time X.  Maybe that is what you meant though.

Edited by xnappo
Link to comment
Share on other sites

Angelblue05

We would need to record websocket events. But only in the event it's requested by the Kodi session and save the list under tied to the userId. I don't think it would be a good idea to have it running constantly. :) Basically while Kodi is offline and it would sort through the events, get the id for UserId items - [add, update], [delete], until Kodi is back online. This way when Kodi compares there is no guessing. If there's no items changed/delete, then no initial scan happens. But how to maintain across multiple instances of Kodi running?

 

Anyway, to me, it seems like a lot of complicated factors, for very little reward. I would rather look into a simple disabling startup sync + manual sync combo....

Link to comment
Share on other sites

jurmb84

Not to mention it would have to be linked not only to the user but also to the device name in the kodi addon... So then you could have multiple devices that could have items in the store with the same user name...

Link to comment
Share on other sites

Angelblue05

How would you gather this information? Would it be dynamic or pre built? What about multiple kodi instances. Let's say you have one Kodi that's been offline for a bit and missed the last sync list? :)

 

 

Sent from my iPhone using Tapatalk

Link to comment
Share on other sites

Angelblue05

I think the better solution is if it's possible to get items per view - sort by last modified / descending order. Then, no plugin would be required. I don't think the underlying issue is having the start up sync run. It's have it run when nothing has been modified since, correct? However - deleted items are still a problem, because it deletes any reference to the item in the server - I believe the server should keep reference for a bit, for the id and report status as deleted... but I don't think it's available. So I don't know after all :(

 

Current possible sort by:

Options: Album, AlbumArtist, Artist, Budget, CommunityRating, CriticRating, DateCreated, DatePlayed, PlayCount, PremiereDate, ProductionYear, SortName, Random, Revenue, Runtime
Edited by Angelblue05
Link to comment
Share on other sites

denethor

I know you guys positions the EMBY as a Kodi backend but I have a proposal:

 

I switched to MYSQL for my Kodi clients.(only for Music and Pictures) I had 2 main motives. First; syncing 20.000 songs Music Library was taking ages. Second; I'm using a plugin called MyPics DB that supports MySQL backend for Kodi (Amazing Picture add-on by the way) for thousands of my Photos. Adding Music and Pictures Libraries to my Kodi clients was a real pain in a** but now I have a perfect solution.

 

Of course I tried to use my SQL backend for Videos Libraries. I thought that Kodi will import all my Movies/TVShows back to MySQL backend with plugin paths (plugin://plugin.video.emby/movies/*) It turns out Kodi does not aware that it has a Video Library because EMBY add-on does not define any Video Library "source" to Kodi.

 

So my recommendation is; use Kodi MySQL backend. I know that using mySQL is for advanced users. But looks like mostly the advanced users are complaining about Add-on sync algorithms anyway :) Benefits:

 

- Add-on is already relies on Kodi SQLite database. It shouldn't be too hard to use MySQL instead :wub:  AFAIK Kodi using the same database structure for both SQLite and MySQL. Looks like you only need a find a way to Kodi think it has a Video Library sources

- Sync only from one Kodi instance and add option to disable sync on others in the add-on configuration. (master sync client configuration can be automated later on by checking some parameters in backends)

- All Kodi instances will instantly get the new items

- 1st time installations won't need to wait for sync to finish

 

Later on;

 

- You don't need Kodi is running for sync.Emby Server can DB Link to MySQL instance for constant always sync.

- You can manage Kodi SQL setup parameters via Emby WEB interface for novice users. Server generates downloadable advancedsettigs.xml files to put it into Kodi userdata folder for starters and later on even the add-on manipulates the file for user by using the data stored in the Emby server.

- You can even share the same MySQL database for Emby itself in version 4.0 :)

 

 

I know that I' m offering something huge that requires fundamental infrastructural changes and lots of hardwork. But this feels like the most future proof solution for me. I believe that current problems are not showstoppers and users can live with them for couple of quarters. So you have time to solve it with the best possible ways (don't get me wrong I am not saying that my idea is best way to go!) Implementing fast and dirty workarounds for complex integration problems is always a problem in the long run. I'm saying this with my 20 yrs of IT experience.

Edited by denethor
Link to comment
Share on other sites

Angelblue05

It would work where your setup is the same user accross many Kodi instances. The moment you have more than one profile, this will be very limiting. With Emby you can personalize profiles/userview, use parental restrictions, have favorites. This would be lost and have to be recreated manually/independently for each profile that are not the same user as your main account. 

 

That is, if I'm understanding correctly what you are saying. If not, disregard :)

Edited by Angelblue05
Link to comment
Share on other sites

denethor

Honestly I am not sure how add-on handles the multiple Emby users. I assumed that it always syncs all server data to Kodi SQLite and displayed them according to user you set in Add-on settings. If this is the case it should be the same with MySQL.

 

If you talking about linking Emby users to Kodi profiles it' same. I believe Kodi generates userdate/profile/<user> folders for each user. Multiple profiles is also supported in Kodi mySQL according to Wiki.

 

Logic is simple; if you have more than one SQLite database for each user you could have the same with MySQL. If you handle it in same SQLite database You could also handle it in MySQL.

 

I'm not saying that if it is easy or not I'm just saying that it should be possible in theory :) Handling multiple users is not easy task to handle It took 2 years with Plex :)

Edited by denethor
Link to comment
Share on other sites

Angelblue05

So let's say your secondary profile is user X where you only allow to view TV Shows, not movies. The server only returns the content accessible by the user X. Only TV Show would be synced for user X.

 

Anyway, if someone wants to step up and give any ideas a try, go for it.... I do not... :)

Edited by Angelblue05
Link to comment
Share on other sites

denethor

Ok I tested it creating new Kodi profile called "Kids" linking to another Emby user. (My "Kids" user with Parental restrictions)

 

Kodi generated new profile folder  in "\userdata\Profiles\Kids" and along with that it generated "...\Kids\database\MyVideos93.db". Emby add-on jumped in, found my server on local network and asked which Emby user I want use. I cancelled it  (by the we still need "manual login" option in user selection for hidden users in setup screen) and put "Kids" manually in add-on config. Emby synced the content into new database.

 

Obviously we should have different MySQL database schemas for each Kodi profiles and it should be managed via individual advancedsettings.xml file. But once the central databases generated for each Emby user all those databases will be available for all your Kodi setups :)

 

Only think we need that is the Kodi think it has a Video library source. I will try to replicate sqllite myvideos93.db to MySQL :)

Link to comment
Share on other sites

Angelblue05

The manual login is by manually entering the user into the add-on settings. Then it should prompt for password. If it does not, let me know.  :) The fix for the source is in the clean up branch.

Edited by Angelblue05
Link to comment
Share on other sites

denethor

The manual login is by manually entering the user into the add-on settings. Then it should prompt for password. If it does not, let me know.  :) The fix for the source is in the clean up branch.

No we don't have a problem with that. I meant first intial add-on setup. It pops-up a user list but you need to cancel that pop-up to set it up manually. You can' t enter your hidden user name in intial setup pop-up window. There might be an option in the pop-up to manually enter username.
  • Like 1
Link to comment
Share on other sites

Guest
This topic is now closed to further replies.
×
×
  • Create New...