Jump to content


Photo

Incremental Sync Takes Almost 5 Minutes to Complete

incremental sync slow

  • This topic is locked This topic is locked
18 replies to this topic

#1 imadunatic OFFLINE  

imadunatic

    Advanced Member

  • Members
  • 95 posts
  • Local time: 05:31 PM

Posted 23 October 2015 - 08:20 PM

So, I've played with incremental sync in the past and my experience has always been that it is much slower than a full sync. I finally decided to take pictures of it tonight and post logs here in hopes that I just have something off. 

 

These are the pics of the items syncing: 

 

562acf0d8d665_IMG_20151023_193549475.jpg

 

562acf1b24900_IMG_20151023_193812606.jpg

 

562acf2e2b751_IMG_20151023_193833014.jpg

 

562acf3bb65b3_IMG_20151023_193903667.jpg

 

These are the logs: 

 

  http://pastebin.com/kjme6Pdc

 

 

With all the praise this receives, I have to believe something is wrong with my setup. I'm running server version 3.0.5768.4 and Kodi Plugin Version is 1.1.52, incremental sync queue version is 1.0.5734.31705. I think I have about 450 tv episodes and 289 movies. 

 

What am I doing wrong? If I restart kodi, incremental sync is fairly quick; the next day when I fire up Kodi again, I get this 5 minute incremental sync. 


Edited by imadunatic, 23 October 2015 - 08:25 PM.


#2 Angelblue05 OFFLINE  

Angelblue05

    Advanced Member

  • Developers
  • 12705 posts
  • Local time: 04:31 PM

Posted 23 October 2015 - 08:40 PM

Do you sync trakt playstate to Emby with the schedule task?



#3 imadunatic OFFLINE  

imadunatic

    Advanced Member

  • Members
  • 95 posts
  • Local time: 05:31 PM

Posted 24 October 2015 - 06:38 AM

Do you sync trakt playstate to Emby with the schedule task?

I do not (at least I don't believe so), I disabled this a few weeks ago the last time I was playing with this (after reading through the incremental sync thread). 

 

This is my scheduled task: 

 

562b5f5b39895_Capture.jpg

 

 

Is it possible to search through the server logs to see if this is still occurring? I used to have playstates imported, is it possible this is residual damage from that? 

 

EDIT: Is that what these are in the logs: "u'LastPlayedDate': u'2015-09-07T23:55:49.0000000Z', u'PlaybackPositionTicks': 0, u'PlayCount': 4}, {u'ItemId': "  


Edited by imadunatic, 24 October 2015 - 06:40 AM.


#4 imadunatic OFFLINE  

imadunatic

    Advanced Member

  • Members
  • 95 posts
  • Local time: 05:31 PM

Posted 24 October 2015 - 06:46 AM

Is it possible this is causing an issue (from the server logs): 

 

 

2015-10-23 20:23:15.3953 Info - App: Emby.Kodi.SyncQueue: Server Time Requested...
2015-10-23 20:23:15.3953 Info - App: Emby.Kodi.SyncQueue: Returned Value: 2015-10-24T00:23:15Z
 
Obviously, at 8:23 last night, it was not 12:03 today... My system clock is right on the server, how do I insure Emby is getting the right time? Or is the server returned value GMT? 
 
 
Are the server logs sterile? Would it do any good to upload those? They're fairly large for log files... seem to be around 6mb/day some days, other days they are 150kb.... 

Edited by imadunatic, 24 October 2015 - 06:55 AM.


#5 xnappo OFFLINE  

xnappo

    Advanced Member

  • Developers
  • 4752 posts
  • Local time: 04:31 PM
  • LocationAustin, TX

Posted 24 October 2015 - 10:36 AM

I am having a problem where any items that fails lookup at themoviedb.org is getting marked as updated on every server scan.  I am not sure it this is related, but check to see if the items it is updating are also failing themoviedb lookup.



#6 imadunatic OFFLINE  

imadunatic

    Advanced Member

  • Members
  • 95 posts
  • Local time: 05:31 PM

Posted 24 October 2015 - 11:09 AM

I am having a problem where any items that fails lookup at themoviedb.org is getting marked as updated on every server scan.  I am not sure it this is related, but check to see if the items it is updating are also failing themoviedb lookup.

 

Is there an easy way to search for items failing themoviedb lookup? I don't recall anything off the top of my head that I struggle with to get it to show up, but I might be missing something. 



#7 xnappo OFFLINE  

xnappo

    Advanced Member

  • Developers
  • 4752 posts
  • Local time: 04:31 PM
  • LocationAustin, TX

Posted 24 October 2015 - 12:35 PM

You will see them in the server log pretty easily..



#8 imadunatic OFFLINE  

imadunatic

    Advanced Member

  • Members
  • 95 posts
  • Local time: 05:31 PM

Posted 24 October 2015 - 12:41 PM

You will see them in the server log pretty easily..

Thanks xnappo, I don't see anything in the logs about themoviedb lookup failures. 



#9 xnappo OFFLINE  

xnappo

    Advanced Member

  • Developers
  • 4752 posts
  • Local time: 04:31 PM
  • LocationAustin, TX

Posted 25 October 2015 - 11:11 AM

So almost certainly your problem is similar to mine, where for some reason particular items are being updated every time the server scans.

 

I had all sort of things causing this (but it is weird, because I didn't have the problem till recently):

 

1. .nfo files that confuse the scanner

2. TV shows not identified correctly all the way down to the episode level

3. Movies that cannot be identified

 

To fix this, you need to go to the web interface and go down to a movie level, then replace the id with the id from the log.

 

The id is in bold below:

http://192.168.1.2:8...fa6fe48c6d51a98

 

Try to figure out what that item would have updated.  If it is an item that you have configured manually, you can hit the 'Lock this item to prevent future changes' checkbox if all else fails.

 

xnappo

 

[EDIT] You did have to reset your DB somewhere around 1.1.50 right?


Edited by xnappo, 25 October 2015 - 11:17 AM.


#10 imadunatic OFFLINE  

imadunatic

    Advanced Member

  • Members
  • 95 posts
  • Local time: 05:31 PM

Posted 25 October 2015 - 07:28 PM

It appears that every time Emby refreshes people, it updates the NFO's of multiple (sometimes 100's) of NFO files. Every morning at 3 a.m. when it runs, this would fall right in line with what I'm seeing (once you let the incremental sync run, it's good for the rest of the day, the next day, you get a 1000 item incremental sync). 

 

What do I gain by letting it refresh people? 

 

Edit: Just checked, it updated 155 NFO's last night. 

 

Edit Edit: Yes, reset DB multiple times, I think it just reset at 1.1.53 also if I'm not mistaken. I've just switched everything back to full sync as they happen to be faster. I assume once I figure out why my NFO's are getting updated every morning, incremental sync will probably work. 


Edited by imadunatic, 25 October 2015 - 07:31 PM.


#11 dazed OFFLINE  

dazed

    Advanced Member

  • Members
  • 98 posts
  • Local time: 10:31 PM

Posted 25 October 2015 - 07:43 PM

 

Edit Edit: Yes, reset DB multiple times, I think it just reset at 1.1.53 also if I'm not mistaken. I've just switched everything back to full sync as they happen to be faster. I assume once I figure out why my NFO's are getting updated every morning, incremental sync will probably work. 

 If you use Sonarr make sure you turn its metadata off as this rewrites nfo files



#12 Angelblue05 OFFLINE  

Angelblue05

    Advanced Member

  • Developers
  • 12705 posts
  • Local time: 04:31 PM

Posted 25 October 2015 - 07:50 PM

I'm plagued with the refresh people nfo issue as well.


Sent from my iPhone using Tapatalk

#13 imadunatic OFFLINE  

imadunatic

    Advanced Member

  • Members
  • 95 posts
  • Local time: 05:31 PM

Posted 25 October 2015 - 08:17 PM

 If you use Sonarr make sure you turn its metadata off as this rewrites nfo files

 

I cannot confirm or deny the use of Sonarr on my computer... but I did just check and I have disabled Sonarr metadata. Only as of a couple weeks ago though, I've finally shaved my head and fully drank the kool aid and Emby is taking care of everything now. 

 

To note, I did have several movies with no themoviedb id, I've went through and corrected those.



#14 imadunatic OFFLINE  

imadunatic

    Advanced Member

  • Members
  • 95 posts
  • Local time: 05:31 PM

Posted 27 October 2015 - 10:17 AM

Well, I did disable the refresh people and as expected, no modified NFO's last night. Too soon to tell if it has fixed the incremental sync problem (I won't be home until tomorrow and I currently have it disabled); however, I personally don't care about refreshing people so I'm hoping this is the fix for incremental sync on my setups. 
 

Thanks to everyone for chiming in; I'm really looking forward to getting this working. 


  • xnappo and Angelblue05 like this

#15 imadunatic OFFLINE  

imadunatic

    Advanced Member

  • Members
  • 95 posts
  • Local time: 05:31 PM

Posted 04 November 2015 - 10:24 AM

Just an update: with the refresh people disabled, my incremental syncs generally take mere seconds to complete. I don't generally use the actors info for anything, so this was a net gain for me. Maybe this will help someone else out in the future. 

 

As an aside, thanks to the devs for the incremental sync! Now that I've got it working correctly, I can see what the hubbub was all about. 



#16 xnappo OFFLINE  

xnappo

    Advanced Member

  • Developers
  • 4752 posts
  • Local time: 04:31 PM
  • LocationAustin, TX

Posted 04 November 2015 - 10:50 AM

Yep - agreed.  @Luke - does this post indicate the people updates are going to change, or are you just referring to the popups here:

http://emby.media/co...t-notification/

 

xnappo



#17 xnappo OFFLINE  

xnappo

    Advanced Member

  • Developers
  • 4752 posts
  • Local time: 04:31 PM
  • LocationAustin, TX

Posted 04 November 2015 - 01:27 PM

Luke confirmed the above in that thread.



#18 imadunatic OFFLINE  

imadunatic

    Advanced Member

  • Members
  • 95 posts
  • Local time: 05:31 PM

Posted 04 November 2015 - 04:29 PM

I thought we fixed the people thing with a change the the server plugin to ignore people updates. Are you running the latest server plugin?

 

When I posted this, I was running sync queue version 1.0.5734.31705. I see now it has updated to v1.0.5782.13378; however, I've disabled people updates on the server side, so I'm not sure if it's still an issue for me or not. 



#19 Angelblue05 OFFLINE  

Angelblue05

    Advanced Member

  • Developers
  • 12705 posts
  • Local time: 04:31 PM

Posted 04 November 2015 - 07:02 PM

@null_pointer it's not part of the server plugin, however if you do have kodi running at the same time the scheduled task runs, it still sends all of them via websocket.


Sent from my iPhone using Tapatalk





Also tagged with one or more of these keywords: incremental sync, slow

0 user(s) are reading this topic

0 members, 0 guests, 0 anonymous users