Jump to content

Refresh people needs way longer in new release


Go to solution Solved by softworkz,

Recommended Posts

horstepipe
Posted

Hey

I set the "refresh people" task to once a week, normally it took about 40 minutes to an hour.

Now, on v3.0.6020 it runs almost 20 hours?

Have you made some changes here?

 

Regards

Posted

No, you probably just have more people that need refreshing now.

Posted

No, you probably just have more people that need refreshing now.

 

I mean, he didn't indicate adding more content or anything like that, where would all those new people come from? Also...20 hours? That seems excessive either way, no?

  • Like 1
Posted (edited)

Luke, I support which says horstepipe.

After take 3 refresh time followed in a row, the tiime is more than 2 hours in my case when before it was around 15 minutes.

 

(This occurs with last release or beta) 

Edited by Teddy
  • Like 1
Posted

No problem found here. Takes a matter of minutes.

horstepipe
Posted

kind of good to hear that I'm not the only one with that problem.

When beginning the task I'll get some errors in log, maybe it is helpful:

2016-08-03 01:59:23.5013 Error HttpClient: Error getting response from https://api.themoviedb.org/3/person/1222093?api_key=f6bd687ffa63cd282b6ff2c6877f2669&append_to_response=credits,images,external_ids
	*** Error Report ***
	Version: 3.0.6020.0
	Command line: /usr/local/emby/share/emby/MediaBrowser.Server.Mono.exe -programdata /usr/local/emby/var -ffmpeg /usr/local/ffmpeg/bin/ffmpeg -ffprobe /usr/local/ffmpeg/bin/ffprobe
	Operating system: Unix 3.2.40.0
	Processor count: 2
	64-Bit OS: False
	64-Bit Process: False
	Program data path: /usr/local/emby/var
	Mono: 4.2.1 (Stable 4.2.1.124/39edf24 Thu Feb 25 13:39:27 CST 2016)
	Application Path: /volume1/@appstore/emby/share/emby/MediaBrowser.Server.Mono.exe
	One or more errors occurred.
	System.AggregateException
	  at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw () <0x378bf698 + 0x00024> in <filename unknown>:0 
	  at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (System.Threading.Tasks.Task task) <0x378bf3a0 + 0x000bf> in <filename unknown>:0 
	  at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Threading.Tasks.Task task) <0x378bf2c0 + 0x0008b> in <filename unknown>:0 
	  at System.Runtime.CompilerServices.TaskAwaiter.ValidateEnd (System.Threading.Tasks.Task task) <0x2d59dab8 + 0x0003f> in <filename unknown>:0 
	  at System.Runtime.CompilerServices.ConfiguredTaskAwaitable`1+ConfiguredTaskAwaiter[TResult].GetResult () <0x2ef4dbf8 + 0x0001b> in <filename unknown>:0 
	  at MediaBrowser.Common.Implementations.HttpClientManager.HttpClientManager+<SendAsyncInternal>d__1d.MoveNext () <0x3378a070 + 0x010a7> in <filename unknown>:0 
	InnerException: System.Net.WebException
	The remote server returned an error: (404) Not Found.
	  at System.Net.HttpWebRequest.EndGetResponse (IAsyncResult asyncResult) <0x341d5080 + 0x00164> in <filename unknown>:0 
	  at System.Threading.Tasks.TaskFactory`1[TResult].FromAsyncCoreLogic (IAsyncResult iar, System.Func`2 endFunction, System.Action`1 endAction, System.Threading.Tasks.Task`1 promise, Boolean requiresSynchronization) <0x341d4c48 + 0x00087> in <filename unknown>:0 
	
2016-08-03 01:59:23.6278 Info HttpClient: HttpClientManager GET: https://api.themoviedb.org/3/person/1222093?api_key=f6bd687ffa63cd282b6ff2c6877f2669&append_to_response=credits,images,external_ids
2016-08-03 01:59:23.7917 Error HttpClient: Error getting response from https://api.themoviedb.org/3/person/1222093?api_key=f6bd687ffa63cd282b6ff2c6877f2669&append_to_response=credits,images,external_ids
	*** Error Report ***
	Version: 3.0.6020.0
	Command line: /usr/local/emby/share/emby/MediaBrowser.Server.Mono.exe -programdata /usr/local/emby/var -ffmpeg /usr/local/ffmpeg/bin/ffmpeg -ffprobe /usr/local/ffmpeg/bin/ffprobe
	Operating system: Unix 3.2.40.0
	Processor count: 2
	64-Bit OS: False
	64-Bit Process: False
	Program data path: /usr/local/emby/var
	Mono: 4.2.1 (Stable 4.2.1.124/39edf24 Thu Feb 25 13:39:27 CST 2016)
	Application Path: /volume1/@appstore/emby/share/emby/MediaBrowser.Server.Mono.exe
	One or more errors occurred.
	System.AggregateException
	  at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw () <0x378bf698 + 0x00024> in <filename unknown>:0 
	  at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (System.Threading.Tasks.Task task) <0x378bf3a0 + 0x000bf> in <filename unknown>:0 
	  at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Threading.Tasks.Task task) <0x378bf2c0 + 0x0008b> in <filename unknown>:0 
	  at System.Runtime.CompilerServices.TaskAwaiter.ValidateEnd (System.Threading.Tasks.Task task) <0x2d59dab8 + 0x0003f> in <filename unknown>:0 
	  at System.Runtime.CompilerServices.ConfiguredTaskAwaitable`1+ConfiguredTaskAwaiter[TResult].GetResult () <0x2ef4dbf8 + 0x0001b> in <filename unknown>:0 
	  at MediaBrowser.Common.Implementations.HttpClientManager.HttpClientManager+<SendAsyncInternal>d__1d.MoveNext () <0x3378a070 + 0x010a7> in <filename unknown>:0 
	InnerException: System.Net.WebException
	The remote server returned an error: (404) Not Found.
	  at System.Net.HttpWebRequest.EndGetResponse (IAsyncResult asyncResult) <0x341d5080 + 0x00164> in <filename unknown>:0 
	  at System.Threading.Tasks.TaskFactory`1[TResult].FromAsyncCoreLogic (IAsyncResult iar, System.Func`2 endFunction, System.Action`1 endAction, System.Threading.Tasks.Task`1 promise, Boolean requiresSynchronization) <0x341d4c48 + 0x00087> in <filename unknown>:0 
	
2016-08-03 01:59:23.7965 Error ProviderManager: MovieDbPersonImageProvider failed in GetImageInfos for type Person
	*** Error Report ***
	Version: 3.0.6020.0
	Command line: /usr/local/emby/share/emby/MediaBrowser.Server.Mono.exe -programdata /usr/local/emby/var -ffmpeg /usr/local/ffmpeg/bin/ffmpeg -ffprobe /usr/local/ffmpeg/bin/ffprobe
	Operating system: Unix 3.2.40.0
	Processor count: 2
	64-Bit OS: False
	64-Bit Process: False
	Program data path: /usr/local/emby/var
	Mono: 4.2.1 (Stable 4.2.1.124/39edf24 Thu Feb 25 13:39:27 CST 2016)
	Application Path: /volume1/@appstore/emby/share/emby/MediaBrowser.Server.Mono.exe
	One or more errors occurred.
	MediaBrowser.Model.Net.HttpException
	  at MediaBrowser.Common.Implementations.HttpClientManager.HttpClientManager+<SendAsyncInternal>d__1d.MoveNext () <0x3378a070 + 0x01744> in <filename unknown>:0 
	--- End of stack trace from previous location where exception was thrown ---
	  at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw () <0x378bf698 + 0x00024> in <filename unknown>:0 
	  at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (System.Threading.Tasks.Task task) <0x378bf3a0 + 0x000bf> in <filename unknown>:0 
	  at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Threading.Tasks.Task task) <0x378bf2c0 + 0x0008b> in <filename unknown>:0 
	  at System.Runtime.CompilerServices.TaskAwaiter.ValidateEnd (System.Threading.Tasks.Task task) <0x2d59dab8 + 0x0003f> in <filename unknown>:0 
	  at System.Runtime.CompilerServices.ConfiguredTaskAwaitable`1+ConfiguredTaskAwaiter[TResult].GetResult () <0x2ef4dbf8 + 0x0001b> in <filename unknown>:0 
	  at MediaBrowser.Common.Implementations.HttpClientManager.HttpClientManager+<SendAsync>d__7.MoveNext () <0x33787678 + 0x002bf> in <filename unknown>:0 
	--- End of stack trace from previous location where exception was thrown ---
	  at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw () <0x378bf698 + 0x00024> in <filename unknown>:0 
	  at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (System.Threading.Tasks.Task task) <0x378bf3a0 + 0x000bf> in <filename unknown>:0 
	  at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Threading.Tasks.Task task) <0x378bf2c0 + 0x0008b> in <filename unknown>:0 
	  at System.Runtime.CompilerServices.TaskAwaiter.ValidateEnd (System.Threading.Tasks.Task task) <0x2d59dab8 + 0x0003f> in <filename unknown>:0 
	  at System.Runtime.CompilerServices.ConfiguredTaskAwaitable`1+ConfiguredTaskAwaiter[TResult].GetResult () <0x2ef4dbf8 + 0x0001b> in <filename unknown>:0 
	  at MediaBrowser.Common.Implementations.HttpClientManager.HttpClientManager+<Get>d__2.MoveNext () <0x341a7f28 + 0x001d7> in <filename unknown>:0 
	--- End of stack trace from previous location where exception was thrown ---
	  at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw () <0x378bf698 + 0x00024> in <filename unknown>:0 
	  at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (System.Threading.Tasks.Task task) <0x378bf3a0 + 0x000bf> in <filename unknown>:0 
	  at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Threading.Tasks.Task task) <0x378bf2c0 + 0x0008b> in <filename unknown>:0 
	  at System.Runtime.CompilerServices.TaskAwaiter.ValidateEnd (System.Threading.Tasks.Task task) <0x2d59dab8 + 0x0003f> in <filename unknown>:0 
	  at System.Runtime.CompilerServices.ConfiguredTaskAwaitable`1+ConfiguredTaskAwaiter[TResult].GetResult () <0x2ef4dbf8 + 0x0001b> in <filename unknown>:0 
	  at MediaBrowser.Providers.Movies.MovieDbProvider+<GetMovieDbResponse>d__21.MoveNext () <0x3166f2d8 + 0x00827> in <filename unknown>:0 
	--- End of stack trace from previous location where exception was thrown ---
	  at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw () <0x378bf698 + 0x00024> in <filename unknown>:0 
	  at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (System.Threading.Tasks.Task task) <0x378bf3a0 + 0x000bf> in <filename unknown>:0 
	  at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Threading.Tasks.Task task) <0x378bf2c0 + 0x0008b> in <filename unknown>:0 
	  at System.Runtime.CompilerServices.TaskAwaiter.ValidateEnd (System.Threading.Tasks.Task task) <0x2d59dab8 + 0x0003f> in <filename unknown>:0 
	  at System.Runtime.CompilerServices.ConfiguredTaskAwaitable`1+ConfiguredTaskAwaiter[TResult].GetResult () <0x2ef4dbf8 + 0x0001b> in <filename unknown>:0 
	  at MediaBrowser.Providers.People.MovieDbPersonProvider+<EnsurePersonInfo>d__26.MoveNext () <0x2e42b430 + 0x00517> in <filename unknown>:0 
	--- End of stack trace from previous location where exception was thrown ---
	  at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw () <0x378bf698 + 0x00024> in <filename unknown>:0 
	  at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (System.Threading.Tasks.Task task) <0x378bf3a0 + 0x000bf> in <filename unknown>:0 
	  at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Threading.Tasks.Task task) <0x378bf2c0 + 0x0008b> in <filename unknown>:0 
	  at System.Runtime.CompilerServices.TaskAwaiter.ValidateEnd (System.Threading.Tasks.Task task) <0x2d59dab8 + 0x0003f> in <filename unknown>:0 
	  at System.Runtime.CompilerServices.ConfiguredTaskAwaitable+ConfiguredTaskAwaiter.GetResult () <0x2d59da70 + 0x00017> in <filename unknown>:0 
	  at MediaBrowser.Providers.People.MovieDbPersonImageProvider+<GetImages>d__1.MoveNext () <0x3168ecc8 + 0x0037b> in <filename unknown>:0 
	--- End of stack trace from previous location where exception was thrown ---
	  at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw () <0x378bf698 + 0x00024> in <filename unknown>:0 
	  at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (System.Threading.Tasks.Task task) <0x378bf3a0 + 0x000bf> in <filename unknown>:0 
	  at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Threading.Tasks.Task task) <0x378bf2c0 + 0x0008b> in <filename unknown>:0 
	  at System.Runtime.CompilerServices.TaskAwaiter.ValidateEnd (System.Threading.Tasks.Task task) <0x2d59dab8 + 0x0003f> in <filename unknown>:0 
	  at System.Runtime.CompilerServices.ConfiguredTaskAwaitable`1+ConfiguredTaskAwaiter[TResult].GetResult () <0x2ef4dbf8 + 0x0001b> in <filename unknown>:0 
	  at MediaBrowser.Providers.Manager.ProviderManager+<GetImages>d__25.MoveNext () <0x3168e030 + 0x002c7> in <filename unknown>:0 
	InnerException: System.AggregateException
	One or more errors occurred.
	  at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw () <0x378bf698 + 0x00024> in <filename unknown>:0 
	  at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (System.Threading.Tasks.Task task) <0x378bf3a0 + 0x000bf> in <filename unknown>:0 
	  at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Threading.Tasks.Task task) <0x378bf2c0 + 0x0008b> in <filename unknown>:0 
	  at System.Runtime.CompilerServices.TaskAwaiter.ValidateEnd (System.Threading.Tasks.Task task) <0x2d59dab8 + 0x0003f> in <filename unknown>:0 
	  at System.Runtime.CompilerServices.ConfiguredTaskAwaitable`1+ConfiguredTaskAwaiter[TResult].GetResult () <0x2ef4dbf8 + 0x0001b> in <filename unknown>:0 
	  at MediaBrowser.Common.Implementations.HttpClientManager.HttpClientManager+<SendAsyncInternal>d__1d.MoveNext () <0x3378a070 + 0x010a7> in <filename unknown>:0 
	InnerException: System.Net.WebException
	The remote server returned an error: (404) Not Found.
	  at System.Net.HttpWebRequest.EndGetResponse (IAsyncResult asyncResult) <0x341d5080 + 0x00164> in <filename unknown>:0 
	  at System.Threading.Tasks.TaskFactory`1[TResult].FromAsyncCoreLogic (IAsyncResult iar, System.Func`2 endFunction, System.Action`1 endAction, System.Threading.Tasks.Task`1 promise, Boolean requiresSynchronization) <0x341d4c48 + 0x00087> in <filename unknown>:0 
Posted

You probably have 50,000 people in your database. The people refresh process is checking to see if any of the people xml files have changed, and for some people it may also be refreshing from the moviedb. Considering the large number of people in the database this is not going to be a fast process. 

 

I would suggest going to Library -> Advanced -> People, and unchecking all of the checkboxes, then click save. 

horstepipe
Posted

You probably have 50,000 people in your database. The people refresh process is checking to see if any of the people xml files have changed, and for some people it may also be refreshing from the moviedb. Considering the large number of people in the database this is not going to be a fast process. 

 

I would suggest going to Library -> Advanced -> People, and unchecking all of the checkboxes, then click save. 

 

But the process went well for months now, and my database didn't get bigger (just a couple of movies and episodes) in the last weeks.

Posted (edited)

My "Refresh People" task is also taking far longer to run, I do have a large collection and I do have all the option enabled under Library -> Advanced -> People, but I've not made any changes to these settings and now from taking about 3 hours it now takes nearly 32 hours  :blink:

57a1d4c78c80f_RefreshPeople.jpg

Server Log attached  ;)

server-63605779199.zip

Edited by Oxide
Posted

Same here ... before update the refresh people scan would take @ 1 minute, now it takes @ 5 mins minimum ..... not as extreme as others mentioned here but still takes @ 5 times longer than it used to. (All my metadata, including people, is populated by an external prog, not Emby)  :)

horstepipe
Posted

@@Luke

please stop ignoring the fact that something's definitely going wrong in the new releases.

Tell us what information you need to identify the problem.

 

Regards

Posted (edited)

@@horstepipe

 

I've not checked my 'Refresh People' since the update. But I'm assuming what would help would be a log file from someone before they upgraded and then after they upgraded (And noticed a difference) so that @@Luke could compare and see what additional workload has been added and where it has come from. 

 

Having logs from the same library would then rule out differences in database size etc

Edited by llygoden
horstepipe
Posted

@@horstepipe

 

I've not checked my 'Refresh People' since the update. But I'm assuming what would help would be a log file from someone before they upgraded and then after they upgraded (And noticed a difference) so that @@Luke could compare and see what additional workload has been added and where it has come from. 

 

Having logs from the same library would then rule out differences in database size etc

 

Unfortunately I could only provide a log file after I upgraded.

I'll start the task now and provide the file tomorrow...or whenever it's finished :-/

Posted

My "Refresh People" task is also taking far longer to run, I do have a large collection and I do have all the option enabled under Library -> Advanced -> People, but I've not made any changes to these settings and now from taking about 3 hours it now takes nearly 32 hours  :blink:

57a1d4c78c80f_RefreshPeople.jpg

Server Log attached  ;)

 

Try going to library -> advanced -> people and unchecked all of the checkboxes.

Posted

@@Luke I know that will speed it up, but I've always had those options enabled, what we are saying is that with the newer builds this has slowed down big time. Something has changed  :huh:

  • Like 1
horstepipe
Posted

@@Luke

I just hope that you have a bad day today,

Your "solution" can't be meant seriously ;-)

Posted

What would you like me to say? The people refresh is checking every xml file to see if it's changed, and then it retries people with empty metadata that haven't been retried for at least 30 days. If you have 50,000 people or 100,000 people in your database this is something that is going to take time. So probably what's different from before is that more people are "up for retry" than in the past. We might just have to go in the direction of not downloading any metadata until you actually click on the person. Then your people refresh will be very fast because there won't be one at all. Or maybe we can just stop retrying people with empty metadata but then of course that brings out another crowd of users complaining that their people have no biographies. So i guess you tell us what you would like but don't say that you'd like 100,000 movie db queries handled in 5 minutes because it just doesn't work that way. 

Guest asrequested
Posted

Maybe only download metadata when you go to the details page of the file you're about to play? The same for refresh, only do it as needed? I haven't looked how long mine takes, and I don't feel the need to... Just adding my two cents :)

Posted

I think downloading people metadata on demand when you open the detail screen is a good approach, but obviously there are going to be others who want to get it all up front.

spootdev
Posted

What would you like me to say? The people refresh is checking every xml file to see if it's changed, and then it retries people with empty metadata that haven't been retried for at least 30 days. If you have 50,000 people or 100,000 people in your database this is something that is going to take time. So probably what's different from before is that more people are "up for retry" than in the past. We might just have to go in the direction of not downloading any metadata until you actually click on the person. Then your people refresh will be very fast because there won't be one at all. Or maybe we can just stop retrying people with empty metadata but then of course that brings out another crowd of users complaining that their people have no biographies. So i guess you tell us what you would like but don't say that you'd like 100,000 movie db queries handled in 5 minutes because it just doesn't work that way. 

For the love of monkies please tell me it's not coded to actually do it this way.........

In themoviedb API: /person/changes

Posted (edited)

Luke, I understand your explanation that 50,000 people need time to download the information, but I think our question is whether before with other versions took a few minutes and are now hours. It means that did not do well before and you've fixed to collect all missed information that were before not collect?

Edited by Teddy
horstepipe
Posted (edited)

The people refresh is checking every xml file to see if it's changed, and then it retries people with empty metadata that haven't been retried for at least 30 days. If you have 50,000 people or 100,000 people in your database this is something that is going to take time. So probably what's different from before is that more people are "up for retry" than in the past.

 

What would you like me to say? The people refresh is checking every xml file to see if it's changed, and then it retries people with empty metadata that haven't been retried for at least 30 days. If you have 50,000 people or 100,000 people in your database this is something that is going to take time. So probably what's different from before is that more people are "up for retry" than in the past. We might just have to go in the direction of not downloading any metadata until you actually click on the person. Then your people refresh will be very fast because there won't be one at all. Or maybe we can just stop retrying people with empty metadata but then of course that brings out another crowd of users complaining that their people have no biographies. So i guess you tell us what you would like but don't say that you'd like 100,000 movie db queries handled in 5 minutes because it just doesn't work that way.

 

Luke, what you say makes no sense or I misunderstand you.

"At least 30 days"

But the last refresh is only one day ago. And it looks that the refresh I started today needs even longer...

So your argumentation is senseless or I'm getting something completely wrong.

 

Sorry, I don't wanna be offensive, but it makes the impression that your support regarding this problem is very unmotivated.

Edited by horstepipe
  • Like 2
Guest asrequested
Posted

What Luke is saying is not everything refreshes every time. Let's say you added a bunch of movies at one time. It would be 30 days for them to be refreshed. In the time between, others will be refreshed. So there won't be the same refresh each time. Depending how much you add, and when, there will be variation of activity.

horstepipe
Posted

What Luke is saying is not everything refreshes every time. Let's say you added a bunch of movies at one time. It would be 30 days for them to be refreshed. In the time between, others will be refreshed. So there won't be the same refresh each time. Depending how much you add, and when, there will be variation of activity.

Ok I understand, but I'm absolutely sure that I added 95% of my whole collection on one day.

The only scenario I could imagine is that the database creation process went over night (so theoretically over 2 days).

If that's the case my third refresh tomorrow should be quick again, when my calculation is right.

Guest asrequested
Posted (edited)

And I think that is probably quite common, as a lot of us will sometimes rebuild, so it all gets added at one time.

 

Luke, maybe an option to toggle between how it is now and download on demand? With how it is now being the default?

Edited by Doofus

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now
×
×
  • Create New...