Jump to content

Refresh people needs way longer in new release


horstepipe
Go to solution Solved by softworkz,

Recommended Posts

horstepipe

Hey

I set the "refresh people" task to once a week, normally it took about 40 minutes to an hour.

Now, on v3.0.6020 it runs almost 20 hours?

Have you made some changes here?

 

Regards

Link to comment
Share on other sites

bfir3

No, you probably just have more people that need refreshing now.

 

I mean, he didn't indicate adding more content or anything like that, where would all those new people come from? Also...20 hours? That seems excessive either way, no?

  • Like 1
Link to comment
Share on other sites

Teddy

Luke, I support which says horstepipe.

After take 3 refresh time followed in a row, the tiime is more than 2 hours in my case when before it was around 15 minutes.

 

(This occurs with last release or beta) 

Edited by Teddy
  • Like 1
Link to comment
Share on other sites

horstepipe

kind of good to hear that I'm not the only one with that problem.

When beginning the task I'll get some errors in log, maybe it is helpful:

2016-08-03 01:59:23.5013 Error HttpClient: Error getting response from https://api.themoviedb.org/3/person/1222093?api_key=f6bd687ffa63cd282b6ff2c6877f2669&append_to_response=credits,images,external_ids
	*** Error Report ***
	Version: 3.0.6020.0
	Command line: /usr/local/emby/share/emby/MediaBrowser.Server.Mono.exe -programdata /usr/local/emby/var -ffmpeg /usr/local/ffmpeg/bin/ffmpeg -ffprobe /usr/local/ffmpeg/bin/ffprobe
	Operating system: Unix 3.2.40.0
	Processor count: 2
	64-Bit OS: False
	64-Bit Process: False
	Program data path: /usr/local/emby/var
	Mono: 4.2.1 (Stable 4.2.1.124/39edf24 Thu Feb 25 13:39:27 CST 2016)
	Application Path: /volume1/@appstore/emby/share/emby/MediaBrowser.Server.Mono.exe
	One or more errors occurred.
	System.AggregateException
	  at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw () <0x378bf698 + 0x00024> in <filename unknown>:0 
	  at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (System.Threading.Tasks.Task task) <0x378bf3a0 + 0x000bf> in <filename unknown>:0 
	  at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Threading.Tasks.Task task) <0x378bf2c0 + 0x0008b> in <filename unknown>:0 
	  at System.Runtime.CompilerServices.TaskAwaiter.ValidateEnd (System.Threading.Tasks.Task task) <0x2d59dab8 + 0x0003f> in <filename unknown>:0 
	  at System.Runtime.CompilerServices.ConfiguredTaskAwaitable`1+ConfiguredTaskAwaiter[TResult].GetResult () <0x2ef4dbf8 + 0x0001b> in <filename unknown>:0 
	  at MediaBrowser.Common.Implementations.HttpClientManager.HttpClientManager+<SendAsyncInternal>d__1d.MoveNext () <0x3378a070 + 0x010a7> in <filename unknown>:0 
	InnerException: System.Net.WebException
	The remote server returned an error: (404) Not Found.
	  at System.Net.HttpWebRequest.EndGetResponse (IAsyncResult asyncResult) <0x341d5080 + 0x00164> in <filename unknown>:0 
	  at System.Threading.Tasks.TaskFactory`1[TResult].FromAsyncCoreLogic (IAsyncResult iar, System.Func`2 endFunction, System.Action`1 endAction, System.Threading.Tasks.Task`1 promise, Boolean requiresSynchronization) <0x341d4c48 + 0x00087> in <filename unknown>:0 
	
2016-08-03 01:59:23.6278 Info HttpClient: HttpClientManager GET: https://api.themoviedb.org/3/person/1222093?api_key=f6bd687ffa63cd282b6ff2c6877f2669&append_to_response=credits,images,external_ids
2016-08-03 01:59:23.7917 Error HttpClient: Error getting response from https://api.themoviedb.org/3/person/1222093?api_key=f6bd687ffa63cd282b6ff2c6877f2669&append_to_response=credits,images,external_ids
	*** Error Report ***
	Version: 3.0.6020.0
	Command line: /usr/local/emby/share/emby/MediaBrowser.Server.Mono.exe -programdata /usr/local/emby/var -ffmpeg /usr/local/ffmpeg/bin/ffmpeg -ffprobe /usr/local/ffmpeg/bin/ffprobe
	Operating system: Unix 3.2.40.0
	Processor count: 2
	64-Bit OS: False
	64-Bit Process: False
	Program data path: /usr/local/emby/var
	Mono: 4.2.1 (Stable 4.2.1.124/39edf24 Thu Feb 25 13:39:27 CST 2016)
	Application Path: /volume1/@appstore/emby/share/emby/MediaBrowser.Server.Mono.exe
	One or more errors occurred.
	System.AggregateException
	  at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw () <0x378bf698 + 0x00024> in <filename unknown>:0 
	  at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (System.Threading.Tasks.Task task) <0x378bf3a0 + 0x000bf> in <filename unknown>:0 
	  at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Threading.Tasks.Task task) <0x378bf2c0 + 0x0008b> in <filename unknown>:0 
	  at System.Runtime.CompilerServices.TaskAwaiter.ValidateEnd (System.Threading.Tasks.Task task) <0x2d59dab8 + 0x0003f> in <filename unknown>:0 
	  at System.Runtime.CompilerServices.ConfiguredTaskAwaitable`1+ConfiguredTaskAwaiter[TResult].GetResult () <0x2ef4dbf8 + 0x0001b> in <filename unknown>:0 
	  at MediaBrowser.Common.Implementations.HttpClientManager.HttpClientManager+<SendAsyncInternal>d__1d.MoveNext () <0x3378a070 + 0x010a7> in <filename unknown>:0 
	InnerException: System.Net.WebException
	The remote server returned an error: (404) Not Found.
	  at System.Net.HttpWebRequest.EndGetResponse (IAsyncResult asyncResult) <0x341d5080 + 0x00164> in <filename unknown>:0 
	  at System.Threading.Tasks.TaskFactory`1[TResult].FromAsyncCoreLogic (IAsyncResult iar, System.Func`2 endFunction, System.Action`1 endAction, System.Threading.Tasks.Task`1 promise, Boolean requiresSynchronization) <0x341d4c48 + 0x00087> in <filename unknown>:0 
	
2016-08-03 01:59:23.7965 Error ProviderManager: MovieDbPersonImageProvider failed in GetImageInfos for type Person
	*** Error Report ***
	Version: 3.0.6020.0
	Command line: /usr/local/emby/share/emby/MediaBrowser.Server.Mono.exe -programdata /usr/local/emby/var -ffmpeg /usr/local/ffmpeg/bin/ffmpeg -ffprobe /usr/local/ffmpeg/bin/ffprobe
	Operating system: Unix 3.2.40.0
	Processor count: 2
	64-Bit OS: False
	64-Bit Process: False
	Program data path: /usr/local/emby/var
	Mono: 4.2.1 (Stable 4.2.1.124/39edf24 Thu Feb 25 13:39:27 CST 2016)
	Application Path: /volume1/@appstore/emby/share/emby/MediaBrowser.Server.Mono.exe
	One or more errors occurred.
	MediaBrowser.Model.Net.HttpException
	  at MediaBrowser.Common.Implementations.HttpClientManager.HttpClientManager+<SendAsyncInternal>d__1d.MoveNext () <0x3378a070 + 0x01744> in <filename unknown>:0 
	--- End of stack trace from previous location where exception was thrown ---
	  at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw () <0x378bf698 + 0x00024> in <filename unknown>:0 
	  at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (System.Threading.Tasks.Task task) <0x378bf3a0 + 0x000bf> in <filename unknown>:0 
	  at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Threading.Tasks.Task task) <0x378bf2c0 + 0x0008b> in <filename unknown>:0 
	  at System.Runtime.CompilerServices.TaskAwaiter.ValidateEnd (System.Threading.Tasks.Task task) <0x2d59dab8 + 0x0003f> in <filename unknown>:0 
	  at System.Runtime.CompilerServices.ConfiguredTaskAwaitable`1+ConfiguredTaskAwaiter[TResult].GetResult () <0x2ef4dbf8 + 0x0001b> in <filename unknown>:0 
	  at MediaBrowser.Common.Implementations.HttpClientManager.HttpClientManager+<SendAsync>d__7.MoveNext () <0x33787678 + 0x002bf> in <filename unknown>:0 
	--- End of stack trace from previous location where exception was thrown ---
	  at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw () <0x378bf698 + 0x00024> in <filename unknown>:0 
	  at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (System.Threading.Tasks.Task task) <0x378bf3a0 + 0x000bf> in <filename unknown>:0 
	  at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Threading.Tasks.Task task) <0x378bf2c0 + 0x0008b> in <filename unknown>:0 
	  at System.Runtime.CompilerServices.TaskAwaiter.ValidateEnd (System.Threading.Tasks.Task task) <0x2d59dab8 + 0x0003f> in <filename unknown>:0 
	  at System.Runtime.CompilerServices.ConfiguredTaskAwaitable`1+ConfiguredTaskAwaiter[TResult].GetResult () <0x2ef4dbf8 + 0x0001b> in <filename unknown>:0 
	  at MediaBrowser.Common.Implementations.HttpClientManager.HttpClientManager+<Get>d__2.MoveNext () <0x341a7f28 + 0x001d7> in <filename unknown>:0 
	--- End of stack trace from previous location where exception was thrown ---
	  at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw () <0x378bf698 + 0x00024> in <filename unknown>:0 
	  at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (System.Threading.Tasks.Task task) <0x378bf3a0 + 0x000bf> in <filename unknown>:0 
	  at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Threading.Tasks.Task task) <0x378bf2c0 + 0x0008b> in <filename unknown>:0 
	  at System.Runtime.CompilerServices.TaskAwaiter.ValidateEnd (System.Threading.Tasks.Task task) <0x2d59dab8 + 0x0003f> in <filename unknown>:0 
	  at System.Runtime.CompilerServices.ConfiguredTaskAwaitable`1+ConfiguredTaskAwaiter[TResult].GetResult () <0x2ef4dbf8 + 0x0001b> in <filename unknown>:0 
	  at MediaBrowser.Providers.Movies.MovieDbProvider+<GetMovieDbResponse>d__21.MoveNext () <0x3166f2d8 + 0x00827> in <filename unknown>:0 
	--- End of stack trace from previous location where exception was thrown ---
	  at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw () <0x378bf698 + 0x00024> in <filename unknown>:0 
	  at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (System.Threading.Tasks.Task task) <0x378bf3a0 + 0x000bf> in <filename unknown>:0 
	  at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Threading.Tasks.Task task) <0x378bf2c0 + 0x0008b> in <filename unknown>:0 
	  at System.Runtime.CompilerServices.TaskAwaiter.ValidateEnd (System.Threading.Tasks.Task task) <0x2d59dab8 + 0x0003f> in <filename unknown>:0 
	  at System.Runtime.CompilerServices.ConfiguredTaskAwaitable`1+ConfiguredTaskAwaiter[TResult].GetResult () <0x2ef4dbf8 + 0x0001b> in <filename unknown>:0 
	  at MediaBrowser.Providers.People.MovieDbPersonProvider+<EnsurePersonInfo>d__26.MoveNext () <0x2e42b430 + 0x00517> in <filename unknown>:0 
	--- End of stack trace from previous location where exception was thrown ---
	  at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw () <0x378bf698 + 0x00024> in <filename unknown>:0 
	  at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (System.Threading.Tasks.Task task) <0x378bf3a0 + 0x000bf> in <filename unknown>:0 
	  at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Threading.Tasks.Task task) <0x378bf2c0 + 0x0008b> in <filename unknown>:0 
	  at System.Runtime.CompilerServices.TaskAwaiter.ValidateEnd (System.Threading.Tasks.Task task) <0x2d59dab8 + 0x0003f> in <filename unknown>:0 
	  at System.Runtime.CompilerServices.ConfiguredTaskAwaitable+ConfiguredTaskAwaiter.GetResult () <0x2d59da70 + 0x00017> in <filename unknown>:0 
	  at MediaBrowser.Providers.People.MovieDbPersonImageProvider+<GetImages>d__1.MoveNext () <0x3168ecc8 + 0x0037b> in <filename unknown>:0 
	--- End of stack trace from previous location where exception was thrown ---
	  at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw () <0x378bf698 + 0x00024> in <filename unknown>:0 
	  at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (System.Threading.Tasks.Task task) <0x378bf3a0 + 0x000bf> in <filename unknown>:0 
	  at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Threading.Tasks.Task task) <0x378bf2c0 + 0x0008b> in <filename unknown>:0 
	  at System.Runtime.CompilerServices.TaskAwaiter.ValidateEnd (System.Threading.Tasks.Task task) <0x2d59dab8 + 0x0003f> in <filename unknown>:0 
	  at System.Runtime.CompilerServices.ConfiguredTaskAwaitable`1+ConfiguredTaskAwaiter[TResult].GetResult () <0x2ef4dbf8 + 0x0001b> in <filename unknown>:0 
	  at MediaBrowser.Providers.Manager.ProviderManager+<GetImages>d__25.MoveNext () <0x3168e030 + 0x002c7> in <filename unknown>:0 
	InnerException: System.AggregateException
	One or more errors occurred.
	  at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw () <0x378bf698 + 0x00024> in <filename unknown>:0 
	  at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (System.Threading.Tasks.Task task) <0x378bf3a0 + 0x000bf> in <filename unknown>:0 
	  at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Threading.Tasks.Task task) <0x378bf2c0 + 0x0008b> in <filename unknown>:0 
	  at System.Runtime.CompilerServices.TaskAwaiter.ValidateEnd (System.Threading.Tasks.Task task) <0x2d59dab8 + 0x0003f> in <filename unknown>:0 
	  at System.Runtime.CompilerServices.ConfiguredTaskAwaitable`1+ConfiguredTaskAwaiter[TResult].GetResult () <0x2ef4dbf8 + 0x0001b> in <filename unknown>:0 
	  at MediaBrowser.Common.Implementations.HttpClientManager.HttpClientManager+<SendAsyncInternal>d__1d.MoveNext () <0x3378a070 + 0x010a7> in <filename unknown>:0 
	InnerException: System.Net.WebException
	The remote server returned an error: (404) Not Found.
	  at System.Net.HttpWebRequest.EndGetResponse (IAsyncResult asyncResult) <0x341d5080 + 0x00164> in <filename unknown>:0 
	  at System.Threading.Tasks.TaskFactory`1[TResult].FromAsyncCoreLogic (IAsyncResult iar, System.Func`2 endFunction, System.Action`1 endAction, System.Threading.Tasks.Task`1 promise, Boolean requiresSynchronization) <0x341d4c48 + 0x00087> in <filename unknown>:0 
Link to comment
Share on other sites

You probably have 50,000 people in your database. The people refresh process is checking to see if any of the people xml files have changed, and for some people it may also be refreshing from the moviedb. Considering the large number of people in the database this is not going to be a fast process. 

 

I would suggest going to Library -> Advanced -> People, and unchecking all of the checkboxes, then click save. 

Link to comment
Share on other sites

horstepipe

You probably have 50,000 people in your database. The people refresh process is checking to see if any of the people xml files have changed, and for some people it may also be refreshing from the moviedb. Considering the large number of people in the database this is not going to be a fast process. 

 

I would suggest going to Library -> Advanced -> People, and unchecking all of the checkboxes, then click save. 

 

But the process went well for months now, and my database didn't get bigger (just a couple of movies and episodes) in the last weeks.

Link to comment
Share on other sites

Oxide

My "Refresh People" task is also taking far longer to run, I do have a large collection and I do have all the option enabled under Library -> Advanced -> People, but I've not made any changes to these settings and now from taking about 3 hours it now takes nearly 32 hours  :blink:

57a1d4c78c80f_RefreshPeople.jpg

Server Log attached  ;)

server-63605779199.zip

Edited by Oxide
Link to comment
Share on other sites

mj3310

Same here ... before update the refresh people scan would take @ 1 minute, now it takes @ 5 mins minimum ..... not as extreme as others mentioned here but still takes @ 5 times longer than it used to. (All my metadata, including people, is populated by an external prog, not Emby)  :)

Link to comment
Share on other sites

horstepipe

@@Luke

please stop ignoring the fact that something's definitely going wrong in the new releases.

Tell us what information you need to identify the problem.

 

Regards

Link to comment
Share on other sites

llygoden

@@horstepipe

 

I've not checked my 'Refresh People' since the update. But I'm assuming what would help would be a log file from someone before they upgraded and then after they upgraded (And noticed a difference) so that @@Luke could compare and see what additional workload has been added and where it has come from. 

 

Having logs from the same library would then rule out differences in database size etc

Edited by llygoden
Link to comment
Share on other sites

horstepipe

@@horstepipe

 

I've not checked my 'Refresh People' since the update. But I'm assuming what would help would be a log file from someone before they upgraded and then after they upgraded (And noticed a difference) so that @@Luke could compare and see what additional workload has been added and where it has come from. 

 

Having logs from the same library would then rule out differences in database size etc

 

Unfortunately I could only provide a log file after I upgraded.

I'll start the task now and provide the file tomorrow...or whenever it's finished :-/

Link to comment
Share on other sites

My "Refresh People" task is also taking far longer to run, I do have a large collection and I do have all the option enabled under Library -> Advanced -> People, but I've not made any changes to these settings and now from taking about 3 hours it now takes nearly 32 hours  :blink:

57a1d4c78c80f_RefreshPeople.jpg

Server Log attached  ;)

 

Try going to library -> advanced -> people and unchecked all of the checkboxes.

Link to comment
Share on other sites

Oxide

@@Luke I know that will speed it up, but I've always had those options enabled, what we are saying is that with the newer builds this has slowed down big time. Something has changed  :huh:

  • Like 1
Link to comment
Share on other sites

What would you like me to say? The people refresh is checking every xml file to see if it's changed, and then it retries people with empty metadata that haven't been retried for at least 30 days. If you have 50,000 people or 100,000 people in your database this is something that is going to take time. So probably what's different from before is that more people are "up for retry" than in the past. We might just have to go in the direction of not downloading any metadata until you actually click on the person. Then your people refresh will be very fast because there won't be one at all. Or maybe we can just stop retrying people with empty metadata but then of course that brings out another crowd of users complaining that their people have no biographies. So i guess you tell us what you would like but don't say that you'd like 100,000 movie db queries handled in 5 minutes because it just doesn't work that way. 

Link to comment
Share on other sites

Guest asrequested

Maybe only download metadata when you go to the details page of the file you're about to play? The same for refresh, only do it as needed? I haven't looked how long mine takes, and I don't feel the need to... Just adding my two cents :)

Link to comment
Share on other sites

I think downloading people metadata on demand when you open the detail screen is a good approach, but obviously there are going to be others who want to get it all up front.

Link to comment
Share on other sites

spootdev

What would you like me to say? The people refresh is checking every xml file to see if it's changed, and then it retries people with empty metadata that haven't been retried for at least 30 days. If you have 50,000 people or 100,000 people in your database this is something that is going to take time. So probably what's different from before is that more people are "up for retry" than in the past. We might just have to go in the direction of not downloading any metadata until you actually click on the person. Then your people refresh will be very fast because there won't be one at all. Or maybe we can just stop retrying people with empty metadata but then of course that brings out another crowd of users complaining that their people have no biographies. So i guess you tell us what you would like but don't say that you'd like 100,000 movie db queries handled in 5 minutes because it just doesn't work that way. 

For the love of monkies please tell me it's not coded to actually do it this way.........

In themoviedb API: /person/changes

Link to comment
Share on other sites

Teddy

Luke, I understand your explanation that 50,000 people need time to download the information, but I think our question is whether before with other versions took a few minutes and are now hours. It means that did not do well before and you've fixed to collect all missed information that were before not collect?

Edited by Teddy
Link to comment
Share on other sites

horstepipe

The people refresh is checking every xml file to see if it's changed, and then it retries people with empty metadata that haven't been retried for at least 30 days. If you have 50,000 people or 100,000 people in your database this is something that is going to take time. So probably what's different from before is that more people are "up for retry" than in the past.

 

What would you like me to say? The people refresh is checking every xml file to see if it's changed, and then it retries people with empty metadata that haven't been retried for at least 30 days. If you have 50,000 people or 100,000 people in your database this is something that is going to take time. So probably what's different from before is that more people are "up for retry" than in the past. We might just have to go in the direction of not downloading any metadata until you actually click on the person. Then your people refresh will be very fast because there won't be one at all. Or maybe we can just stop retrying people with empty metadata but then of course that brings out another crowd of users complaining that their people have no biographies. So i guess you tell us what you would like but don't say that you'd like 100,000 movie db queries handled in 5 minutes because it just doesn't work that way.

 

Luke, what you say makes no sense or I misunderstand you.

"At least 30 days"

But the last refresh is only one day ago. And it looks that the refresh I started today needs even longer...

So your argumentation is senseless or I'm getting something completely wrong.

 

Sorry, I don't wanna be offensive, but it makes the impression that your support regarding this problem is very unmotivated.

Edited by horstepipe
  • Like 2
Link to comment
Share on other sites

Guest asrequested

What Luke is saying is not everything refreshes every time. Let's say you added a bunch of movies at one time. It would be 30 days for them to be refreshed. In the time between, others will be refreshed. So there won't be the same refresh each time. Depending how much you add, and when, there will be variation of activity.

Link to comment
Share on other sites

horstepipe

What Luke is saying is not everything refreshes every time. Let's say you added a bunch of movies at one time. It would be 30 days for them to be refreshed. In the time between, others will be refreshed. So there won't be the same refresh each time. Depending how much you add, and when, there will be variation of activity.

Ok I understand, but I'm absolutely sure that I added 95% of my whole collection on one day.

The only scenario I could imagine is that the database creation process went over night (so theoretically over 2 days).

If that's the case my third refresh tomorrow should be quick again, when my calculation is right.

Link to comment
Share on other sites

Guest asrequested

And I think that is probably quite common, as a lot of us will sometimes rebuild, so it all gets added at one time.

 

Luke, maybe an option to toggle between how it is now and download on demand? With how it is now being the default?

Edited by Doofus
Link to comment
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now
×
×
  • Create New...