Backup / Restore Actors - Beta with local or web based restore.

[tr0nllam 34]

1 hour ago, mickle026 said:

@tr0nllam

Are you wanting the person Image collector to behave like this? Be able to do one or the other or both?

 

Before the scrapers were disabled and then re-added back in, if the Get Actor Images option was turned off, it wouldn't search the custom scrapers when you looked for an actor image. So yes, it would be nice to have the option to turn that on/off without having to turn off the whole plugin.

I'd also like to be able to turn off the fetch metadata option separately because it slows down the loading of actor pages for anyone who doesn't have metadata.

[mickle026 401]

14 minutes ago, tr0nllam said:

Before the scrapers were disabled and then re-added back in, if the Get Actor Images option was turned off, it wouldn't search the custom scrapers when you looked for an actor image. So yes, it would be nice to have the option to turn that on/off without having to turn off the whole plugin.

I'd also like to be able to turn off the fetch metadata option separately because it slows down the loading of actor pages for anyone who doesn't have metadata.

I have turned off metadata fetchers here and the pages with no metadata still load slowly.

Investigating this, shows its not the fetching metadata (or trying to) that slows it down.  Servers are still being queried for an image by Emby even if there is one because the metadata is missing.

Ok so now we have a problem that we have to find the best solution to.

Emby uses 1 function trigger to find images and 1 function trigger to get metadata, both are triggered if anything is missing. (would be better if only 1 was triggered but Emby is made that way because traditionally it got all of the items from one place)

So the next logic would be to ignore fetchers/scrapers if the person has an image, but testing that it prevents image searches (clicking edit images) unless you delete the image.

so;

has metadata and image = ignore image collector  (this is good, and happens anyway but implimenting this in the plugin code prevents searches)

has image but no metadata  = ignore image collector  (works and pages load lots faster - however, this prevents searches)

no image but has metadata  = use image collector  (this is ok)

 

Any thoughs what I could do here? - some sort of toggle on/off for this scenerio ....maybe ...

So what I have tested is just that

 

Let me know if you can think of a better solution ?

CustomProvider.zip

[tr0nllam 34]

1 hour ago, mickle026 said:

I have turned off metadata fetchers here and the pages with no metadata still load slowly.

Investigating this, shows its not the fetching metadata (or trying to) that slows it down.  Servers are still being queried for an image by Emby even if there is one because the metadata is missing.

Ok so now we have a problem that we have to find the best solution to.

Emby uses 1 function trigger to find images and 1 function trigger to get metadata, both are triggered if anything is missing. (would be better if only 1 was triggered but Emby is made that way because traditionally it got all of the items from one place)

So the next logic would be to ignore fetchers/scrapers if the person has an image, but testing that it prevents image searches (clicking edit images) unless you delete the image.

so;

has metadata and image = ignore image collector  (this is good, and happens anyway but implimenting this in the plugin code prevents searches)

has image but no metadata  = ignore image collector  (works and pages load lots faster - however, this prevents searches)

no image but has metadata  = use image collector  (this is ok)

 

Any thoughs what I could do here? - some sort of toggle on/off for this scenerio ....maybe ...

So what I have tested is just that

 

Let me know if you can think of a better solution ?

CustomProvider.zip 3.52 MB · 1 download

What's the difference between how this works and the old custom person provider plugin? With that plugin, there was no delay related to missing metadata. The only slowdown from that plugin was when the person was missing an image, which makes sense. 

Edited April 2, 2022 by tr0nllam

[mickle026 401]

The only difference I know of if is I never had the Metadata interfaces in my code.

I instantly return nothing, ie no code is run, they are just there waiting to be used.  Perhaps the Emby core might have a wait/timeout for these?

I'll look if I can send a task cancellation from within them.  That might kill a timeout if there is one.

[TallBoiDez 8]

@mickle026does the plug-in restore and actor fanart from the .actor folder or just the actor primary image?

[mickle026 401]

29 minutes ago, TallBoiDez said:

@mickle026does the plug-in restore and actor fanart from the .actor folder or just the actor primary image?

Please elaborate your question, because Im not 100% sure what you mean..

Currently:
Restore Actor Images (Restore Tab) only restores from a backup. Primary,backdrop,logo,Banner (only at the current time)
Use Actor Images from .actors (Repair tab) only restores the actor primary image.

So is your fanart for your actors in the hidden ".actors" folder? - if so I have not got a routine for that, but if you let me know how it is saved there and its naming convetion I can certainly add it in a future update.

 

[tr0nllam 34]

I've noticed that the "use .actor images (Missing only)" is no longer adding actor images to individual episodes. Movies and the TV Show Series Cast works fine, but once it gets to checking individual episodes, it throws an error. I tried a test server with only 1 TV Show and it worked fine, but on my main database, it's not working.

This is the error I get when it gets to that portion of the restore:

2022-04-06 12:16:32.572 Error Server: Error processing request
	*** Error Report ***
	Version: 4.7.0.33
	Command line: C:\Emby Test Server\system\EmbyServer.dll
	Operating system: Microsoft Windows 10.0.19044
	Framework: .NET 6.0.2
	OS/Process: x64/x64
	Runtime: C:/Emby Test Server/system/System.Private.CoreLib.dll
	Processor count: 4
	Data path: C:\Emby Test Server\programdata
	Application path: C:\Emby Test Server\system
	System.NullReferenceException: System.NullReferenceException: Object reference not set to an instance of an object.
	   at CustomProvider.CustomProvider.PathAddBackslash(String path)
	   at CustomProvider.CustomProvider.Get(UsaActorsfOLDERMissingPeopleImages result)
	   at Emby.Server.Implementations.Services.ServiceController.Execute(HttpListenerHost appHost, Object requestDto, IRequest req)
	   at Emby.Server.Implementations.Services.ServiceHandler.ProcessRequestAsync(HttpListenerHost appHost, IRequest httpReq, IResponse httpRes, RestPath restPath, String responseContentType, CancellationToken cancellationToken)
	   at Emby.Server.Implementations.HttpServer.HttpListenerHost.RequestHandler(IRequest httpReq, ReadOnlyMemory`1 urlString, ReadOnlyMemory`1 localPath, CancellationToken cancellationToken)
	Source: CustomProviderbeta
	TargetSite: System.String PathAddBackslash(System.String)

This is with the latest Emby beta and latest plugin beta, but I'm not sure when it started.

Edited April 6, 2022 by tr0nllam

[mickle026 401]

43 minutes ago, tr0nllam said:

I've noticed that the "use .actor images (Missing only)" is no longer adding actor images to individual episodes. Movies and the TV Show Series Cast works fine, but once it gets to checking individual episodes, it throws an error. I tried a test server with only 1 TV Show and it worked fine, but on my main database, it's not working.

This is the error I get when it gets to that portion of the restore:

2022-04-06 12:16:32.572 Error Server: Error processing request
	*** Error Report ***
	Version: 4.7.0.33
	Command line: C:\Emby Test Server\system\EmbyServer.dll
	Operating system: Microsoft Windows 10.0.19044
	Framework: .NET 6.0.2
	OS/Process: x64/x64
	Runtime: C:/Emby Test Server/system/System.Private.CoreLib.dll
	Processor count: 4
	Data path: C:\Emby Test Server\programdata
	Application path: C:\Emby Test Server\system
	System.NullReferenceException: System.NullReferenceException: Object reference not set to an instance of an object.
	   at CustomProvider.CustomProvider.PathAddBackslash(String path)
	   at CustomProvider.CustomProvider.Get(UsaActorsfOLDERMissingPeopleImages result)
	   at Emby.Server.Implementations.Services.ServiceController.Execute(HttpListenerHost appHost, Object requestDto, IRequest req)
	   at Emby.Server.Implementations.Services.ServiceHandler.ProcessRequestAsync(HttpListenerHost appHost, IRequest httpReq, IResponse httpRes, RestPath restPath, String responseContentType, CancellationToken cancellationToken)
	   at Emby.Server.Implementations.HttpServer.HttpListenerHost.RequestHandler(IRequest httpReq, ReadOnlyMemory`1 urlString, ReadOnlyMemory`1 localPath, CancellationToken cancellationToken)
	Source: CustomProviderbeta
	TargetSite: System.String PathAddBackslash(System.String)

This is with the latest Emby beta and latest plugin beta, but I'm not sure when it started.

Do you have the  "CustomProvider-MEDIAFOLDER.actors(MISSING)_Repair_Report" that includes this time?

2022-04-06 12:16:32.572

 

Can you pm it to me?

There seems to be a file path error in something and I want to see what it is and see why its not being handled in the code.  The error seems to suggest it might be an url for some reason, but it might not be..

 

Thanks

 

 

 

[mickle026 401]

New Build with Error Trap and continue for "use .actor images" - both methods, when an error on an individual is caused for some reason.

Will also throw a pile of information out to the CustomProvider-MEDIAFOLDER.actors(....)_Repair_Report so that we can try and trace what caused it.

 

It should continue and finish even if anything is wrong and trips it up.

 

CustomProvider.zip

Edited April 6, 2022 by mickle026

[tr0nllam 34]

57 minutes ago, mickle026 said:

New Build with Error Trap and continue for "use .actor images" - both methods, when an error on an individual is caused for some reason.

Will also throw a pile of information out to the CustomProvider-MEDIAFOLDER.actors(....)_Repair_Report so that we can try and trace what caused it.

 

It should continue and finish even if anything is wrong and trips it up.

 

CustomProvider.zip 3.82 MB · 1 download

That fixed it. I don't even see an error that caused the initial problem.

Thanks!

[mickle026 401]

10 hours ago, tr0nllam said:

That fixed it. I don't even see an error that caused the initial problem.

Thanks!

Glad its fixed it, in the next build it will throw errors into an error log so they are easy to find.

 

Currently if an error is thrown search for this

Quote

AN ERROR OCCURED AT PERSON:

or

Quote

DEBUG LOGIC :

and there will be some info that will help me find out what threw the error

[TallBoiDez 8]

@mickle026 the scheduled task for the back is giving an error on my server 

 

Object reference not set to an instance of an object.
at CustomProvider.CustomProvider.DatabasePath(String DBType)
at CustomProvider.CustomProviderBackupPeople.BackupPeopleScheduledTask(IProgress`1 progress, CancellationToken cancellationToken)
at CustomProvider.ScheduledTasks.ScheduledTaskBackupPeople.Execute(CancellationToken cancellationToken, IProgress`1 progress)
at Emby.Server.Implementations.ScheduledTasks.ScheduledTaskWorker.ExecuteInternal(TaskOptions options)

[Spegeli 18]

15 hours ago, TallBoiDez said:

@mickle026 the scheduled task for the back is giving an error on my server 

 

Object reference not set to an instance of an object.
at CustomProvider.CustomProvider.DatabasePath(String DBType)
at CustomProvider.CustomProviderBackupPeople.BackupPeopleScheduledTask(IProgress`1 progress, CancellationToken cancellationToken)
at CustomProvider.ScheduledTasks.ScheduledTaskBackupPeople.Execute(CancellationToken cancellationToken, IProgress`1 progress)
at Emby.Server.Implementations.ScheduledTasks.ScheduledTaskWorker.ExecuteInternal(TaskOptions options)

Can confirm, got this error, too.

[mickle026 401]

New build

 

Set task triggers to every 30 days (720 hrs), I dont know why but you cannot do this inside embys options to add a trigger.

You can of course delete and add your own, but before you delete it add another and see if you can do what you need.

CustomProvider.zip

[tr0nllam 34]

I had to start a fresh database yesterday and I'm once again experiencing the problem where missing episode .actor images aren't being added to the database. Same behavior as before where the log shows TV Series actors being checked, but it completely ignores the episode actors in my main TV Shows library. I have a couple other small TV Show libraries and those do get checked, but not the main one.

The CustomProvider log is not showing any errors, but the embyserver log has a similar error as before: 

2022-04-06 12:16:32.572 Error Server: Error processing request
    *** Error Report ***
    Version: 4.7.0.33
    Command line: C:\Emby Test Server\system\EmbyServer.dll
    Operating system: Microsoft Windows 10.0.19044
    Framework: .NET 6.0.2
    OS/Process: x64/x64
    Runtime: C:/Emby Test Server/system/System.Private.CoreLib.dll
    Processor count: 4
    Data path: C:\Emby Test Server\programdata
    Application path: C:\Emby Test Server\system
    System.NullReferenceException: System.NullReferenceException: Object reference not set to an instance of an object.
       at CustomProvider.CustomProvider.PathAddBackslash(String path)
       at CustomProvider.CustomProvider.Get(UsaActorsfOLDERMissingPeopleImages result)
       at Emby.Server.Implementations.Services.ServiceController.Execute(HttpListenerHost appHost, Object requestDto, IRequest req)
       at Emby.Server.Implementations.Services.ServiceHandler.ProcessRequestAsync(HttpListenerHost appHost, IRequest httpReq, IResponse httpRes, RestPath restPath, String responseContentType, CancellationToken cancellationToken)
       at Emby.Server.Implementations.HttpServer.HttpListenerHost.RequestHandler(IRequest httpReq, ReadOnlyMemory`1 urlString, ReadOnlyMemory`1 localPath, CancellationToken cancellationToken)
    Source: CustomProviderbeta
    TargetSite: System.String PathAddBackslash(System.String)

 

Edited April 17, 2022 by tr0nllam

[mickle026 401]

2 hours ago, tr0nllam said:

I had to start a fresh database yesterday and I'm once again experiencing the problem where missing episode .actor images aren't being added to the database. Same behavior as before where the log shows TV Series actors being checked, but it completely ignores the episode actors in my main TV Shows library. I have a couple other small TV Show libraries and those do get checked, but not the main one.

The CustomProvider log is not showing any errors, but the embyserver log has a similar error as before: 

2022-04-06 12:16:32.572 Error Server: Error processing request
    *** Error Report ***
    Version: 4.7.0.33
    Command line: C:\Emby Test Server\system\EmbyServer.dll
    Operating system: Microsoft Windows 10.0.19044
    Framework: .NET 6.0.2
    OS/Process: x64/x64
    Runtime: C:/Emby Test Server/system/System.Private.CoreLib.dll
    Processor count: 4
    Data path: C:\Emby Test Server\programdata
    Application path: C:\Emby Test Server\system
    System.NullReferenceException: System.NullReferenceException: Object reference not set to an instance of an object.
       at CustomProvider.CustomProvider.PathAddBackslash(String path)
       at CustomProvider.CustomProvider.Get(UsaActorsfOLDERMissingPeopleImages result)
       at Emby.Server.Implementations.Services.ServiceController.Execute(HttpListenerHost appHost, Object requestDto, IRequest req)
       at Emby.Server.Implementations.Services.ServiceHandler.ProcessRequestAsync(HttpListenerHost appHost, IRequest httpReq, IResponse httpRes, RestPath restPath, String responseContentType, CancellationToken cancellationToken)
       at Emby.Server.Implementations.HttpServer.HttpListenerHost.RequestHandler(IRequest httpReq, ReadOnlyMemory`1 urlString, ReadOnlyMemory`1 localPath, CancellationToken cancellationToken)
    Source: CustomProviderbeta
    TargetSite: System.String PathAddBackslash(System.String)

 

See if this build Throws an error log, this is the type of logic it should throw - again it should continue with the next film/series even with the error thrown (I did only say should!)

/log?name=CustomProvider-MEDIAFOLDER.actors(ERROR)_Repair_Report_17.Apr.2022-08.42.54.txt

For me im getting a directory path not found thrown

17.Apr.2022  -  08.44.43 == ERROR ================================================
17.Apr.2022  -  08.44.43 AN ERROR OCCURED AT MovieID:Iron Man
17.Apr.2022  -  08.44.43 DEBUG LOGIC :-
17.Apr.2022  -  08.44.43 MovieName/SeriesName:Iron Man
17.Apr.2022  -  08.44.43 MoviePath/SeriesPath:C:\Users\xxx\Drives\Marvel\Iron Man (2008)
17.Apr.2022  -  08.44.43 MoviePath2/SeriesPath2:C:\Users\xxx\Drives\Marvel\Iron Man (2008)
17.Apr.2022  -  08.44.43 KodiActorsPath:C:\Users\xxx\Drives\Marvel\Iron Man (2008)\.actors
17.Apr.2022  -  08.44.43 KodiActorsPath2:C:\Users\xxx\Drives\Marvel\Iron Man (2008)\.actors
17.Apr.2022  -  08.44.43 ActorsInMovieFolder:C:\Users\xxx\Drives\Marvel\Iron Man (2008)\
17.Apr.2022  -  08.44.43 ActorsInMovieFolder2:C:\Users\xxx\Drives\Marvel\Iron Man (2008)\
17.Apr.2022  -  08.44.43 ERROR THROWN BY SERVER :-
17.Apr.2022  -  08.44.43 System.IO.DirectoryNotFoundException: Could not find a part of the path 'C:\Users\xxx\Drives\Marvel\Iron Man (2008)'.
   at System.IO.Enumeration.FileSystemEnumerator`1.CreateDirectoryHandle(String path, Boolean ignoreNotFound)
   at System.IO.Enumeration.FileSystemEnumerator`1.Init()
   at System.IO.Enumeration.FileSystemEnumerableFactory.UserDirectories(String directory, String expression, EnumerationOptions options)
   at System.IO.Directory.InternalEnumeratePaths(String path, String searchPattern, SearchTarget searchTarget, EnumerationOptions options)
   at System.IO.Directory.GetDirectories(String path)
   at CustomProvider.CustomProviderRepairPeopleImages.Repair_Missing_People_from_actors_folder(IProgress`1 progress, CancellationToken cancellationToken)

and i am also getting a denied access error

17.Apr.2022  -  08.52.32 == ERROR ================================================
17.Apr.2022  -  08.52.32 AN ERROR OCCURED AT PERSON: Theodore Gottlieb
17.Apr.2022  -  08.52.32 DEBUG LOGIC :-
17.Apr.2022  -  08.52.32 ActorNameWithUnderscore: Theodore_Gottlieb.jpg
17.Apr.2022  -  08.52.32 ActorID: Theodore Gottlieb
17.Apr.2022  -  08.52.32 ImagePath: 
17.Apr.2022  -  08.52.32 KodiImagePath: \\xxx.xxx.xxx.xxx\Movies\T\The 'Burbs (1989)\.actors\Theodore_Gottlieb.jpg
17.Apr.2022  -  08.52.32 KodiImagePathWOUnderscore: \\xxx.xxx.xxx.xxx\Movies\T\The 'Burbs (1989)\.actors\Theodore Gottlieb.jpg
17.Apr.2022  -  08.52.32 FolderImagesPath: \\xxx.xxx.xxx.xxx\Movies\T\The 'Burbs (1989)\Theodore_Gottlieb.jpg
17.Apr.2022  -  08.52.32 FolderImagePathWOUnderscore: \\xxx.xxx.xxx.xxx\Movies\T\The 'Burbs (1989)\Theodore Gottlieb.jpg
17.Apr.2022  -  08.52.32 EpisodePath: \\xxx.xxx.xxx.xxx\Movies\T\
17.Apr.2022  -  08.52.32 EpisodePathWOUnderscore: \\xxx.xxx.xxx.xxx\Movies\T\
17.Apr.2022  -  08.52.32 ERROR THROWN BY SERVER :-
17.Apr.2022  -  08.52.32 System.UnauthorizedAccessException: Access to the path '\\xxx.xxx.xxx.xxx\Movies\T\The 'Burbs (1989)\.actors\Theodore_Gottlieb.jpg' is denied.
   at System.IO.FileSystem.CopyFile(String sourceFullPath, String destFullPath, Boolean overwrite)
   at CustomProvider.CustomProviderRepairPeopleImages.Repair_Missing_People_from_actors_folder(IProgress`1 progress, CancellationToken cancellationToken)

CustomProvider.zip

Edited April 17, 2022 by mickle026

[tr0nllam 34]

Yep, this new version is correctly adding episode images again. I'll report back if any errors pop up.

Thanks again.

[mickle026 401]

6 hours ago, tr0nllam said:

Yep, this new version is correctly adding episode images again. I'll report back if any errors pop up.

Thanks again.

Did it not throw an error log? - im more worried that its continuing from an error and not telling you where it is.  Remind me, are you a linux or windows user?

However I think I see what might cause that on Linux and will fix that for the next build too.

Edited April 17, 2022 by mickle026

[tr0nllam 34]

1 hour ago, mickle026 said:

Did it not throw an error log? - im more worried that its continuing from an error and not telling you where it is.  Remind me, are you a linux or windows user?

However I think I see what might cause that on Linux and will fix that for the next build too.

I'm a Windows user.

There were no errors in the CustomProvider log or the embyserver log. Since I had started a fresh database, there were a bunch of images to add and it added them all without any issues. No idea what was causing the previous error.

[mickle026 401]

Latest build.  Zip contains build for stable server (CustomProviderStable.dll), and beta server (CustomProviderbeta.dll).

 

Adds Localise Images from http to local cache

CustomProvider.zip

Edited April 21, 2022 by mickle026

[TallBoiDez 8]

12 hours ago, mickle026 said:

Latest build.  Zip contains build for stable server (CustomProviderStable.dll), and beta server (CustomProviderbeta.dll).

 

Adds Localise Images from http to local cache

CustomProvider.zip 3.84 MB · 5 downloads

Can you explain exactly what this new features does

[mickle026 401]

32 minutes ago, TallBoiDez said:

Can you explain exactly what this new features does

It downloads all remote images to the cache for the chosen items.

When you add a new item and a plugin is used to fetch the metadata and images its image path is likely to be a http link.

This utility will make that remote image local, ie stored on you ssd, nvme or hdd.  This is a user preference, its not a necessity.

The example below shows actor Terrence Terrell, in my library his image was always loaded from online, after running the tool his image is now always from my storage drive.

example:

https://image.tmdb.org/t/p/original/to5u2Mumj1zp9J9JSV1QCzI3XuN.jpg

21.Apr.2022 - 18.21.06 --- (154002) of (161996) --- Terrence Terrell 
21.Apr.2022 - 18.21.06 Library Type: Person
21.Apr.2022 - 18.21.06 Image Type: Primary
21.Apr.2022 - 18.21.06 Server Image: https://image.tmdb.org/t/p/original/to5u2Mumj1zp9J9JSV1QCzI3XuN.jpg
21.Apr.2022 - 18.21.06 Localised Image: D:\metadata\people\Terrence Terrell-tmdb-2601103\folder.jpg

 

[TallBoiDez 8]

5 minutes ago, mickle026 said:

It downloads all remote images to the cache for the chosen items.

When you add a new item and a plugin is used to fetch the metadata and images its image path is likely to be a http link.

This utility will make that remote image local, ie stored on you ssd, nvme or hdd.  This is a user preference, its not a necessity.

The example below shows actor Terrence Terrell, in my library his image was always loaded from online, after running the tool his image is now always from my storage drive.

example:

https://image.tmdb.org/t/p/original/to5u2Mumj1zp9J9JSV1QCzI3XuN.jpg

21.Apr.2022 - 18.21.06 --- (154002) of (161996) --- Terrence Terrell 
21.Apr.2022 - 18.21.06 Library Type: Person
21.Apr.2022 - 18.21.06 Image Type: Primary
21.Apr.2022 - 18.21.06 Server Image: https://image.tmdb.org/t/p/original/to5u2Mumj1zp9J9JSV1QCzI3XuN.jpg
21.Apr.2022 - 18.21.06 Localised Image: D:\metadata\people\Terrence Terrell-tmdb-2601103\folder.jpg

 

So, for example in my metadata for movies I have an url link to the image for that actor, will it pull the image from that URL or not at all.

[mickle026 401]

As in the example above it pulled the image from tmdb URL to my ssd metadata folder (not media folder)

Edited April 21, 2022 by mickle026

[TallBoiDez 8]

3 hours ago, mickle026 said:

As in the example above it pulled the image from tmdb URL to my ssd metadata folder (not media folder)

ah okay.