Backup / Restore Actors - Beta with local or web based restore.

[mickle026 401]

To Clarify My plugin Does NOT download anything from IMDB or TMDB during the repair process of actors.

What it does:

It removes the blocks that prevent Emby from working properly by,

1. Removing corrupt files that are linked in the Emby database and prevent the actor image loading or updating.
2. Removes invalid links in the database that point to files on servers that do not exist.

Therefore resetting the actor card to its state before anything is downloaded for that actor.

As you have found out, downloading from TMDB requires either a full metarefresh>replacing images to be run, or clicking the actor card/placeholder.  This is done by Emby itself and its other plugins such as TMDB or TVDB.

if you have put http://media-and-meta.com in the server config tab (which it seems you did), then the plugin looks on my own private webspace as tries to find the person image.  In order to find it on there I have to have uploaded it, so therefore the person would have come from my actor backup.  There are no language variations here, just the images that I have.
I do not have all the images that TMDB has, however I also do have quite a few that TMDB doesn't have.  So both plugins are the best of both worlds.

Quote

19.Mar.2022  -  12.32.42 PersonName: Guo Jingfei
19.Mar.2022  -  12.32.42 TVDB:  IMDB: nm3530937 TMDB: 1353688
19.Mar.2022  -  12.32.43 CustomProvider Server (IMDB) Primary Doesn't Exist for Guo Jingfei
19.Mar.2022  -  12.32.43 CustomProvider Server (TMDB) Primary Doesn't Exist for Guo Jingfei
19.Mar.2022  -  12.32.44 CustomProvider Server (PersonByName) Primary Doesn't Exist for Guo Jingfei
19.Mar.2022  -  12.32.44 Search Completed.

This is searching one of the servers in the global tab, not imdb or tmdb.  The references here are to ids for imdb or tmbd on my backup server.  The plugin has looked on my personal server for 3 seperate references.  Imdb, tmdb and the actor by name.  This person is not on my server.

If you have external images with your media in a hidden actors folder ".actors" (usually created by either kodi or external media management software), the plugin can also import those on a button click.

Quote

"Total: 100461 Missing Image(s)" and then i did a Force Actor Images, after that it was "Total: 99685 Missing Image(s)".

At least 776 Images were automatically replaced, as well as the "other" blanks repaired so that they will work again.

So thank you for your idea, but thats not for this plugin, that is what Lukes plugin is doing.

Edited March 19, 2022 by mickle026

[warrentc3 40]

Love what all you're trying to accomplish with this plugin!

 

With doing a movie backup, noticed there's some issues with non-alphanumeric characters.  

"FileSafe Name: Hellboy II: The Golden Army"

"/opt/media/emby_backup/MovieDB/MovieByName/Hellboy II: The Golden Army/Backdrop0.nfo"

 

24.Mar.2022  -  00.24.03 Processing 590 of xxx, Name: Hellboy II: The Golden Army
24.Mar.2022  -  00.24.03 PrimaryImage /opt/media/movies/films/Hellboy II - The Golden Army (2008)/Hellboy II - The Golden Army (2008)-poster.jpg, FileSafe Name: Hellboy II: The Golden Army
24.Mar.2022  -  00.24.03 MovieName: Hellboy II: The Golden Army

24.Mar.2022  -  00.24.03 MovieBackdropImage: /opt/media/movies/films/Hellboy II - The Golden Army (2008)/Hellboy II - The Golden Army (2008)-fanart.jpg
24.Mar.2022  -  00.24.03 FILECOPY ERROR ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! !
24.Mar.2022  -  00.24.03 Error Copying File From: /opt/media/movies/films/Hellboy II - The Golden Army (2008)/Hellboy II - The Golden Army (2008)-fanart.jpg
24.Mar.2022  -  00.24.03 Error Copying File To: /opt/media/emby_backup/MovieDB/Backdrop/da47e10b57e956e9dc027ff77ec1f35bdd0251a7f2497468fc17cc347ffaeb66.jpg
24.Mar.2022  -  00.24.03 Backdrop Image Copied To: /opt/media/emby_backup/MovieDB/Backdrop/da47e10b57e956e9dc027ff77ec1f35bdd0251a7f2497468fc17cc347ffaeb66.jpg
24.Mar.2022  -  00.24.03 Backdrop NFO BackupPath: /opt/media/emby_backup/MovieDB/IMDB/tt0411477/Backdrop0.nfo
24.Mar.2022  -  00.24.03 Backdrop NFO BackupPath: /opt/media/emby_backup/MovieDB/TMDB/11253/Backdrop0.nfo
24.Mar.2022  -  00.24.03 Backdrop NFO BackupPath: /opt/media/emby_backup/MovieDB/MovieByName/Hellboy II: The Golden Army/Backdrop0.nfo

 

[warrentc3 40]

Also, am curious about when 2 movies have the same name since one of the movie backup folders is "MovieByName" and there's no year designation.

For example:

Scream (1996)

Scream (2022)

[mickle026 401]

4 hours ago, warrentc3 said:

Love what all you're trying to accomplish with this plugin!

 

With doing a movie backup, noticed there's some issues with non-alphanumeric characters.  

"FileSafe Name: Hellboy II: The Golden Army"

"/opt/media/emby_backup/MovieDB/MovieByName/Hellboy II: The Golden Army/Backdrop0.nfo"

 

24.Mar.2022  -  00.24.03 Processing 590 of xxx, Name: Hellboy II: The Golden Army
24.Mar.2022  -  00.24.03 PrimaryImage /opt/media/movies/films/Hellboy II - The Golden Army (2008)/Hellboy II - The Golden Army (2008)-poster.jpg, FileSafe Name: Hellboy II: The Golden Army
24.Mar.2022  -  00.24.03 MovieName: Hellboy II: The Golden Army

24.Mar.2022  -  00.24.03 MovieBackdropImage: /opt/media/movies/films/Hellboy II - The Golden Army (2008)/Hellboy II - The Golden Army (2008)-fanart.jpg
24.Mar.2022  -  00.24.03 FILECOPY ERROR ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! !
24.Mar.2022  -  00.24.03 Error Copying File From: /opt/media/movies/films/Hellboy II - The Golden Army (2008)/Hellboy II - The Golden Army (2008)-fanart.jpg
24.Mar.2022  -  00.24.03 Error Copying File To: /opt/media/emby_backup/MovieDB/Backdrop/da47e10b57e956e9dc027ff77ec1f35bdd0251a7f2497468fc17cc347ffaeb66.jpg
24.Mar.2022  -  00.24.03 Backdrop Image Copied To: /opt/media/emby_backup/MovieDB/Backdrop/da47e10b57e956e9dc027ff77ec1f35bdd0251a7f2497468fc17cc347ffaeb66.jpg
24.Mar.2022  -  00.24.03 Backdrop NFO BackupPath: /opt/media/emby_backup/MovieDB/IMDB/tt0411477/Backdrop0.nfo
24.Mar.2022  -  00.24.03 Backdrop NFO BackupPath: /opt/media/emby_backup/MovieDB/TMDB/11253/Backdrop0.nfo
24.Mar.2022  -  00.24.03 Backdrop NFO BackupPath: /opt/media/emby_backup/MovieDB/MovieByName/Hellboy II: The Golden Army/Backdrop0.nfo

 

Thank you for spotting this.

I am already implimenting changes for duplicate names in the music albums section, but that is easier because I have things I can use to make it unique such as trackcount, artist counts etc. (this is in the version I havent released yet)

That in itself has challenges, whilst its fine for backup and restore, its not ideal for someone else looking up information from your backup.  The other person might not have the "year" ...

I know there are only a few movies that still have the same name and year (both together),  so finding something unique for this that allows cross over of users is more difficult.  ie to look up on my backup a movie that you have in you collection - especially with no other information such as imdb and tmdb id's.  I can add the year for now thoough as its better than not having it.

The movie by name is the 3rd lookup option after imdb and tmdb provider ids, so its mainly a fallback, but if you do not have those provider ids, it becomes  the only lookup option.

Edited March 24, 2022 by mickle026

[tr0nllam 34]

Is the plugin still working for anyone in the latest Emby beta?

I've tried restoring .actor images and generating reports, but I'm not seeing any evidence that the plugin is working.

Edited March 25, 2022 by tr0nllam

[mickle026 401]

On 25/03/2022 at 03:57, tr0nllam said:

Is the plugin still working for anyone in the latest Emby beta?

I've tried restoring .actor images and generating reports, but I'm not seeing any evidence that the plugin is working.

I'll take a look at restore actors, but reports are working fine here (reports tab)

Which plugin version are you using? for the Stable server  or for the Beta server?

 

Here is the latest rework.

When using repair people it now has the option to force the person to Full refresh, so if you have providers such as tvdb, tmdb etc when it runs it will try to grab for repaired or missing people during the run, without the need to click every actor card or refresh everything.  This is not tested yet, but should work (famous last words), if it does, then images grabbed by tmdb, tvdb etc will not report or appear in the logs (except embyserver) as they are not done by this plugin but triggered to be done by the others.

 

I have started to add other code, first of all a new menu.  Removed the Tab Manager.

 

Still some improvements and reorganising. 

Removed the TabManager as it wasn't suiting my needs anymore.

Added a few Image tools, there will be more...

There are a few other fixes , 
A real change Album Backup to make it more ideal from the plugin perspective, helps make them unique (AlbumByName) as the are quite a few albums with the same names.  A whole new album backup is required after removing the exisiting backup for Albums.

A lot of work has been done in the Audio department, Albums, Covers, Track Images etc.  Artists still have more to do. 
You may notice an audio fingerprint button.  It wont work externally, its for an internal identify function, its visible while i tested it, and i havent removed the click button yet.  Its also not an acousticid fiingerprint either, its my own function so is unique to this plugin, and will enable a lot more presice lookups and identifies on the backups..as it seems to work very well.

I will look at restore functions as others need writing as well.  It will all come eventually

 

 

 

CustomProvider.zip

Edited March 28, 2022 by mickle026

[GrimReaper 3301]

1 minute ago, mickle026 said:

This is not tested yet, but should work (famous last words)

[mickle026 401]

7 minutes ago, GrimReaper said:

I no longer have any corrupt images to trigger it - lol

[GrimReaper 3301]

3 minutes ago, mickle026 said:

I no longer have any corrupt images to trigger it - lol

Me neither, courtesy of your oooold plugin.

[tr0nllam 34]

Not sure what was going on, but the new plugin is working when restoring actor images from .actors.

Edited March 28, 2022 by tr0nllam

[mickle026 401]

10 minutes ago, tr0nllam said:

Not sure what was going on, but the new plugin is working when restoring actor images from .actors.

No idea too??

Not touched that part of the code side we decided it was working a while ago.

[tr0nllam 34]

1 minute ago, mickle026 said:

No idea too??

Not touched that part of the code side we decided it was working a while ago.

Another issue I have seen is that sometimes when I run it, it randomly stops, and I have to run it again. The second time I run it, it seems to work fine, based on the logs.

[mickle026 401]

When it randomly stops is there an entry in embyserver.txt file at the time?

You should be able to find easily with the last time in the customprovider log and going to the time in the embyserver log.

[tr0nllam 34]

3 minutes ago, mickle026 said:

When it randomly stops is there an entry in embyserver.txt file at the time?

You should be able to find easily with the last time in the customprovider log and going to the time in the embyserver log.

 

This is the matching entry in the log:

 

Quote

2022-03-28 16:19:52.862 Error Server: Error processing request
    *** Error Report ***
    Version: 4.7.0.32
    Command line: C:\Users\Anon\AppData\Roaming\Emby-Server\system\EmbyServer.dll -noautorunwebapp
    Operating system: Microsoft Windows 10.0.19044
    Framework: .NET 6.0.2
    OS/Process: x64/x64
    Runtime: C:/Users/Anon/AppData/Roaming/Emby-Server/system/System.Private.CoreLib.dll
    Processor count: 4
    Data path: C:\Users\Anon\AppData\Roaming\Emby-Server\programdata
    Application path: C:\Users\Anon\AppData\Roaming\Emby-Server\system
    System.IO.IOException: System.IO.IOException: The process cannot access the file 'C:\Users\Anon\AppData\Roaming\Emby-Server\programdata\logs\CustomProvider-MEDIAFOLDER.actors(MISSING)_Repair_Report_28.Mar.2022-16.19.12.txt' because it is being used by another process.
       at Microsoft.Win32.SafeHandles.SafeFileHandle.CreateFile(String fullPath, FileMode mode, FileAccess access, FileShare share, FileOptions options)
       at Microsoft.Win32.SafeHandles.SafeFileHandle.Open(String fullPath, FileMode mode, FileAccess access, FileShare share, FileOptions options, Int64 preallocationSize)
       at System.IO.Strategies.OSFileStreamStrategy..ctor(String path, FileMode mode, FileAccess access, FileShare share, FileOptions options, Int64 preallocationSize)
       at System.IO.File.AppendAllText(String path, String contents)
       at CustomProvider.CustomProvider.LogToMyFile(String FileName, String Text)
       at CustomProvider.CustomProvider.Get(UsaActorsfOLDERMissingPeopleImages result)
       at Emby.Server.Implementations.Services.ServiceController.Execute(HttpListenerHost appHost, Object requestDto, IRequest req)
       at Emby.Server.Implementations.Services.ServiceHandler.ProcessRequestAsync(HttpListenerHost appHost, IRequest httpReq, IResponse httpRes, RestPath restPath, String responseContentType, CancellationToken cancellationToken)
       at Emby.Server.Implementations.HttpServer.HttpListenerHost.RequestHandler(IRequest httpReq, ReadOnlyMemory`1 urlString, ReadOnlyMemory`1 localPath, CancellationToken cancellationToken)
    Source: System.Private.CoreLib
    TargetSite: Microsoft.Win32.SafeHandles.SafeFileHandle CreateFile(System.String, System.IO.FileMode, System.IO.FileAccess, System.IO.FileShare, System.IO.FileOptions)

 

So, maybe it's not stopping, it just can't access the log when I open it in notepad and thus stops writing to it.

Although, I have had the log open before and it's kept updating.

Edited March 28, 2022 by tr0nllam

[mickle026 401]

13 minutes ago, tr0nllam said:

 

This is the matching entry in the log:

 

 

So, maybe it's not stopping, it just can't access the log when I open it in notepad and thus stops writing to it.

Although, I have had the log open before and it's kept updating.

One of the fixes

13.03.2022
*Crashes caused by logfile in use fixed

Edited March 28, 2022 by mickle026

[tr0nllam 34]

4 minutes ago, mickle026 said:

One of the fixes

13.03.2022
*Crashes caused by logfile in use fixed

Ah, well this is using the latest beta plugin. It was a previous issue, but it happened when I just ran it after upgrading to the new plugin.

Edited March 28, 2022 by tr0nllam

[mickle026 401]

39 minutes ago, tr0nllam said:

Ah, well this is using the latest beta plugin.

Still stopping? - oooh I'll take a looksie at that then, thanks for reporting it.

 

Here you go, it shouldn't now.

I set it to try to write, if fail wait 500ms, try again for up to 10 times, if fail abort log entry and continue so log entry just wont be written if it hasnt been able to in 5 seconds.

I am also aware that sometimes if the webscrapers that are working are on , they can also cause this..  Lets hope this fixed that too.

 

CustomProvider.zip

Edited March 28, 2022 by mickle026

[Spegeli 18]

On 3/28/2022 at 10:44 PM, mickle026 said:

Still stopping? - oooh I'll take a looksie at that then, thanks for reporting it.

 

Here you go, it shouldn't now.

I set it to try to write, if fail wait 500ms, try again for up to 10 times, if fail abort log entry and continue so log entry just wont be written if it hasnt been able to in 5 seconds.

I am also aware that sometimes if the webscrapers that are working are on , they can also cause this..  Lets hope this fixed that too.

 

CustomProvider.zip 3.47 MB · 1 download

When i use the Beta Version, its not showen Up in the Plugin list?! Do i need to clear old configs or anything else?

The Stable version works

 

"Collaborate

Enable Collaboration, allow Emby to upload images and metadata to a collaboration server.
Collaborating helps ensure everyone is able to receive images and metadata."

 

@mickle026Is this function new? Is it possible to upload Images to your Server? Because i own a very very big Movie & Series Libary with lots of Images i can upload to your server if your Interested. When yes contact me via PN.

Edited March 30, 2022 by Spegeli

[TallBoiDez 8]

@mickle026is it possible to make the backup process a schedule task?

Edited March 30, 2022 by TallBoiDez

[mickle026 401]

1 hour ago, Spegeli said:

When i use the Beta Version, its not showen Up in the Plugin list?! Do i need to clear old configs or anything else?

The Stable version works

 

"Collaborate

Enable Collaboration, allow Emby to upload images and metadata to a collaboration server.
Collaborating helps ensure everyone is able to receive images and metadata."

 

@mickle026Is this function new? Is it possible to upload Images to your Server? Because i own a very very big Movie & Series Libary with lots of Images i can upload to your server if your Interested. When yes contact me via PN.

Its not operational yet, its part of my ideas in the testing stages, but I hope to get it working at some point.

1 hour ago, TallBoiDez said:

@mickle026is it possible to make the backup process a schedule task?

scheduled tasks are on my radar ! (or roadmap).  Im just not there yet. 

I'm still converting things to log to my logs rather than fill embyserver.txt with junk (still lots of old code in the plugin). 
I'm still trying to figure out a good layout of pages.
I'm still coding for new issues I come accross, and of course still being side tracked by my own needs etc ....

Dev road map is all over the place!

Still very busy although it may seem progress is slow.

Yesterdays code makes my audio fingerprinter work well, but its not implimented in general code yet.  Also added Restore Albums to restore tab (not tested that yet).
Also figured out a bit more of skia (that might not mean anything, but its to do with cross platform image manipulation).

Today's code or sidetrack diversion    adds the ability to remove LiveTV Channel logos, so a guide refresh will correct them (incase you have wrong logos on LieTV channels) - I have just finished testing it and it works just fine!, it doesn't tamper with anything else.  I suggested this as a feature request but I don't think  its going to happen so I just coded it myself.  (Suprised myself actually how well it works - no errors at all on first test!).

Now unfortunatelly its back off to work - Feels like I live there!

[mickle026 401]

Heres the latest build

Stable = use on the stable server
Beta = for use with the latest beta server

This is in direct response to this thread: https://emby.media/community/index.php?/topic/107259-update-live-tv-channel-icons/

I dont have a clue why this would affect other things, my coded example does not touch anything other than the logo images.  Works as described.

When a lineup changes or you change make changes the tv channel logos can be showing the incorrect images, this removes them all, so that when the guide is refreshed, they re-appear correctly.

Added this to the Clean Tab

Running a guide refresh after removing all channel logos in my test replaced them all with the correct logos

 

CustomProvider.zip

Edited March 31, 2022 by mickle026

[mickle026 401]

Updated and Re-enabled some scrapers, They are not all converted yet.  See build notes too see which are working.

Note:  Library scan times will take longer depending on which you have enabled (turn them off when not in use) as the library scan will also fetch from scrapers at the moment.  I recommend disabling them until you want to use them.

1.04.2022
*Added Friendly Ember Servers Person Scraper Update - now operational
*Added Trakt Person Scraper Update - now operational
*Added RottenTomatoes Person Scraper Update - now operational
*Added CinemaClock Person Scraper Update - now operational
*Added SpicyOnion Person Scraper Update - now operational
*Added NextEpisode Person Scraper Update - now operational
*Added Listal Person Scraper Update - now operational
*Added MarvelCinema Person Scraper Update - now operational
*Added Mandy Person Scraper Update - now operational
*Added Alchetron Person Scraper Update - now operational
*Added FandomDisney Person Scraper Update - now operational
*Added FandomWarner Person Scraper Update - now operational
*Added Movietimes Person Scraper Update - now operational
*Added DCExtendedUniverse Person Scraper Update - now operational
*Added FilmiFeed Person Scraper Update - now operational
Still some to Add/Update

30.03.2022
*Added IMDB Person Scraper Update - now operational
- aware that IMDB scraper doesn't always fetch images available (currently investigating)'
*Added Mubi Person Scraper Update - now operational
! Scraper cache is not currently funtional for Persons
*Fixed Crash in multiple scrapers selected (All scrapers)

30.03.2022
*Added TV Channel Logo Cleaner

CustomProvider.zip

Edited April 1, 2022 by mickle026

[tr0nllam 34]

In new new version, if you have some of the People Web Scrapers enabled, it will search those scrapers whether you have the "Get Actor Images from the Custom Provider servers" enabled or not.

Also, it's still attempting to download metadata when loading someone without metadata despite the option being unselected.

[mickle026 401]

11 hours ago, tr0nllam said:

In new new version, if you have some of the People Web Scrapers enabled, it will search those scrapers whether you have the "Get Actor Images from the Custom Provider servers" enabled or not.

Also, it's still attempting to download metadata when loading someone without metadata despite the option being unselected.

Currently Yes.  They are all treated like individual fetchers at the moment.

Its hard to explain how it works at the moment, but it will get sorted out eventually.

If you refer to the "not implemented exceptions" thrown in the log for metadata, i'll remove that message in the next build.  That is there for me, so I know what emby is doing and when.  Those metadata fetchers are triggered by emby and not the plugin (even though they come from the plugin).  This might sound strange right now, but it will all work eventually.  Those fetchers are not actually fetching yet, but telling me emby asked to use them.

 

I have so many loose ends atr them moment, but im slowly getting there

[mickle026 401]

@tr0nllam

Are you wanting the person Image collector to behave like this? Be able to do one or the other or both?