Script: Remove Guest star from episode if already present on Series

[ginjaninja 533]

Background

I find it less than ideal when looking at a person's biography, to see them appear as guests stars on episodes as well as at the episode's series level

 

Goal

Remove duplicate guest stars and actors from episodes, whilst protecting legitimate duplication for non acting roles e.g Director.

Remove spurious noise from person's bio page. Easier to see where an actor has directed their main series or indeed other episodes in other series.

 

Design

Remove any guest stars on episodes which are actors on the series.

testmode (default on) to log proposed removals rather than make changes.

 

Installation

populate username and password and emby url in config.

Note - testmode = $false/$true in Config

may require latest powershell and relaxed powershell execution policy.

Backup your database prior to use

Use with extreme caution on live databases, best tried on a test database first

Would not use on a database with manually curated metadata [person especially] (without alot of testing)

 

Notes

tested on Windows Powershell but should be agnostic

Does not lock metadata so future library refreshes may revert changes, script could be scheduled for regular cleaning.

Does not actively respect metadata locking, may ignore, have not tested as i don't use.

I have run it on my live database but results have not undergone any long term testing.

For me, 10000 episodes with duplicates cleaned out of 40000 episodes, took about 30 minutes to run. reduced database by 50MB down to 440MB

 

Caveats

On very fast systems Powershell may open more sessions with the api than the OS network stack will support - havent found a solution to this but i was fine on my library since i rewrote script to reduce api calls. In any case the script can just be rerun and it will continue from where it finds the next duplicate.

 

issues / suggestions welcome.

 

PersonCleaner v0.0.0.3.zip

 

 

Edited January 4, 2022 by ginjaninja
update to fix utf8 characters

[Oxide 139]

Thx @ginjaninja been looking for something like this for a while 

I am getting this error when running the script, was hoping you could help out 

Method invocation failed because [Selected.System.Management.Automation.PSCustomObject] does not contain a method
named 'foreach'.
At D:\PersonCleaner v0.0.0.1\PersonCleaner.ps1:70 char:13
+             $removed.foreach{
+             ~~~~~~~~~~~~~~~~~
    + CategoryInfo          : InvalidOperation: (foreach:String) [], RuntimeException
    + FullyQualifiedErrorId : MethodNotFound

 

 

 

[ginjaninja 533]

i think this points to not having the latest powershell installed

https://github.com/PowerShell/PowerShell/releases

(go to assets section)

are you on Windows?

edit

and you need to launch with latest version of powershell..on windows this may mean launching pwsh.exe first  and calling fron there (as opposed to double clickkng) or perhaps associating .ps1 with pwsh.exe

edit

https://docs.microsoft.com/en-us/powershell/scripting/install/installing-powershell-core-on-windows?view=powershell-7

suggests you should get a new open with pwsh when right clicking.

 

 

 

 

Edited June 12, 2020 by ginjaninja

[Oxide 139]

Thanks @ginjaninja installing v7.0.2 and opening with pwsh.exe did the trick 

[ginjaninja 533]

Thanks @ginjaninja installing v7.0.2 and opening with pwsh.exe did the trick [emoji16]

interested to know how you find it. for me i schedule the script nightly to keep everything clean. not earth shattering but a meaningful improvement when browsing biographies.

there is a feature request for the server to include this functionality if youre interested.
https://emby.media/community/index.php?/topic/83577-server-prevent-people-assigned-to-series-being-assigned-to-episode/&tab=comments#comment-855622

Sent from my SM-G955F using Tapatalk

[gillmacca01 149]

This is something I desperately need, currently doing it manually, and with my collection will take months.

Do you know if this can be installed on a QNAP, or does it have to run on windows

[ginjaninja 533]

alas i dont know.

https://www.adilhindistan.com/2011/06/running-powershell-from-nas-without.html

suggests powershell CAN work on qnap

There isnt much help on the internet for powershell on QNAP so its perhaps not going to be straight forward. You could always like the feature request linked in post above and hope it gets enough upvotes to come to dev teams attention to add in as standard, it is a worthwhile improvement imo.

if you can  get powershell 7 working on qnap there  is a good chance you will get the script to work (it just needs powershell 7 + http access to api). I believe QNAP is linux based and there is a linux release.

https://www.qnapclub.eu/en/qpkg/1011#

seems to suggest that someone has packaged powershell 7.03 for QNAP but i cant vouch for it.

if you do give it a go, i would be interested to hear if you can get powershell working on qnap.

 

[gillmacca01 149]

This works great.

I have a request:

Would it be possible to create a script, that does the same, but for 'Parental Rating', 'Genres' and 'Studios'?

Edited October 11, 2020 by gillmacca01

[ginjaninja 533]

On 10/10/2020 at 22:05, gillmacca01 said:

This works great.

I have a request:

Would it be possible to create a script, that does the same, but for 'Parental Rating', 'Genres' and 'Studios'?

glad to hear it works on qnap.

i wont get time to develop for genres and studios in near future. Are you tempted to give it a go yourself? happy to offer advice.

this is not tested , but gives an idea on how the code could be tweaked, the apis calls need to retrieve the genredata, but the required logic is going to be very similar just working on the genredata. From what i can tell the api used fields=genres to return genres, and the resultant datastructure has genres under .genreitems.

i wouldnt run on a live system without some testing but its not going to be too far off from complete. just need a windows pc and visual studio code to give it a go.

although the case for removing genres on episodes when present on series is not entirely clear to me.

 

<#
v0.0.0.1

Removes Genres from Episodes if they are present on series object

#>

$Config = (Import-PowerShellDataFile -Path "$PSScriptRoot\Config.psd1")
$EmbyURL = $Config.Emby.URL
$EmbyApiKey = $Config.Emby.ApiKey

#Wipe the Duplciate logging file
if (Test-Path $PSScriptRoot\$($Config.Files.Duplicates)) {
    Clear-Content $PSScriptRoot\$($Config.Files.Duplicates)
}

#Retrieve all series from all librarys
$ApiURL = $EmbyURL + "/emby/Users/$EmbyApiKey/Items?Fields=Genres&IncludeItemTypes=Series&Recursive=true&SortBy=Name&api_key=$EmbyApiKey"
$serieslist = Invoke-WebRequest -Uri $ApiUrl -Method get -ContentType "application/json" -SessionVariable 'Session'
$serieslist = $serieslist.content | ConvertFrom-Json


$serieslist.items.foreach{
    Write-Host "Series "$_.name
    #Read-Host "pause for each series in loop"
    $series = $_.Name
    $seriesgenres = $_.genreitems


    #Get Episodes for Series
    $ApiURL = $EmbyURL + "/emby/Items?Recursive=true&Fields=Genres&ParentId=$($_.id)&IncludeItemTypes=Episode&api_key=$EmbyApiKey"
    $episodes = Invoke-WebRequest -Uri $ApiUrl -Method get -ContentType "application/json" -WebSession $Session
    $episodes = $episodes.content | ConvertFrom-Json



    #Loop through the episodes for the series
    $episodes.items.foreach{
        
        #$episode = $_
        $precount = $_.genreitems.count
        $pre = $_.genreitems


        $_.genreitems = $_.genreitems.where{
            $_.id -notin $seriesgenres.id
        }

    
        $postcount = $_.genreitems.count
        $post = $_.genreitems

        $Removed = $pre.where{ $_.name -notin $post.name }
        $removed = $removed | Select-Object -Property Name, Id

        $pre = $_.genreitems.name -join ","
        $post = $_.genreitems.Name -join ","
        $seriesg = $seriesgenres.name -join ","
    
        #If duplicates are found
        if ($precount -ne $postcount) {
        
            Write-Host "Duplicate Found" $Series $($_.id) $($_.Name) $($_.ParentIndexNumber) $($_.IndexNumber) $precount $postcount
            #Add-Content $PSScriptRoot\$($Config.Files.Duplicates) "`n$Series Season $($_.ParentIndexNumber) Episode $($_.IndexNumber)`nSeriesPeople $seriesp `nEpisodeAsis $pre`nEpisodeTobe $post"
            Add-Content $PSScriptRoot\$($Config.Files.Duplicates) "$Series Season $($_.ParentIndexNumber) Episode $($_.IndexNumber)"
            Add-Content $PSScriptRoot\$($Config.Files.Duplicates) "SeriesGenre $seriesg"
            $removed.foreach{
                Add-Content $PSScriptRoot\$($Config.Files.Duplicates) "Remove $($_.name) $($_.id)"

            }

            #if testmode is false removed duplicates
            if (!$Config.Testmode) {
                Write-Host "Cleaning " $Series Season $($_.ParentIndexNumber) Episode $($_.IndexNumber)
                $ApiURL = $EmbyURL + "/emby/Users/$EmbyApiKey/Items/$($_.id)?api_key=$EmbyApiKey"
                $fullmetadata = Invoke-WebRequest -Uri $ApiUrl -Method get -ContentType "application/json" -WebSession $Session
                $fullmetadata = $fullmetadata.Content | ConvertFrom-Json
                $fullmetadata.genreitems = $_.genreitems
                    
                $body = $fullmetadata | ConvertTo-Json

                $ApiURL = $EmbyURL + "/emby/Items/$($_.id)?api_key=$EmbyApiKey"
                $response = Invoke-WebRequest -Uri $ApiUrl -Method Post -Body $Body -ContentType "application/json" -WebSession $Session

            }
                
        }
  
    }




}

 

 

 

 

 

 

 

 

 

 

[gillmacca01 149]

I'll take a look when I get the chance. Thanks.

The reason for removing them from episode level is, for example, I select the genre tab, and then action, it shows the series, plus every episode in that series that has that genre 

[gillmacca01 149]

Works perfectly.

Looking at the script, it shouldn't be that hard to change it for studios/parental rating.

 

Many thanks. You have saved me weeks and maybe months of work

[gillmacca01 149]

Been using this successfully for quite a while, but I am now coming across an issue.

I have made no changes to the server (that i am aware of), or my QNAP server.

I am running the latest beta 4.7.0.17

When running the script, I am now getting the following error:

Invoke-WebRequest: C:\Users\gillm\Downloads\Person Cleaner\PersonCleaner.ps1:86
Line |
  86 |  … $response = Invoke-WebRequest -Uri $ApiUrl -Method Post -Body $Body - …
     |                ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
     | User does not have admin access.

 

[ginjaninja 533]

1 hour ago, gillmacca01 said:

Been using this successfully for quite a while, but I am now coming across an issue.

I have made no changes to the server (that i am aware of), or my QNAP server.

I am running the latest beta 4.7.0.17

When running the script, I am now getting the following error:

Invoke-WebRequest: C:\Users\gillm\Downloads\Person Cleaner\PersonCleaner.ps1:86
Line |
  86 |  … $response = Invoke-WebRequest -Uri $ApiUrl -Method Post -Body $Body - …
     |                ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
     | User does not have admin access.

 

this has been acknowledge by devs as an issue in .17, so i assume it will be fixed soon.

 

[gillmacca01 149]

Must have missed that post. Thanks

[ginjaninja 533]

 

updated to v0.0.0.2 to fix foreign characters in persons/actors names not getting written back correctly...not entirely sure what it does (i dont understand character encoding) but it worked on 'money heist' when tested.

 

removed apikey from config, script now requires population of  admin username and password

Edited January 1, 2022 by ginjaninja

[gillmacca01 149]

Config file is incomplete. Only mentions username and password, the rest of the script is missing

[ginjaninja 533]

On 02/01/2022 at 09:12, gillmacca01 said:

Config file is incomplete. Only mentions username and password, the rest of the script is missing

i couldnt see the issue, but have uploaded again

[gillmacca01 149]

Unfortunately, I am aware from home for the next 4 days, so won't be able to check until I return home at the weekend,

Will update then

[gillmacca01 149]

Tested, and working great. Thanks

[gillmacca01 149]

Need some help with something simple, but I just can't get it to work.

I'm trying to set this up in Windows task scheduler, so that I can get it to run automatically, but although the task says its running, nothing is actually happening.

I know how to create tasks, and in fact, have some running, but having difficulty with powershell in it

Can you help?

 

[ginjaninja 533]

On 12/07/2022 at 18:44, gillmacca01 said:

Need some help with something simple, but I just can't get it to work.

I'm trying to set this up in Windows task scheduler, so that I can get it to run automatically, but although the task says its running, nothing is actually happening.

I know how to create tasks, and in fact, have some running, but having difficulty with powershell in it

Can you help?

 

sure. i would run it manually against pwsh.exe to confirm its working, before scheduling it. This is what mine looks like.

(i use pwsh.exe / powershell 7, but i dont know what the actual minimum powershell version required is)

my working task settings are..

EmbyTasksDaily.xml

takes about 22 seconds to finish on my library (1020 shows cleaned every day)

 for troubleshooting might as well use a local administrator to run the task against, at least initially.

i cant account for why the task starts but never finishes, there is no opportunity for the script to halt...best guess its going v. slowly? name resolution? disk space for log file? i..running it manually outside of scheduler should give a good indication...

worth also rebooting/turning off scheduling for manual checking just in case there are loads of unfinished PowerShell instances clogging up the works....

..ensure using latest version/same as me...v0.0.0.3

Does that point you in the right direction? or can you think of a more specific question/idea?

 

 

 

Edited July 13, 2022 by ginjaninja

[gillmacca01 149]

Thanks for the info.

It runs fine manually. I did look on Google, but different sites said different things and I could get none to work.

I'll try your way in the next few days

[gillmacca01 149]

Well it appears to be working, but getting stuck on one show - American Gods.

The window shows: Duplicate Found Series:American Gods Episodeid:5235680 EpisodeName:A Winters Tale S3E1 OriginalPersonCount:33 NewPersonCount:28
Cleaning  American Gods Season 3 Episode 1

The duplicates file shows: American Gods Season 3 Episode 1
SeriesPeople Ian McShane,Emily Browning,Peter Stormare,Kristin Chenoweth,Pablo Schreiber,Ricky Whittle,Orlando Jones,Bruce Langley,Crispin Glover,Yetide Badaki
Remove Ricky Whittle Actor
Remove Ian McShane Actor
Remove Emily Browning Actor
Remove Bruce Langley Actor
Remove Yetide Badaki Actor

 

Nothing has changed in 11 mins

[gillmacca01 149]

Also notice my RAM shoots up to 95% (emby using a lot of memory). It had successfully cleaned other series prior to this one

Going to try removing this episode and see what happens then

Had to remove the whole of series 3 for it to carry on successfully.

Seems to be cleaning very slowly on 'an idiot abroad'.

The only things similar to this shows, is that I have video files and strm files

Edited July 14, 2022 by gillmacca01

[ginjaninja 533]

I am very surprised if it turns out file types have an effect..the script doesnt concern itself with files just the persons against series and episodes in the database.

running out of memory is never a 'good thing' so might be related, but not much the script can do about Emby process memory utilisation.

powershell's invoke-webrequest has a default timeout of 60 seconds...so i dont think the script can hang indefinitely unless powershell has 'fallen over', perhaps memory or the emby api fed powershell something it didnt like?

unfortunately i think this is going to require some boring troubleshooting to investigate.

you could try test mode $true in config to isolate if the issue happens in read only mode.

if you are pinning it down to falling over in the same place, and removing objects allows the script to continue...does putting the media objects back in cause the issue to return...  getting at a database corruption being root cause. ultimately you could backup the database and do a fresh scan and see if that affects the prominence of the issue.

you could add an if series -in or -notin statement to skip/align to troublesome shows as a tool to speed up troubleshooting..'return' here means "stop processing series/loop and move to the next series"

$series = $_.Name
    if ($series -notin "iZombie","itunes festival","black-ish") {return}
$seriespeople = $_.people.where{ $_.type -eq 'Actor' }

you could add some additional logging lines at points in the script to gain a more thorough understanding of where the script is hanging.

Add-Content $PSScriptRoot\$($Config.Files.Duplicates) "logging text goes here"

if the script falls over at the same point in an unchanging emby database, it implies powershell/script does not like response from api. if the script falls over at different points its more likely a system issue like memory utilisation i would think. worth trying to free up as much memory as possible for the test.

i cleaned both an idiot abroad and american gods after a full metadata retrieval which is some evidence theres not an inherent issue in the script itself..but that could be a red herring because its what comes after in the log which is the point of the halt. the logging suggestion above can help pin point the exact point eg reading the episode list? writing the episode list?

Duplicate Found Series:American Gods Episodeid:351996 EpisodeName:A Winter's Tale S3E1 OriginalPersonCount:33 NewPersonCount:28
Cleaning  American Gods Season 3 Episode 1
Duplicate Found Series:American Gods Episodeid:351997 EpisodeName:Serious Moonlight S3E2 OriginalPersonCount:32 NewPersonCount:26
Cleaning  American Gods Season 3 Episode 2
Duplicate Found Series:American Gods Episodeid:351998 EpisodeName:Ashes and Demons S3E3 OriginalPersonCount:19 NewPersonCount:14
Cleaning  American Gods Season 3 Episode 3
Duplicate Found Series:American Gods Episodeid:351999 EpisodeName:The Unseen S3E4 OriginalPersonCount:22 NewPersonCount:16
Cleaning  American Gods Season 3 Episode 4
Duplicate Found Series:American Gods Episodeid:352000 EpisodeName:Sister Rising S3E5 OriginalPersonCount:25 NewPersonCount:20
Cleaning  American Gods Season 3 Episode 5
Series American Gothic : No Change
Series American Vandal : No Change
Series Amy Schumer Learns to Cook : No Change
Duplicate Found Series:An Idiot Abroad Episodeid:99202 EpisodeName:China S1E1 OriginalPersonCount:3 NewPersonCount:0
Cleaning  An Idiot Abroad Season 1 Episode 1
Duplicate Found Series:An Idiot Abroad Episodeid:99203 EpisodeName:India S1E2 OriginalPersonCount:3 NewPersonCount:0
Cleaning  An Idiot Abroad Season 1 Episode 2
Duplicate Found Series:An Idiot Abroad Episodeid:99204 EpisodeName:Jordan S1E3 OriginalPersonCount:3 NewPersonCount:0   
Cleaning  An Idiot Abroad Season 1 Episode 3
Duplicate Found Series:An Idiot Abroad Episodeid:99205 EpisodeName:Mexico S1E4 OriginalPersonCount:3 NewPersonCount:0
Cleaning  An Idiot Abroad Season 1 Episode 4
Duplicate Found Series:An Idiot Abroad Episodeid:99206 EpisodeName:Egypt S1E5 OriginalPersonCount:3 NewPersonCount:0
Cleaning  An Idiot Abroad Season 1 Episode 5
Duplicate Found Series:An Idiot Abroad Episodeid:99207 EpisodeName:Brazil S1E6 OriginalPersonCount:3 NewPersonCount:0   
Cleaning  An Idiot Abroad Season 1 Episode 6
Duplicate Found Series:An Idiot Abroad Episodeid:99208 EpisodeName:Peru S1E7 OriginalPersonCount:3 NewPersonCount:0
Cleaning  An Idiot Abroad Season 1 Episode 7
Duplicate Found Series:An Idiot Abroad Episodeid:426117 EpisodeName:Karl Comes Home: The Debrief S1E8 OriginalPersonCount:3 NewPersonCount:0
Cleaning  An Idiot Abroad Season 1 Episode 8
Duplicate Found Series:An Idiot Abroad Episodeid:99210 EpisodeName:Desert Island S2E1 OriginalPersonCount:4 NewPersonCount:1
Cleaning  An Idiot Abroad Season 2 Episode 1
Duplicate Found Series:An Idiot Abroad Episodeid:99211 EpisodeName:Trans-Siberian Express S2E2 OriginalPersonCount:3 NewPersonCount:0
Cleaning  An Idiot Abroad Season 2 Episode 2
Duplicate Found Series:An Idiot Abroad Episodeid:99212 EpisodeName:Dolphin Swim S2E3 OriginalPersonCount:3 NewPersonCount:0
Cleaning  An Idiot Abroad Season 2 Episode 3
Duplicate Found Series:An Idiot Abroad Episodeid:99213 EpisodeName:Whale Watching S2E4 OriginalPersonCount:3 NewPersonCount:0
Cleaning  An Idiot Abroad Season 2 Episode 4
Duplicate Found Series:An Idiot Abroad Episodeid:99214 EpisodeName:Meet a Gorilla S2E5 OriginalPersonCount:3 NewPersonCount:0
Cleaning  An Idiot Abroad Season 2 Episode 5
Duplicate Found Series:An Idiot Abroad Episodeid:99215 EpisodeName:Route 66 S2E6 OriginalPersonCount:3 NewPersonCount:0
Cleaning  An Idiot Abroad Season 2 Episode 6
Duplicate Found Series:An Idiot Abroad Episodeid:99216 EpisodeName:Climb Mount Fuji S2E7 OriginalPersonCount:3 NewPersonCount:0
Cleaning  An Idiot Abroad Season 2 Episode 7
Duplicate Found Series:An Idiot Abroad Episodeid:99217 EpisodeName:Karl Comes Home S2E8 OriginalPersonCount:3 NewPersonCount:0
Cleaning  An Idiot Abroad Season 2 Episode 8
Duplicate Found Series:An Idiot Abroad Episodeid:426114 EpisodeName:The Short Way Round: Venice S3E1 OriginalPersonCount:4 NewPersonCount:1
Cleaning  An Idiot Abroad Season 3 Episode 1
Duplicate Found Series:An Idiot Abroad Episodeid:426115 EpisodeName:The Short Way Round: India S3E2 OriginalPersonCount:4 NewPersonCount:1
Cleaning  An Idiot Abroad Season 3 Episode 2
Duplicate Found Series:An Idiot Abroad Episodeid:426116 EpisodeName:The Short Way Round: China S3E3 OriginalPersonCount:4 NewPersonCount:1
Cleaning  An Idiot Abroad Season 3 Episode 3

 

Edited July 15, 2022 by ginjaninja