Jump to content

Duplicate People (New)

TeamB

By TeamB
in General/Windows

 Share

Recommended Posts

TeamB

TeamB 2438

Posted
Posted (edited)

@@Luke

 

I am starting a new topic on this as the other one is marked answered and this one I think is a different issue.

 

I am seeing duplicate people in some items (Movies and TV Shows)

 

post-1880-0-24370700-1584315014_thumb.png

 

I tracked this down to duplicate entries in the ItemPeople table

select ROWID, *
from ItemPeople
where ItemId = 61159
order by ListOrder

results

967782	61159	27162	24070	Jean-Luc Picard	0	0
967783	61159	27162	24070	Jean-Luc Picard	0	1
967784	61159	27162	61185	Dahj / Soji Asha	0	2
967785	61159	27162	22915	Raffi Musiker	0	3
967786	61159	27162	26154	Cristobal 'Chris' Rios	0	4
967787	61159	27162	61186	Agnes Jurati	0	5
967788	61159	27162	61187	Narek	0	6
967789	61159	27162	61188	Elnor	0	7

I can delete these duplicates using the following SQL

delete
from ItemPeople
where RowID in
(
    select RowID
    from
    (
        select
            min(ROWID) as ROWID,
            ItemId,
            TopParentId,
            PersonId,
            Role,
            PersonType,
            min(ListOrder) as ListOrder,
            COUNT(1) AS RowCount
        from ItemPeople
        group by ItemId, TopParentId, PersonId, Role, PersonType
        having RowCount > 1
    )
)

and that fixes it, until I do a meta data refresh and then they come back.

 

I have attached a debug log snip of the meta data refresh

 

This is reproducible and in the current stable release.

embyserver.txt

Edited by TeamB
Happy2Play

Happy2Play 9780

Posted
Posted (edited)

@@Luke is the a TVDB metadata issue? I can't reproduce this though.

 

actors.xml for Star Trek: Picard

<?xml version="1.0"?>
<Actors>
  <Actor>
    <id>65436298</id>
    <Image>actors/540295.jpg</Image>
    <Name>Patrick Stewart</Name>
    <Role>Jean-Luc Picard</Role>
    <SortOrder>1</SortOrder>
  </Actor>
  <Actor>
    <id>65436298</id>
    <Image>person/284167/65436298.jpg</Image>
    <Name>Patrick Stewart</Name>
    <Role>Jean-Luc Picard</Role>
    <SortOrder>1</SortOrder>
  </Actor>
  <Actor>
    <id>65436446</id>
    <Image>person/8160522/65436446.jpg</Image>
    <Name>Isa Briones</Name>
    <Role>Dahj / Soji Asha</Role>
    <SortOrder>2</SortOrder>
  </Actor>
  <Actor>
    <id>65436296</id>
    <Image>person/269074/65436296.jpg</Image>
    <Name>Michelle Hurd</Name>
    <Role>Raffi Musiker</Role>
    <SortOrder>3</SortOrder>
  </Actor>
  <Actor>
    <id>65436299</id>
    <Image>person/349415/65436299.jpg</Image>
    <Name>Santiago Cabrera</Name>
    <Role>Cristobal 'Chris' Rios</Role>
    <SortOrder>4</SortOrder>
  </Actor>
  <Actor>
    <id>65436294</id>
    <Image>person/317143/65436294.jpg</Image>
    <Name>Alison Pill</Name>
    <Role>Agnes Jurati</Role>
    <SortOrder>5</SortOrder>
  </Actor>
  <Actor>
    <id>65436295</id>
    <Image>person/486763/65436295.jpg</Image>
    <Name>Harry Treadaway</Name>
    <Role>Narek</Role>
    <SortOrder>6</SortOrder>
  </Actor>
  <Actor>
    <id>65437524</id>
    <Image>person/315275/65437524.jpg</Image>
    <Name>Jonathan Del Arco</Name>
    <Role>Hugh</Role>
    <SortOrder>7</SortOrder>
  </Actor>
  <Actor>
    <id>65436293</id>
    <Image>person/277950/65436293.jpg</Image>
    <Name>Jeri Ryan</Name>
    <Role>Seven of Nine</Role>
    <SortOrder>8</SortOrder>
  </Actor>
  <Actor>
    <id>65436293</id>
    <Image>person/277950/65436297.jpg</Image>
    <Name>Jeri Ryan</Name>
    <Role>Seven of Nine</Role>
    <SortOrder>8</SortOrder>
  </Actor>
  <Actor>
    <id>65436447</id>
    <Image>person/8160523/65436447.jpg</Image>
    <Name>Evan Evagora</Name>
    <Role>Elnor</Role>
    <SortOrder>9</SortOrder>
  </Actor>
</Actors>

en.xml series section

<Series>
    <id>364093</id>
    <Actors>|Patrick Stewart|Patrick Stewart|Isa Briones|Michelle Hurd|Santiago Cabrera|Alison Pill|Harry Treadaway|Jonathan Del Arco|Jeri Ryan|Jeri Ryan|Evan Evagora|</Actors>
    <Airs_DayOfWeek>Thursday</Airs_DayOfWeek>
    <Airs_Time>3:00 AM</Airs_Time>
    <ContentRating>TV-MA</ContentRating>
    <FirstAired>2020-01-23</FirstAired>
    <Genre>|Science Fiction|</Genre>
    <IMDB_ID>tt8806524</IMDB_ID>
    <Language>en</Language>
    <Network>CBS All Access</Network>
    <NetworkID>1097</NetworkID>
    <Overview>Jean-Luc Picard, former captain of the starship U.S.S. Enterprise, is pulled out of forced retirement and the next chapter of his life, to help a young woman who is pursued by assassins. 
</Overview>
    <Rating>0</Rating>
    <RatingCount>0</RatingCount>
    <Runtime>45</Runtime>
    <SeriesID/>
    <SeriesName>Star Trek: Picard</SeriesName>
    <Status>Continuing</Status>
    <added>2019-05-15 22:17:16</added>
    <addedBy/>
    <banner>graphical/5d2649b512977.jpg</banner>
    <fanart>fanart/original/5ce6d7f608fe6.jpg</fanart>
    <lastupdated>1584308618</lastupdated>
    <poster>posters/5d262b402df21.jpg</poster>
    <zap2it_id/>
  </Series>

You can see Actor is listed twice.

Edited by Happy2Play
Happy2Play

Happy2Play 9780

Posted
Posted (edited)

To me this would be a interpretation of same name actor do to metadata provider defect, at least this example

Edited by Happy2Play
TeamB

TeamB 2438

Posted
  • Author
Posted

what is the API for the meta data provider?

TeamB

TeamB 2438

Posted
  • Author
Posted (edited)

Yes it looks like a meta data source issue

https://www.thetvdb.com/api/B89CE93890E9419B/series/364093/all/en.zip

has duplicates.

 

this data source (thetvdb) looks like it has gone off the rails recently, why is it still used, is there no alternative?

 

Can Emby Dedupe the data at load time? The above are very obvious delicate data in the source data.

Edited by TeamB
Happy2Play

Happy2Play 9780

Posted
Posted (edited)

 

 

Yes it looks like a meta data source issue

https://www.thetvdb.com/api/B89CE93890E9419B/series/364093/all/en.zip
show duplicates.

 

this data source (thetvdb) looks like it has gone off the rails recently, why is it still used, is there no alternative?

 

Only TMDB for people.  But there is no provider logic for people any more.

Edited by Happy2Play
Happy2Play

Happy2Play 9780

Posted
Posted

I just looked in the cache folder per provider. but grabbing a new zip should show the same as above.

Luke Emby Team

Luke 42078

Posted
Posted

The issue isn't so much the fact that they're in the database, it's how did they get there. Duplicates in one single series won't be a problem unless they're spelled differently, which I'm assuming they are not here.

 

My guess is this is actually being caused by defects in older server versions and is not actually an issue anymore. For example, remember last year when we were using the tvdb person identifier as a unique person id...well it turns out that id is per show, which means the same people will have different id's for different shows, and with older server versions this would result in duplicates. It could still result in duplicates now depending on what bad data you may have from older versions.

 

The way to climb out of this is to pick one of these people, and remove all media from your library that the person appears in - movies, series, episodes, etc. Then run a full scan of all libraries to clean out the database, and ideally the people refresh task as well.

 

Then add the media back and see if you can still reproduce the problem. My guess is you will not be able to.

TeamB

TeamB 2438

Posted
  • Author
Posted

The duplicate data was in the original the TV db data source. I think it was just pulled in by emby on meta data refresh

Luke Emby Team

Luke 42078

Posted
Posted

The duplicate data was in the original the TV db data source. I think it was just pulled in by emby on meta data refresh

 

This is the action that made the problem visible but i think the real issue happened from an earlier server version as I outlined above.

Luke Emby Team

Luke 42078

Posted
Posted

@@Happy2Play probably remembers this. I pushed the tvdb plugin fix in July. I'll see if i can find out how long the plugin had that code.

Luke Emby Team

Luke 42078

Posted
Posted

Ok it was added in November 2018, then later fixed/removed in July 2019.

Happy2Play

Happy2Play 9780

Posted
Posted

Can look for the topic if needed, but TVDB does per show/series character images, so you could ended up with multiple same person image do to different TVDB person ids.  So this behaviour has changed in Emby over time.

TeamB

TeamB 2438

Posted
  • Author
Posted

The issue isn't so much the fact that they're in the database, it's how did they get there. Duplicates in one single series won't be a problem unless they're spelled differently, which I'm assuming they are not here.

 

My guess is this is actually being caused by defects in older server versions and is not actually an issue anymore. For example, remember last year when we were using the tvdb person identifier as a unique person id...well it turns out that id is per show, which means the same people will have different id's for different shows, and with older server versions this would result in duplicates. It could still result in duplicates now depending on what bad data you may have from older versions.

 

The way to climb out of this is to pick one of these people, and remove all media from your library that the person appears in - movies, series, episodes, etc. Then run a full scan of all libraries to clean out the database, and ideally the people refresh task as well.

 

Then add the media back and see if you can still reproduce the problem. My guess is you will not be able to.

Ok so just removing the duplicate data form the ItemPeople Table is not going to be enough?

Luke Emby Team

Luke 42078

Posted
Posted

Well if you're finding your own workaround in the database that might be fine. The advice I gave above was mostly targeting end users who won't be able to do that.

TeamB

TeamB 2438

Posted
  • Author
Posted

Just to be clear, deleting to dupe data from the itempeople table did not work, as soon as I did a meta data refresh on that item it added back the dupe data.

Luke Emby Team

Luke 42078

Posted
Posted

Yea i think you need to eradicate that person entirely from your database. 

TeamB

TeamB 2438

Posted
  • Author
Posted (edited)

I dont think it is anything to do with old data or older versions.

 

I just tested a clean install of 4.3.1.0 with just one TV show "Star Trek- Picard" and I see duplicate actors.

 

This is a very easy test to set up and reproduce.

Edited by TeamB
Happy2Play

Happy2Play 9780

Posted
Posted (edited)

I dont think it is anything to do with old data or older versions.

 

I just tested a clean install of 4.3.1.0 with just one TV show "Star Trek- Picard" and I see duplicate actors.

 

This is a very easy test to set up and reproduce.

I find this very odd as I just added "Star Trek- Picard" to a new system and do not get duplicates, but have been testing on betas.

Edited by Happy2Play
PenkethBoy

PenkethBoy 2068

Posted
Posted

I tested this last night on .29 and ST-Picard gave me duplicates of JLP and 7of9 - i also had this when i first added STP to my main system - just deleted the duplicates and moved on

 

the beta test server has a new db (approx 1 week old) - so never had STP before - no baggage from old db's - which i think is a red herring

 

its repeatable - on a refresh of the series the duplicates comeback

 

As this is duplicate data coming from TVDB - you just need to delete the dup cast from the series metadata screen - no db sql needed - as i believe its "just" an import  issue

 

For me it happens in Stable and Beta

  • Like 1
TeamB

TeamB 2438

Posted
  • Author
Posted

Yes you can just delete the dupe actor but does it come back when you do a meta data refresh?

CBers

CBers 7450

Posted
Posted

I recently added the TV show "9-1-1: The Lone Star" and I see duplicate actor images, including multiple duplicates of "Ronen Rubinstein".

 

5e74cc8988812_EmbyDuplicateActors1.jpg

 

5e74cc9ff0096_EmbyDuplicateActors2.jpg

 

5e74ccb33f632_EmbyDuplicateActors3.jpg

 

He doesn't appear an any other TV show or Movie in Emby.

CBers

CBers 7450

Posted
Posted

Deleting the duplicate actors from the TV show works great.

 

5e74cdc8f1413_EmbyDuplicateActors4.jpg

 

But, as @penkeyboth says, if you refresh the metadata of the TV show, you get the duplicates back :rolleyes:

CBers

CBers 7450

Posted
Posted

Deleting the duplicate actors from the TV show works great.

 

5e74cdc8f1413_EmbyDuplicateActors4.jpg

 

But, as @penkeyboth says, if you refresh the metadata of the TV show, you get the duplicates back :rolleyes:

 

Partial server log attached, which covers the deletion of the duplicate actors and the metadata refresh.

embyserver.txt

Luke Emby Team

Luke 42078

Posted
Posted

You need to get them all deleted, not just the extras.

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now
 Share
Go to topic listing
×
×
  • Create New...