Duplicate media files after update to 4.8.0.80

[queenbuzzo 8]

Hi, 

I have already posted in this thread but my issue actually is different so I thought I post it here as well. After the update to 4.8.0.80 all my media files are listed twice and I can't figure out what is going on. Emby seems to think there are two files/version for the same movies or series but there are not.

This is how it looks like for an episode of a series I'm currently watching:

Both entries has different IDs and play fine which even puzzles me more:
http://192.168.0.254:8096/web/index.html#!/item?id=17724221&serverId=XXX&context=home
http://192.168.0.254:8096/web/index.html#!/item?id=22444088&serverId=XXX&context=home

For movies it looks like this:

 

I can see that there are two versions listed but it is actually the very same file.

Edit: I have checked the Reports plugin and all series, seasons and episodes are listed twice (they have a different rating though). Movies seem normal. What a mess...

 

Any help would be greatly appreciated. I'm even considering downgrading for now, if that is possible (but I'm not sure about all the changes, especially the DB). I'm using the linuxserver.io container.

Thanks

Edited February 5 by queenbuzzo

[sargenthp 29]

I will need to look.  I have the same issue with music.

https://emby.media/community/index.php?/topic/125273-emby-server-beta-48075-song-list-duplicate/#comment-1320080

[queenbuzzo 8]

Interesting, could be related. In the meanwhile I have checked the DB itself and there are two different entries for the same file. I just don't understand what is going on there and why Emby is behaving so weird. 

[queenbuzzo 8]

So I removed my libraries and guess what? I can still see them in the client and play back my stuff. How do I get my server running properly again? Do I have to reset everything?

@LukeCan you please have a look?

[queenbuzzo 8]

I did another more thorough investigation, here are my findings:

First of all: I have 8 libraries listed in Emby: 4K Movies, HD Movies, Series, Documentaries, Trash Movies, Recordings (unused), Playlists (from Emby), Collections (from Emby).

Only the HD Movies and series libraries are showing the issue. All other libs are fine. Also newly added things are fine.

Since Emby sees two different versions of my files, I grabbed the library database and checked it. What I can see is that each of my movie files or episodes have a duplicate entry but they do have the same creation date:

This is here seems to be the interesting part. The bottom timestamp is the time my Emby server was update to the newest version so something happened that created these duplicate entries. 

 

Besides the IDs there's only one major difference I can spot and that is the ratings stuff:

 

As I mentioned earlier I can remove the library completely and what I can tell now is that new "library" will stay in the background. It's not listed as a library anywhere but I can search for the items in it. 

I'm wondering if there's anything I can do or if this is something you will take care of? If not, I need to know where the watch states are saved so I can rebuild my server, maybe trying it on a database level... and have a longer back up retention and disable auto updates....

 

Thanks 

queenbuzzo

Edited February 6 by queenbuzzo

[sargenthp 29]

Interesting.  Appears my main music is the only library with the duplicates.  I also have a Seasonal Music library which is fine.  I didn't think about it while I upgraded, but I had music playing during the upgrade.  Do you know if someone was watching a movie when you started the upgrade?

Edited February 6 by sargenthp
More details

[queenbuzzo 8]

No, no one used the server at all on Friday when the update happened. I only noticed it a day later when I saw the duplicated entries.

[Luke 37260]

Hi, please see here for my thoughts and suggested action on this: 

 

[queenbuzzo 8]

Thank you! This doesn't seem to work for me. Even when I delete the libraries with the issue (series and movies) and do a scan, there is still some kind of a phantom library. I can still search for the items and the reports plugin also still sees them. Is there a way to reset the DB without losing my watched states (and users etc.)?

Edit: I'm trying it on the Database itself, but there are too many key constraints to get to a quick solution. Maybe you or someone from your team has an idea for this? 

Edited February 6 by queenbuzzo

[Luke 37260]

2 minutes ago, queenbuzzo said:

Thank you! This doesn't seem to work for me. Even when I delete the libraries with the issue (series and movies) and do a scan, there is still some kind of a phantom library. I can still search for the items and the reports plugin also still sees them. Is there a way to reset the DB without losing my watched states (and users etc.)?

Edit: I'm trying it on the Database itself, but there are too many key constraints to get to a quick solution. Maybe you or someone from your team has an idea for this? 

You could try to remove all libraries. Also make note of the plugins I mentioned in that thread in case you have any of those. 

[queenbuzzo 8]

No chance, even if I remove all libraries and plugins (besides reports) there is still a ghost library. It doesn't show up in the settings or sidebar but I can still search for the items and see them in the reports plugin.

[Luke 37260]

Right I still don't think it was caused by this update, those entries were probably lingering before but just not visible until now. The question is how to remove them. In the database you'd have to walk through all of the constraints and first delete records in other tables in order to get the mediaitems deletes to succeed.  There's about 8-10 tables that you'll first have to delete records from. If you can get through that then you can avoid the destructive route of a full fresh start.

[queenbuzzo 8]

I'm not sure, but to me it all points to the update since all new entries have a DateLastRefreshed on the 2nd of February when the update happened, some even have a creation date on this day. It might all be coincidence but at least hints to the update.

I'm now going the database route and while I think I found all constraints, I still get a foreign key issue. I tried to ignore the FK constraints but was working on the DB on my laptop and that shut down during the night while the operation was running and the deletion was not finished. I have no experience with SQLite at all (but have some with other databases) and am wondering how long deletion of roughly 10k items would take. Do you maybe have a rough idea? 

Oh and another question: Should I delete the old or new duplicate entry? What do you think? Currently I'm opting to delete all entries that have DateLastRefreshed > 02.02.2024 10am which is a good hour before the update started but I could also do it the other way around. 

 

[sargenthp 29]

I would delete the new entries (at least for the music the original ID, not the newer ID, was linked into my playlists).  Wasn't sure if I could just delete from the MediaItems table and the other associated tables would clean themselves up, or if I would have to clean all linking tables.  My guess the later.

10k records is nothing for a computer.  My guess a couple minutes at the most.

With my duplicating music I did just remove and readd my Library.  It did keep my playlists and favorites intact.

[queenbuzzo 8]

Yeah I'm also intrigued to remove the new entries. Unfortunately you will have to delete the linked items that use the Id as a foreign key as well so it's a bit cumbersome. The deletion process I started yesterday took I think half an hour already so I wanted to leave it for the night but the laptop died. I will try it again later and hope for the best. 

[queenbuzzo 8]

I haven't tried to alter the DB yet but today I noticed that also newly added items will show twice. So @Luke's theory about the items being there from the start already doesn't seem to be correct. I think there is some kind of ghost library where these things are getting added to. I really need a solution for this, this is clearly an issue with the update. 

Some questions: 
Are the libraries saved in the database somewhere so I could look for them? I'm guessing it's the ParentID, is that correct?
Is there a way to downgrade and see what happens? 
What would be the best way to reset the server if needed. Is there any way to save my watched states?

Edited February 8 by queenbuzzo

[queenbuzzo 8]

I had some time now during the weekend and I think it's solved (and hopefully for good). I'm still pretty sure the second libraries for moves and series were created during the update, but I don't know why. I had to do some trial and error in the database but eventually I got there.

So what I did:

- Ran a query that gets the Ids of all duplicates in the DB with the newer refresh date and use this to delete all foreign keys in any other table

- Deleted all row that had the ParentId of the new libraries in MediaItems

- Deleted the two new libraries

- Deleted all other duplicates from MediaItems with the newer refresh date 

 

Library seems clean now and adding new files is also working fine. And lessons learned: Disable auto updates and keep backups longer, 5 days retention was not enough as I noticed the issues too late. 

[wolong_zb 8]

I have the same problem, about 5% of movies repeat twice and up to 4 times, all pointing to the same path (soft links point to other paths). After deleting the item, it will continue to appear after refreshing. Unless the path of the file is changed, it will not appear after rescan.
If you create a new server and then add the media library again, rescan add, then this image will not appear.

The impression is from emby 4.8.x.

Ubuntu 22.10/emby 4.8.1.0

Edited February 27 by wolong_zb

[wolong_zb 8]

I have the same problem, about 5% of movies repeat twice and up to 4 times, all pointing to the same path (soft links point to other paths). After deleting the item, it will continue to appear after refreshing. Unless the path of the file is changed, it will not appear after rescan.
If you create a new server and then add the media library again, rescan add, then this image will not appear.

The impression is from emby 4.8.x.

Ubuntu 22.10/emby 4.8.1.0

[Luke 37260]

2 hours ago, wolong_zb said:

I have the same problem, about 5% of movies repeat twice and up to 4 times, all pointing to the same path (soft links point to other paths). After deleting the item, it will continue to appear after refreshing. Unless the path of the file is changed, it will not appear after rescan.
If you create a new server and then add the media library again, rescan add, then this image will not appear.

The impression is from emby 4.8.x.

Ubuntu 22.10/emby 4.8.1.0

Hi, did you try the suggestions I mentioned earlier?

[wolong_zb 8]

21 hours ago, Luke said:

Hi, did you try the suggestions I mentioned earlier?

Because the project data is large, it is not operated by deleting the media library for the time being;
I delete all duplicate items by editing the library.db file using the following sql statement; Then re-scan refresh, can normally display a single item; But it reappeared the next day. After checking the log, the log describes that the task started at 04:10 and item was added again.
After analyzing the mediaitems table in the database, it is found that among the duplicate items, the field externalId of the item added later points to the previous item, as shown in the figure below.

del-same-items.txt embyserver-63844708508.txt

[Luke 37260]

8 minutes ago, wolong_zb said:

Because the project data is large, it is not operated by deleting the media library for the time being;
I delete all duplicate items by editing the library.db file using the following sql statement; Then re-scan refresh, can normally display a single item; But it reappeared the next day. After checking the log, the log describes that the task started at 04:10 and item was added again.
After analyzing the mediaitems table in the database, it is found that among the duplicate items, the field externalId of the item added later points to the previous item, as shown in the figure below.

del-same-items.txt 1.57 kB · 1 download embyserver-63844708508.txt 681.85 kB · 0 downloads

The duplicate is caused by the watch history plugin as I mentioned in the other topic. Please remove that plugin and then you shouldn't see any new duplicates.

[wolong_zb 8]

3 hours ago, Luke said:

The duplicate is caused by the watch history plugin as I mentioned in the other topic. Please remove that plugin and then you shouldn't see any new duplicates.

Thanks for your answer, I have uninstalled the viewing history plugin.