Migration Plugin

[pünktchen 1258]

As most of you have noticed, there were a lot of database and metadata structure changes between Emby 4.5 and Emby 4.6.
The Emby devs did a great job to make that update as seamless as possible and it least from an end user perspective, looking at the frontend all is well.

But i'm not really happy with the transition that happens in the background of Emby. If you have the nfo plugin installed, for existing collections nfo files will be created under "metadata\collection", but the images in the database are still referenced under "data\collection" (boxset folders). Similar things happen to people. New people and their images will be stored under "metadata\people", while existing people images are still referenced under "metadata\library\xx\yyyyyyyyyyyyyyyy". Those things make the great goal to make Emby really portable and to simplify backup/restore processes impossible, except you've started with a new Emby 4.6 installation.

Here comes my plugin into play.
Of course i don't have to tell you: Make a backup of Emby's programdata folder first!!!

How does the plugin work? Please read carefully!

People migration
There are two modes to choose from. Either the people source is Emby's database or it is an older people folder with A to Z subfolders.

What happens by selecting legacy people folder as source?
For every person the plugin finds in the old A to Z folders, the images and metadata is first imported into the database to matched persons. After this the same process as with database as a source is happening. For persons where no match exists, a named folder under metadata\people is created and a nfo is written that holds the legacy xml metadata. The images are copied over from the old A to Z folders to the new people folder structure.
 

What happens by selecting database as source?
For every person the plugin finds in the database, a named folder under metadata\people is created and a nfo is written that holds the database metadata.
The images are copied over from library\xx\yyyyyyyyyyyyyyyy to the new people folder structure and the database references for the images will be corrected.
The now no more necessary library\xx\yyyyyyyyyyyyyyyy folder gets deleted.

No matter what source mode you've chosen, if the metadata\people folder still has the old A to Z folders, they will be moved to a new folder "_legacy".
This "_legacy" folder will also be migrated if you've selected legacy people folder as source, even if the path input field is empty.

No existing people nfo or images under metadata\people will be touched, only new data gets written.

Collection migration
The collection migration queries the database for your collections. For each collection it finds, it will create a named folder under metadata\collections and writes the metadata to a collection.nfo file. It will not touch existing nfo files. Further if the images of the collection are still referencing files under the deprecated boxset folders, the images are copied over to the new collection folder and the database entries will be updated. Only image files that are stored in the database will be considered!

Try to recreate missing collections from boxset folders, as the name suggests creates collections in the database that were not picked up by Emby's update procedure.
There's a good chance that this will restore some missing music album collections, but at the same time it can also create some unwanted collections.

Write set tags to existing media item nfo where possible trys to add missing "set tags" to the nfo files of the collection items. It will not create those nfo files, only modify them.
This should work without problems for movies, tv shows and tv episodes. It maybe fails for music albums or music artists.

Delete old boxset folders will delete the whole data\collections folder after the plugin migration has finished to make sure Emby will not pickup images from this location anymore.

Although this all worked fine for me, use this at your own risk! I'm not responsible for any damage of your Emby installation!
Make a backup of Emby's programdata folder first!!!

MigrationPlugin.0.2.0_Emby.4.5.zip MigrationPlugin.0.2.0_Emby.4.6.zip MigrationPlugin.0.2.0_Emby.4.7.zip

Edited August 1, 2022 by pünktchen
Added version 0.2.0.0 -> Write people provider ids to nfo also for directors and writers

[GrimReaper 3286]

Before I dive into this, could you pls tell us what is the future of People backup plugin, will it be (or is it already) updated for 4.6 stable? 

Of all the plugins I have, that's the one I use most, by far, as I scrape all actor images via TMM in and import them to Emby via same plugin. Would really hate to see it being abandoned. 

Thanks

[pünktchen 1258]

The people backup plugin will be modified to only backup/restore people from media folders.

[GrimReaper 3286]

2 minutes ago, pünktchen said:

The people backup plugin will be modified to only backup/restore people from media folders.

Excellent, thanks. 

[Ceten 27]

Thanks, I used your plugin to move the collection images to the new folder, it worked perfectly!

[letterman 33]

Incredible, thx für People Legacy Migration!
Worked fine. I was waiting for this tool over 1 year.

But Migrate collections did not work at all:
"Migration unexpectedly aborted", independently, which settings I selected.
An idea what may be wrong?

[pünktchen 1258]

1 hour ago, letterman said:

But Migrate collections did not work at all:
"Migration unexpectedly aborted", independently, which settings I selected.
An idea what may be wrong?

Maybe if you send me your Emby log.

[letterman 33]

I sent via PM

[PeteMB 12]

Migrating Collections and People that exist in the database worked. But now I have problems with migration of People that exists in legacy people folders, I got an error "Migration unexpectedly aborted". When I look in the logs I see this error.

2021-05-26 19:00:19.639 Info Migration Plugin: People Migration > Legacy people Person: E:\Emby-Server\metadata\People\_legacy\A\Aaron D. Spears
2021-05-26 19:00:19.641 Error Migration Plugin: Migration of People from Legacy Folder unexpectedly aborted
    *** Error Report ***
    Version: 4.6.1.0
    Command line: E:\Emby-Server\system\EmbyServer.dll
    Operating system: Microsoft Windows 6.3.9600
    Framework: .NET Core 3.1.13
    OS/Process: x64/x64
    Runtime: E:/Emby-Server/system/System.Private.CoreLib.dll
    Processor count: 8
    Data path: E:\Emby-Server
    Application path: E:\Emby-Server\system
    System.ArgumentException: System.ArgumentException: An item with the same key has already been added. Key: imdb
       at System.Collections.Generic.Dictionary`2.TryInsert(TKey key, TValue value, InsertionBehavior behavior)
       at MigrationPlugin.Migration.MigratePeopleMetadata.ParseXmlNodes(XmlReader reader, PersonItem personItem)
       at MigrationPlugin.Migration.MigratePeopleMetadata.ParseXmlFile(PersonItem personItem, String xmlFile, XmlReaderSettings settings, Encoding encoding)
       at MigrationPlugin.Migration.MigratePeopleMetadata.MigrateLegacyPeopleMetadata(String person, String metadataPath)
       at MigrationPlugin.MigrationPlugin.MigratePeopleFromLegacyFolder(IProgress`1 progress, String metadataPath, List`1 legacyPeopleFolders)
       at MigrationPlugin.MigrationPlugin.MigratePeople(IProgress`1 progress)
    Source: System.Private.CoreLib
    TargetSite: Void ThrowAddingDuplicateWithKeyArgumentException[T](T)
    

I can fix this by deleting the folder, in the case of this error "Aaron D. Spears", but it seems there are too many of these errors  

During the migration of the People in the database I see also a lot of errors like this:

2021-05-26 17:47:00.856 Info Migration Plugin: People Metadata Migration > Person: April Turner; Source: Database; Destination: E:\Emby-Server\metadata\people\April Turner\person.nfo
2021-05-26 17:47:00.858 Warn Migration Plugin: People Image Migration > Person: April Turner; Source: E:\Emby-Server\metadata\People\A\April Turner\folder.jpg; File not found!

And this is to be expected because you moved the folders to E:\Emby-Server\metadata\People\_legacy

[pünktchen 1258]

2 hours ago, PeteMB said:

I can fix this by deleting the folder, in the case of this error "Aaron D. Spears", but it seems there are too many of these errors  

Give the next version a chance, it's hopefully fixed then.

2 hours ago, PeteMB said:

During the migration of the People in the database I see also a lot of errors like this:

2021-05-26 17:47:00.856 Info Migration Plugin: People Metadata Migration > Person: April Turner; Source: Database; Destination: E:\Emby-Server\metadata\people\April Turner\person.nfo
2021-05-26 17:47:00.858 Warn Migration Plugin: People Image Migration > Person: April Turner; Source: E:\Emby-Server\metadata\People\A\April Turner\folder.jpg; File not found!

And this is to be expected because you moved the folders to E:\Emby-Server\metadata\People\_legacy

I will change the order of the plugin options, so it's more logical and adjust the code that "people\_legacy" will always be migrated before the database.

[pünktchen 1258]

I've updated the plugin in the start post.

[dvesely 1]

Thanks for making this.

I'm kind of new to Emby. Can you (or anyone here) tell me...

1. How do I install a plugin like this that is only one file? Where does it go? How do I configure it? Is there a manual?

2. I have all my movies, tv shows, etc on and external disk. I want to take it with me and attach it to my laptop when I travel.  I believe the database is on my C drive and needs to be moved to the external disk and my laptop version of Emby needs to be configured to know that it's data is all on the external disk. Can anyone help me do this? I'm hoping this tool will help.

 Thanks!

[PeteMB 12]

The People migration was now completed successful, but I had to run the "Scan Metadata Folder" scheduled task for the people images to show up.

Can I now remove the _legacy folder? (after I made a backup)

[pünktchen 1258]

2 minutes ago, PeteMB said:

Can I now remove the _legacy folder? (after I made a backup)

For sure.

[Happy2Play 8273]

7 hours ago, PeteMB said:

I had to run the "Scan Metadata Folder" scheduled task for the people images to show up.

Yes as any manual changes to items in the Metadata folder requires this.

[nnylray 14]

How can I get this plugin to run on my old collection .xml files?  I installed Emby Server fresh, and copied over my old collections to the same folder that they were in.  When I run the plugin, it doesn't seem to do anything.

[pünktchen 1258]

6 hours ago, nnylray said:

When I run the plugin, it doesn't seem to do anything.

It depends on the options you've chosen and the type of collection you are trying to restore, eg. movie or music albums...

[nnylray 14]

They are movies and are under data\collections, where they resided in my old build.

[pünktchen 1258]

2 hours ago, nnylray said:

They are movies and are under data\collections, where they resided in my old build.

And you are using exactly the same folders in your library that you have used in the past? That's mandatory!
I've never tried such a case. The plugin was created to run against an updated Emby install, not a new one. But in theory it should work with "Migrate Collections" and "Try to recreate missing collections from boxset folders" checked. You could send me the Emby log to take a look.

[nnylray 14]

Yeah it was the same folder, but I decided to just make a script to update all the .nfo files.  I'm good.  Thank you.

[TallBoiDez 8]

this plugin helped restore some of my actor pictures, but the issue I've had with the people back legacy (a-z folder) is that it doesn't back up of my actors.

[pünktchen 1258]

2 hours ago, TallBoiDez said:

this plugin helped restore some of my actor pictures

Why only some?

2 hours ago, TallBoiDez said:

the issue I've had with the people back legacy (a-z folder) is that it doesn't back up of my actors

There's no need to backup people to legacy folders anymore because the current Emby version does store all actors with their names under metadata\people and not like in previous versions under metadata\library\somecrypticfolder.

[TallBoiDez 8]

1 hour ago, pünktchen said:

Why only some?

There's no need to backup people to legacy folders anymore because the current Emby version does store all actors with their names under metadata\people and not like in previous versions under metadata\library\somecrypticfolder.

Idk why it only back up some people, but I've had better results with backing up to actors folders, so I'll just till the people back up plugin has been updated

[TallBoiDez 8]

On 5/26/2021 at 5:45 PM, pünktchen said:

I've updated the plugin in the start post.

what plugin was updated @pünktchen

[pünktchen 1258]

What is this thread about?