Migration Plugin

[pünktchen 1259]

@pju here's the new version.

[pju 10]

Super, thanks

A question, if I may:
If I uncheck "Migrate people that exist in the database" a lot of people folders, containing only .jpg files, are left behind in the people folder.
So, what does "Migrate people that exist in the database" mean?

To be honest I'm not sure if the left overs are a result of unchecking the database stuff or because the job run a lot faster.

Another observation:
1100+ metadata folders are created before I see the first subfolder.
And when running with  "Migrate people that exist in the database" checked, these subfolders are moved in place into the correct subfolder..

D:\Emby\metadata>dir  people | head & dir people | tail
 Volume in drive D is D-data
 Volume Serial Number is 186E-E129

 Directory of D:\Emby\metadata\people

06-06-2022  21:52    <DIR>          .
06-06-2022  21:52    <DIR>          ..
06-06-2022  21:51    <DIR>          Aage Haugland-tmdb-1155234
06-06-2022  21:52    <DIR>          Aage Winther-Jorgensen-tmdb-579148
06-06-2022  21:52    <DIR>          Aaron Paul-tmdb-84497
06-06-2022  21:51    <DIR>          Xzibit-tmdb-336
06-06-2022  21:52    <DIR>          Yngve Dahlberg-tmdb-934148
06-06-2022  21:51    <DIR>          Yoann Blanc-tmdb-1333330
06-06-2022  21:51    <DIR>          Yvonne Elliman-tmdb-72741
06-06-2022  21:51    <DIR>          Zach Galifianakis-tmdb-58225
06-06-2022  21:51    <DIR>          Ziggy Marley-tmdb-154857
06-06-2022  21:51    <DIR>          Zubin Mehta-tmdb-141318
06-06-2022  21:51    <DIR>          Trjan Ramberg-tmdb-141617
               0 File(s)              0 bytes
            1257 Dir(s)  54.975.623.168 bytes free

D:\Emby\metadata>dir  people | head & dir people | tail
 Volume in drive D is D-data
 Volume Serial Number is 186E-E129

 Directory of D:\Emby\metadata\people

06-06-2022  21:52    <DIR>          .
06-06-2022  21:52    <DIR>          ..
06-06-2022  21:52    <DIR>          a
06-06-2022  21:51    <DIR>          Aage Haugland-tmdb-1155234
06-06-2022  21:52    <DIR>          Aage Winther-Jorgensen-tmdb-579148
06-06-2022  21:51    <DIR>          Yoann Blanc-tmdb-1333330
06-06-2022  21:52    <DIR>          z
06-06-2022  21:51    <DIR>          Zubin Mehta-tmdb-141318
06-06-2022  21:52    <DIR>          +
06-06-2022  21:52    <DIR>          ¢
06-06-2022  21:51    <DIR>          Trjan Ramberg-tmdb-141617
06-06-2022  21:52    <DIR>          o
06-06-2022  21:52    <DIR>          ?
               0 File(s)              0 bytes
             708 Dir(s)  54.974.246.912 bytes free

 

Edited June 6, 2022 by pju

[pünktchen 1259]

3 hours ago, pju said:

So, what does "Migrate people that exist in the database" mean?

It will fetch every person of your library and checks the folders where the person images exist. If the path differs from the new target folder hierarchy, it will copy the images to the new hierarchy and update the database entries. But there was another bug with those letter subfolders that where not possible with Emby 4.6. The truth is i'm coding it blind without any further testing against Emby 4.7 with letter subfolders. You are my tester  So thank you so much!!!
But i'm very positive this version should be the last round.

[tr0nllam 34]

I tested this out and it seems to work well except for when people have the same name. Even though they have different provider IDs, if there are two people with the same name, it won't move them from their old directory.

[pju 10]

16 hours ago, pünktchen said:

It will fetch every person of your library and checks the folders where the person images exist. If the path differs from the new target folder hierarchy, it will copy the images to the new hierarchy and update the database entries. But there was another bug with those letter subfolders that where not possible with Emby 4.6. The truth is i'm coding it blind without any further testing against Emby 4.7 with letter subfolders. You are my tester  So thank you so much!!!
But i'm very positive this version should be the last round.

You're welcome - PM me anytime if I can help.
I owe you big time for this plugin
And as far as I can see, I agree on the last round. Since my legacy data is so old, I cannot have the problem tr0nllam mentions.

[KobayashiM 11]

@pünktchen Is there a download link that's still online? Much appreciated in advance.

I just upgraded Emby to 4.7.6.0 in Unraid because the mobile app suddenly stopped working and told me to upgrade the server. I had no idea I'd lose my collections. 

All of my collections are manual categories that I created (no boxsets) so I have no interest in auto-generating them. I spent a lot of time organizing them and creating my own custom images. It would have been nice to have been given a heads up. Even nicer if there was an official conversion GUI in cases where the upgrade failed to do it.

[pünktchen 1259]

The attachment links from the start post are still working fine for me.

[Happy2Play 8296]

2 minutes ago, pünktchen said:

The attachment links from the start post are still working fine for me.

Was there something different in 0.5.0.0 as first post appears to be 0.2.0.0?

[pünktchen 1259]

Where do you see 0.5.0.0, in a local copy of an older version that you have downloaded?

[KobayashiM 11]

13 hours ago, pünktchen said:

The attachment links from the start post are still working fine for me.

Thanks! Odd, yesterday it was telling me it couldn't be found. Muchos gracias

I ran the plugin to migrate collections only and ticked off 'Try to recreate missing collections from boxset folders" and "write set tags to existing media item nfo where possible". It said it completed the migration successfully but the /metadata/collections folder is still empty. Looked at log and only see this:

2022-09-16 17:06:06.011 Info Migration Plugin: Fixing Collection Database started

2022-09-16 17:06:06.013 Info Migration Plugin: Fixing Collection Database finished

2022-09-16 17:06:06.014 Info Migration Plugin: Migration of Collections started

Edited September 16, 2022 by KobayashiM

[KobayashiM 11]

Ok got it to work by doing the following:

1. Copied one old collection folder from /data/collections to /metadata/collections
2. Removed "[boxset]" from the folder name
3. In Emby, created a collection and named it exactly the same as the folder
4. Configured plugin to 'Migrate Collections' and 'Try to recreate missing collections from boxset folders'
5. Save and Run plugin

I watched as the rest of the old collections were copied over into the new directory, success!

All I have to do now is to tweak the metadata and fix a poster or two which is far and away better than manually rebuilding all of my collections.

Thank you again @pünktchen you're awesome! 

[ulrick65 20]

@pünktchenI am trying to utilize the migration tool to bring my old people folder (with the first letter as a folder, then the people with that letter beneath it...the old way it was done, now back again thank goodness).  However, I am having an issue that I assume is because I am not sure how to set the migration tool properly.

In my metadata/people folder it is all organized in the new way (I am on the current release of Emby).  When I run the tool, I click the check on "Migrate people that exist in a legacy people (A-Z) folder" check box and put my old people folder in path path field, I check the box for "Create subfolders by people name first letter (Emby 4.7 and above)" and the box "Migrate only people that have images".  Nothing else checked.  I then save and click Run Migration.  It runs and fine but:

I moves all the subfolders (a through z, etc.) to a folder under people called _legacy and then only brings in about 29,000 files and folders...but my people folder has over 100,000.  Also, it appears to leave the _legacy folder there with all the stuff that was in it before.  No errors, it finishes with "Migration finished successfully".

Am I setting something wrong here?  What I want is for it to bring in my old people folder and update the pictures.  I would be OK even it even just brought in the ones that didn't already exist.

Any help would be appreciated...I am sure it is me here, but I must be missing something?

Thanks.

EDIT:  I think I got it sorted out.  The _legacy folder only appears to have left items that where either no picture, or they are duplicates.  I am reprocessing now and will see how it works out.  My problem now is that I failed to think about the fact that the metadata folder is in my /var/lib/emby folder and that drive is not that big.  Given the cluster sizing, all these tiny files are likely to fill that disk (which of course is not cool, given that is my OS drive and I am thinking bad stuff is going to happen when it fills up).   

Edited June 21, 2023 by ulrick65

[ulrick65 20]

Well, I thought I had it sorted out...but I had to move the metadata folder due to size.  Everything is in there...the new folder contains the people in the folder, separated by a folder with the first letter and I updated the config to <EnablePeopleLetterSubFolders>true</EnablePeopleLetterSubFolders> ran the "Scan Metadata Folder" task (multiple times now) rescanned the library (multiple times) and none of the cast images are showing up at all.  I waited for last 18 hours and still nothing.

Am I missing something else?

Thanks.

[ulrick65 20]

On 6/21/2023 at 7:08 PM, ulrick65 said:

Well, I thought I had it sorted out...but I had to move the metadata folder due to size.  Everything is in there...the new folder contains the people in the folder, separated by a folder with the first letter and I updated the config to <EnablePeopleLetterSubFolders>true</EnablePeopleLetterSubFolders> ran the "Scan Metadata Folder" task (multiple times now) rescanned the library (multiple times) and none of the cast images are showing up at all.  I waited for last 18 hours and still nothing.

Am I missing something else?

Thanks.

Anyone have any suggestions on how to get the People pictures to show up again since I relocated my Metadata folder?  Everything is working OK, except no pictures show up even though they are in the People folder.  Do I need to rerun the Migration tool again?  I am afraid to mess something up by doing it again if there is something simpler that I am just missing. 

Thanks.

[pünktchen 1259]

I have a hard time to understand what was your people metadata structure before running the plugin and when did you change the system config file.

[pünktchen 1259]

1 hour ago, ulrick65 said:

Anyone have any suggestions on how to get the People pictures to show up again since I relocated my Metadata folder?

Did the pictures show up after running the plugin, before relocating the metadata folder?

[ulrick65 20]

6 hours ago, pünktchen said:

Did the pictures show up after running the plugin, before relocating the metadata folder?

Yes.  Of course I did not check all of them....but check many and they were there.  But once I moved the metadata folder, they no longer show up.

Edit:  None show up...I just wanted to point that out.  Also, I went into some and went to edit the picture and then that one shows up.  It doesn't download them or anything, I checked the file date and it is still the same picture that was already in there.  That is why I was thinking maybe I have to run the migration tool again to scan them or something...but I thought that running the Metadata folder scanner would do that, but it doesn't.

Thanks.

Edited June 25, 2023 by ulrick65

[pünktchen 1259]

7 hours ago, ulrick65 said:

But once I moved the metadata folder, they no longer show up.

Okay, i've tested it myself. Moved metadata folder to an other location, updated metadata location setting in Emby, scanned library, ran scan metadata folder task -> NOTHING!
It's not only people that do not show up anymore, also collection images, genre images and so on. After searching the forum it seems, that the metadata path setting in Emby only applies to new content. That means a user has to keep his "old" metadata folder in its original location and gets new metadata stored in the new location path. What a big design flaw by the Emby devs!!! The only thing you could do is to manually update the Emby database with some sql queries. But that's not my topic. Others know more about this. @Carlo

[Luke 37109]

3 hours ago, pünktchen said:

Okay, i've tested it myself. Moved metadata folder to an other location, updated metadata location setting in Emby, scanned library, ran scan metadata folder task -> NOTHING!
It's not only people that do not show up anymore, also collection images, genre images and so on. After searching the forum it seems, that the metadata path setting in Emby only applies to new content. That means a user has to keep his "old" metadata folder in its original location and gets new metadata stored in the new location path. What a big design flaw by the Emby devs!!! The only thing you could do is to manually update the Emby database with some sql queries. But that's not my topic. Others know more about this. @Carlo

No, this isn't true. All path assignments under the metadata folder are virtual, so you can move the folder anytime you like and everything will just work.

What you should do is change the setting, and then run the scan metadata folder scheduled task immediately after that.

[ulrick65 20]

19 minutes ago, Luke said:

No, this isn't true. All path assignments under the metadata folder are virtual, so you can move the folder anytime you like and everything will just work.

What you should do is change the setting, and then run the scan metadata folder scheduled task immediately after that.

@Luke Thats what I have done many times, and library scans, etc.  Still nothing shows up.

[Luke 37109]

32 minutes ago, ulrick65 said:

@Luke Thats what I have done many times, and library scans, etc.  Still nothing shows up.

What do you mean nothing shows up?

[pünktchen 1259]

1 hour ago, Luke said:

No, this isn't true. All path assignments under the metadata folder are virtual, so you can move the folder anytime you like and everything will just work.

What you should do is change the setting, and then run the scan metadata folder scheduled task immediately after that.

That's not what i've experienced. After moving the metadata folder, non of the images that reside inside the metadata folder are used anymore: people images, collection images, genre images and so on. They do not show up anymore anywhere!

[pünktchen 1259]

Ha, i've found the issue:
We cannot point directly to the "new" metadata folder, we have to use the folder above, although the original path directly points to "metadata". What a mess @Luke!!!

Original folder location:        E:\Emby-Server\programdata\metadata
Original path in Emby's UI:   E:\Emby-Server\programdata\metadata

New folder location:              F:\EmbyData\metadata
New path in Emby's UI:         F:\EmbyData\metadata -> Will not work!!!
New path in Emby's UI:         F:\EmbyData -> Works!!!

[ulrick65 20]

2 hours ago, pünktchen said:

Ha, i've found the issue:
We cannot point directly to the "new" metadata folder, we have to use the folder above, although the original path directly points to "metadata". What a mess @Luke!!!

Original folder location:        E:\Emby-Server\programdata\metadata
Original path in Emby's UI:   E:\Emby-Server\programdata\metadata

New folder location:              F:\EmbyData\metadata
New path in Emby's UI:         F:\EmbyData\metadata -> Will not work!!!
New path in Emby's UI:         F:\EmbyData -> Works!!!

You're right!  Changed the path as you show above (point to the parent folder of Metadata) and it works!  Ran the scheduled scan...and immediately all the pictures are there.

For example on my server it is:  /disk3/emby/metadata/people/a  (b, c, d, etc.)

When I have /disk3/emby/metadata in the Library Advanced, nothing shows up.  Changed it to /disk3/emby and now they are all there.

I don't know if this causes other issues, but it does work!  

[ulrick65 20]

@LukeI am not sure if there are any plans to change this behavior, but it would be good to know about it in advance before it is changed.  Currently, everything works fine as long as the path points to the parent (metadata) folder and a "people" folder is located within it as @pünktchendiscovered above.

Just wondering...no problem as it stands now at all.

Thanks.