People folder issues

[MrAudio 5]

I updated to the current Emby version a couple of weeks ago, and since them, Emby has started writing to my People folder again - behavior it hasn't done in many months. But not only is it not creating folders under the alphabet folders (for example, "Adrian Young" was under \\servername\Metadata\people\A\Adrian Young, but now the "Adrian Young" folder is being created in the root of the People directory, which will soon become a problem of having too many folders in a single directory), but it's also appending the TMDB ID number (\\servername\metadata\people\Adrian Young-tmdb-181460). That's not a big problem, per se, in fact, it could be a valuable thing, but it's inconsistent with what came before.

Also, I know that I have tons of "People" and other metadata that is in random numbered folders under the library folder that I can't easily get to, because there's no way to know where a given person's folder is, as this has been Emby's behavior for the last couple of years, and I'm happy to see it start to use the proper folders again, but can we expect this to continue, and can we expect Emby to start using the letter folders so that it isn't dumping 100,000 "people" into the same root folder?

[Happy2Play 8307]

Yes People changes over the years has been chaotic.

But \metadata\people\name-providerid is the current method.

You could look at the Migration plugin to consolidate, but in the end you may still end up with bloat as emby converts name folders to name-providerid folder as the name folder is not cleaned up.

[MrAudio 5]

But not having the letter folders is asking for serious problems. File systems don't like tens of thousands of folders under the same folder. We got this fixed before; I'm not sure why it was broken again.

[Luke 37125]

On 8/20/2021 at 10:51 PM, MrAudio said:

But not having the letter folders is asking for serious problems. File systems don't like tens of thousands of folders under the same folder. We got this fixed before; I'm not sure why it was broken again.

The difference is that older versions of the server would mass create tens of thousands of people folders up front, thus making it a requirement. Nowadays it is only on demand, so there's not going to be anywhere near as many. But I suppose if people can tolerate this changing again, we could go back to the letter subfolders.

[MrAudio 5]

Just as background, I've got 8 years/several thousand hours worth of carefully curated pictures (lots of them are not Hollywood celebs), and I'm sure I'm not the only one (I've been using Emby for a LONG time). Thousands of them are now "missing" because they're no longer linking to the right place somehow, because of these changes. If we have to go through the pain to fix all of them, manually, one-at-a-time, then let's at least have the letter subfolders so that we have a sustainable design long-term.

It used to be Emby's philosophy that local artwork was always the priority, with artwork being fetched only when not already locally available, but that philosophy was changed for People a couple of years ago by moving them into the random-numbered folders, breaking a bunch of pics and forcing us into the UI to fix them one-at-a-time, saved to who-knows where, so I'm happy to see that it's back to using the People folder again, where we can find the files and folders, but it would be even better if Emby would look there by default and use those files if they exist, before attempting to pull in new ones from someplace else. The idea being that we wouldn't have to manually go into Emby's UI and click in 4 levels deep to replace a picture - we could just put it in the proper folder in the proper place and it would show up (Emby used to work exactly that way). It's a good idea to append the TMDB ID to allow duplicates of names, and it would be even better if Emby appended these IDs onto existing folders that lacked them (but that's a lesser priority). Either way, if I create a properly-named People folder and drop a "folder.jpg" in it, Emby should automatically (by rescanning the database/people database) pick it up and use that as the image for that person, without having to go into the UI and manually make changes. That would save lots and lots of hours per year for many of us.

I'm not sure I understand the behavior that would make people folders "on demand", but the whole point of having local images is both so that I can control those images, and so that I'm not having to pull them from some other online database each time a user is accessing my server - they should all be stored locally on the server and pulled from there (my metadata is on very fast SSD storage for that very reason). And if that's the case, I'm not sure how the server wouldn't need tens of thousands of people folders (at least).

I'm a HUGE fan of Emby and I appreciate all of the work you guys do, but as I'm sure you can understand, losing hours and hours of work because of a server-side change is a big deal, and having efficient workflows is a big deal, especially when you've got a large media collection. Knowing where to find the files (or where to create them so they'll be used), and being able to manipulate those files directly without having to use the UI (which is slower due to having a lot more steps) would make managing this huge database much, much easier for folks like me.

I thank you for your consideration.

Edited August 23, 2021 by MrAudio

[GrimReaper 3313]

You should check out @pünktchen's Migration plugin.

 

[MrAudio 5]

On 8/24/2021 at 3:09 AM, GrimReaper said:

You should check out @pünktchen's Migration plugin.

 

I appreciate that, but before I mess with anything further, it seems to me that we need to use some long-term thinking and have a file structure that's going to work long-term and be documented. We can't be chasing a constantly moving target.

Here's a simple example: very old Emby installs created a "People" directory. Current versions create a "people" directory. For some OSs (Windows), these are the same thing, and in others (Linux), they are different (capitalized vs. lower case). This kind of thing causes big problems and breaks links. I had both a People and a people directory as a result of recent changes, and it broke all my links. IMO, all paths should be all lower case as a standard, and this should be documented, and that will solve this problem going forward.

And as mentioned before, this new setup is going to result in tens of thousands, maybe even hundreds of thousands, of folders under the People folder. This creates all kinds of problems and makes searches and even directory listings slow, and some devices won't handle it at all. We've had this problem before, and it was solved by using letter folders (A-Z, 0-9, etc.) and sorting the people folders under the letter folders. It was a simple solution that worked great - but we're back to dumping all folders back under the one "people" folder. It makes sense to me to bring back the letter folders.

We're now also appending the TMDB ID number to the end of the person's name (\Aaron Smolinski-tmdb-180137), but old folder names lack these, which means more links get broken. We need a standard - a published standard that can be expected to work going forward without any further changes. Appending the TMDB ID is a good idea because you have to have a way of handling duplicate "people" names, and knowing which one goes to which entry. So, fine, let's keep that because it makes sense (though I'm not sure why we need the "TMDB" part - the number alone should be sufficient and makes for smaller and simpler file paths).

I appreciate what pünktchen has done, but until we have a final, fixed, published design, it's utility is limited. We're making changes so there's going to be pain, but let's do it once, all the way right, so we can deal with the pain once and move forward.

[pünktchen 1259]

2 hours ago, MrAudio said:

Appending the TMDB ID is a good idea because you have to have a way of handling duplicate "people" names, and knowing which one goes to which entry. So, fine, let's keep that because it makes sense (though I'm not sure why we need the "TMDB" part - the number alone should be sufficient and makes for smaller and simpler file paths).

Because "IMDB" also works and maybe some day also "TVDB".

[the-dumb1 121]

It doesn't matter what the implementation is.  What matters is that the change in the implementation without an official/announced migration path broke installs.  A plugin ... by someone who probably isn't a part of the dev team ... is what came to the rescue.

 

FYI it's not just the people stuff hats missing.  It's also the user favorites.  Amd before anyone asks, yes, I ran the backup/restore plugin ... more than once.  I can deal with a loss of favorites.  I can't deal with a loss of other metadata.

[blim5001 8]

I for one would like to see it go back to separate alphabetical folders.

As for those of us that would prefer to rely on our own data rather than calling in images on demand, dealing with a 100,000 or more folders in a single directory is a complete pain for most OS's

[Luke 37125]

@Happy2Play what do you think?

[Carlo 4330]

On 9/25/2021 at 2:19 PM, blim5001 said:

I for one would like to see it go back to separate alphabetical folders.

As for those of us that would prefer to rely on our own data rather than calling in images on demand, dealing with a 100,000 or more folders in a single directory is a complete pain for most OS's

Absolutely.  Huge amounts of folders & files in one directory is never a good idea on any OS platform.
The slower the drive the more pronounced this is.

[Happy2Play 8307]

3 hours ago, Luke said:

@Happy2Play what do you think?

I think it should be alphabetical folders also.

[GrimReaper 3313]

One more vote for alphabetical subfolders. 

[Luke 37125]

OK, I'm not going to do a migration, but I can add a hidden config switch to activate this if that's what you guys want. You'll need to move the files manually after activating it and then run the scheduled task to scan the server metadata folder. This can also be default behavior for new installations.

[Carlo 4330]

That will be a smart solution that should work for us!

[MrAudio 5]

Sounds like the best way to go. Having said that, is there any advice on cleaning out the previous metadata location (for the last 2 or so years, "people" have been going into random numbered folders). I'd like to reduce excess drive usage. I'm fine with moving/deleting manually, but is there a better way?

Also, I'd just like to reiterate that capitalization matters (especially because different OSs/file systems treat them differently), so can we standardize on lower case paths?

[Luke 37125]

Yes they'll be lowercase.

[Abobader 2948]

On 10/5/2021 at 5:59 PM, Luke said:

@Happy2Play what do you think?

Yes it good idea as well (since old members like me used that for long time), but do not remove the support for "id" as well, many times actors have same name, so using id still good idea.

Or I came late and you remove the ID support already?

 

[Happy2Play 8307]

12 minutes ago, Abobader said:

Or I came late and you remove the ID support already?

No id folders can't go as that is the only way to have same named people.

[GrimReaper 3313]

4 hours ago, MrAudio said:

Having said that, is there any advice on cleaning out the previous metadata location (for the last 2 or so years, "people" have been going into random numbered folders). I'd like to reduce excess drive usage. I'm fine with moving/deleting manually, but is there a better way?

Migration plugin that was linked few posts above:

On 5/24/2021 at 5:20 PM, pünktchen said:

What happens by selecting database as source?

For every person the plugin finds in the database, a named folder under metadata\people is created and a nfo is written that holds the database metadata.
The images are copied over from library\xx\yyyyyyyyyyyyyyyy to the new people folder structure and the database references for the images will be corrected.
The now no more necessary library\xx\yyyyyyyyyyyyyyyy folder gets deleted.

 

 

[Happy2Play 8307]

On 10/7/2021 at 9:41 PM, Luke said:

Yes they'll be lowercase.

Just confirming these subfolders will be lowercase?

[Luke 37125]

1 minute ago, Happy2Play said:

Just confirming these subfolders will be lowercase?

The letters will be, yes.

[Happy2Play 8307]

Did not have any issues changing "hidden config switch for people letter subfolders", moving images to a-z folders and running "Scan Metadata Folder" task.  People images still displayed.

4.7.0.14

[Luke 37125]

Thanks for the feedback.