Sort metadata has been mangled :-(

[jmos1277 9]

While editing some metadata via the web view, I noticed many (but not all) of the "Sort title" fields have been modified by Emby.  For example, I had the sort titles for my Batman movies set to "Batman 1", "Batman 2", "Batman 3", etc.  However, they are now set to "batman 0000000001", "batman 0000000002", etc.  

I looked at my .NFO files and they STILL contain the correct "Sort title" ... phew.
However, when I looked into the library.db file, I see the "Sort title" fields are modified in there.

Was this an intentional change?  I really don't like it -- particularly because it seems inconsistent.  Some of my movies have been changed others have not been.

If this was unintentional, is there an easy way to grab my CORRECT "Sort title" from my .NFO files and rewrite "Sort title" fields in the database?

 

[jmos1277 9]

Here are a couple of screenshots from my database.  Notice that even studio names like "1821 Pictures" have had the "Sort title" field altered.

 

[Luke 42078]

Hi, a long time ago we used to do that in order to sort numbers within the titles. Nowadays we have better techniques available.

What you can do for existing ones that are impacted is reset the value as desired, and optionally undo the lock on the field if the value you enter matches the original title.

[Happy2Play 9780]

Devs will have to comment but not sure if this is something that has changed over time.

Does refreshing an item that you see this issue on correct it?

[jmos1277 9]

4 minutes ago, Luke said:

Hi, a long time ago we used to do that in order to sort numbers within the titles. Nowadays we have better techniques available.

What you can do for existing ones that are impacted is reset the value as desired, and optionally undo the lock on the field if the value you enter matches the original title.

Luke, thanks for the response.  Do you have a recommendation on how to do this efficiently?  I have thousands of titles, some of which are affected, some of which are not.  The slow way would be to open each via the web view, and enter a new value into the text box.  I REALLY don't want to do this for thousands of titles.

I'm a seasoned at writing terminals scripts in just about any language ... could I just write a script that greps the 'Sort Title" from my .NFO files and write it to the DB?

Of course, this doesn't fix sort titles for the various other data types like studios.

1) What are the implications of me completely resetting my current DB?
2) How might I go about doing that and would a simple scan rebuild everything from my existing .NFO files?
3) Lastly, I have about 300 Collections.  If I reset my entire database I'll end up losing all of my collections.  Is there an easy way to backup my collections so I don't have to manually recreate them?  Some of my collections are custom collections, so relying on TMBD to rebuild them automatically won't really work.

 

[jmos1277 9]

24 minutes ago, Happy2Play said:

Does refreshing an item that you see this issue on correct it?

Yes, if I first unlock an item (I always keep my metadata locked) and then refresh the metadata is set back to the default sort title.
Oftentimes, however, I alter the default sort title.  For example, instead of keeping "Batman Returns" as my sort title I set the sort title to "Batman 2".

So, refreshing all items could do some of the heavy lifting here, but I'll still have a lot of manual adjustments to make afterwards.  

Edited November 30, 2022 by jmos1277

[Happy2Play 9780]

7 minutes ago, jmos1277 said:

1) What are the implications of me completely resetting my current DB?

2) How might I go about doing that and would a simple scan rebuild everything from my existing .NFO files?

Since you save metadata with media should be not issue beside allowing it to rebuild.

Note Library options will depict how long that takes as I normally disable all providers and so the initial scan is quicker.  Then reenable afterwords.  But can be tedious if you have a lot of libraries.

7 minutes ago, jmos1277 said:

3) Lastly, I have about 300 Collections.  If I reset my entire database I'll end up losing all of my collections.  Is there an easy way to backup my collections so I don't have to manually recreate them?  Some of my collections are custom collections, so relying on TMBD to rebuild them automatically won't really work.

Collections are built off of item metadata ie the <set> field in each items metadata.  So it shouldn't be an issue.

But have you tested Refreshing some affected items?  Could be as easy as Touching every media file to force Emby to reread existing metadata.

[jmos1277 9]

1 minute ago, Happy2Play said:

Since you save metadata with media should be not issue beside allowing it to rebuild.

Note Library options will depict how long that takes as I normally disable all providers and so the initial scan is quicker.  Then reenable afterwords.  But can be tedious if you have a lot of libraries.

Collections are built off of item metadata ie the <set> field in each items metadata.  So it shouldn't be an issue.

But have you tested Refreshing some affected items?  Could be as easy as Touching every media file to force Emby to reread existing metadata.

Hmm ... great suggestion.  I might go try this as it should only take a few seconds to find/touch all .NFO files.  I'll probably do a small test first.

I'd still like to clean up other MediaItems in my database.  For example, the studio fields are all mangled as noted in my second post above.  Is there a quick way to clean these up?

 

[Happy2Play 9780]

8 minutes ago, jmos1277 said:

Hmm ... great suggestion.  I might go try this as it should only take a few seconds to find/touch all .NFO files.  I'll probably do a small test first.

Not positive if you have to touch nfo or actual media file.

9 minutes ago, jmos1277 said:

I'd still like to clean up other MediaItems in my database.  For example, the studio fields are all mangled as noted in my second post above.  Is there a quick way to clean these up?

Guessing here but make a backup of your database and possibly clear the SortName column, start Emby and do a library scan.  Or clear by type 29 (studios).  You could test on a couple items.

But will guess you are talking about this from an old database.

[jmos1277 9]

14 minutes ago, jmos1277 said:

Hmm ... great suggestion.  I might go try this as it should only take a few seconds to find/touch all .NFO files.  I'll probably do a small test first.

 

Ok ... I'm glad I did a small test because my first test didn't go as planned.

Here's what I just tried and the results:

1)  I unlocked the metadata for a couple of movies.
2)  I ran 'touch' to update the timestamps of the .NFO files for those movies.
3)  I scanned my media library

RESULTS:
Emby overwrote my correct/preferred 'Sort title' in the .NFO file with the mangled 'Sort title' from the database.
Definitely not the desired result as I obviously want the .NFO file to take priority over the database.

[Happy2Play 9780]

Just now, jmos1277 said:

Ok ... I'm glad I did a small test because my first test didn't go as planned.

Here's what I just tried and the results:

1)  I unlocked the metadata for a couple of movies.
2)  I ran 'touch' to update the timestamps of the .NFO files for those movies.
3)  I scanned my media library

RESULTS:
Emby overwrote my correct/preferred 'Sort title' in the .NFO file with the mangled 'Sort title' from the database.
Definitely not the desired result as I obviously want the .NFO file to take priority over the database.

Yes as the internal code matches Title to SortName minus Remove Words (A, The, ect..).

[jmos1277 9]

2 minutes ago, Happy2Play said:

But will guess you are talking about this from an old database.

 

Yes, exactly that.  Many studio names are mangled as in your image.
It's not yet clear to me the extent to which numbers were modified in my database.  I'm leaning toward making a backup of my entire Emby installation, blowing it away and starting from scratch.  It may be the shortest path to victory.

[Happy2Play 9780]

1 minute ago, jmos1277 said:

Yes, exactly that.  Many studio names are mangled as in your image.
It's not yet clear to me the extent to which numbers were modified in my database.  I'm leaning toward making a backup of my entire Emby installation, blowing it away and starting from scratch.  It may be the shortest path to victory.

Not knowing your platform but a full backup would be the entire Emby programdata folder.

 

Does touching the file with it locked make any difference?

[jmos1277 9]

27 minutes ago, Happy2Play said:

Does touching the file with it locked make any difference?

If I touch the .nfo and the .mkv file, then run a scan while the metadata is still locked I see no change.
The sort title in the database remains the same mangled name.

[Happy2Play 9780]

Sound like the only options is rebuilding as existing imported data is not changing.

So removing an item, doing a library scan and readding the item appears correctly?

Other than that, it is just testing, ensuring you have a backup and play with the database possibly clearing SortName.

[jmos1277 9]

4 minutes ago, Happy2Play said:

So removing an item, doing a library scan and readding the item appears correctly?

 

Yes, removing and re-adding an item seems to work.  I think my best bet is to rebuild this database from scratch and see how it goes.
I appreciate all your suggestions.  Of course, I'll back up everything before I blow it all away.

I'd like to keep all my configuration settings, so I'll probably export those and then re-import after I reset everything.

This sounds like a weekend project. lol

Edited November 30, 2022 by jmos1277

[softworkz 5066]

@jmos1277 - for getting a better understanding of metadata and providers, you may take a look at this:

 

Just giving it a read might provide some insights, even without installing the plugin (needs the beta server right now).

 

3 hours ago, jmos1277 said:

RESULTS:
Emby overwrote my correct/preferred 'Sort title' in the .NFO file with the mangled 'Sort title' from the database.
Definitely not the desired result as I obviously want the .NFO file to take priority over the database.

Regarding local metadata providers (like Nfo): you can choose them as "Savers" and as "Loaders". But it's not meant to be used in a way that you enable both - savers and loaders - and that would change your data storage from the Emby db to the file system. There might be some use cases to enable both, but normally, you need to see it as a flow of data that goes into a certain direction. Often it's used for sharing data among several applications where one does all the metadata retrieval (or there you might be doing all your manual edits), and the other are reading the data only. That means, for that one application you would activate the savers but not the loaders and in the other applications, you would activate loaders only.

Edited November 30, 2022 by softworkz

[jmos1277 9]

@softworkz, thanks for the link.  I'll give that a thorough read sometime tomorrow before messing around with a full reset of my Emby.

I'm not 100% clear on how I might take advantage of the "savers" vs. "loaders" yet but maybe after reading through the plugin documentation things might make more sense.  For now, I just finished grep'ing all of my .nfo files to ensure that they still contain my desired 'sorttitle' data.  I'm happy to say, all of my .nfo data is intact.

[softworkz 5066]

24 minutes ago, jmos1277 said:

@softworkz, thanks for the link.  I'll give that a thorough read sometime tomorrow before messing around with a full reset of my Emby.

I'm not 100% clear on how I might take advantage of the "savers" vs. "loaders" yet but maybe after reading through the plugin documentation things might make more sense.  For now, I just finished grep'ing all of my .nfo files to ensure that they still contain my desired 'sorttitle' data.  I'm happy to say, all of my .nfo data is intact.

1 minute ago, jmos1277 said:

@softworkz, thanks for the link.  I'll give that a thorough read sometime tomorrow before messing around with a full reset of my Emby.

I'm not 100% clear on how I might take advantage of the "savers" vs. "loaders" yet but maybe after reading through the plugin documentation things might make more sense.  For now, I just finished grep'ing all of my .nfo files to ensure that they still contain my desired 'sorttitle' data.  I'm happy to say, all of my .nfo data is intact.

Okay, then you will definitely want to disable all savers. This will make sure that your nfo data won't be overwritten.

Now, if your intention would be to get all of the nfo data into Emby, you'D do like this:

• In libary options,
  • disable all remote (and other) metadata providers
  • except the nfo loaders
  • disable all savers
• Next, you might want to unlock your locked fields/properties - in case you want them to be updated
  An SQL statement might be the best way to do this - after opening t

• Finally, you perform a metadata refresh

• Once you have all the right data back in Emby Server, you might decide that from then on, you want Emby to be in charge again, to handle metadat.
  Then you would

  • Disable the loaders
    and

  • enable the savers

If you do so, the new data that Emby will discover will be written to the nfo files.

[softworkz 5066]

One more note: You shouldn't see the Emby database as a target for storing your carefully hand-edited data. It's not the right place for that and it's not made for that.
You can make a little edit here and there and lock the edited fields, but that's more meant for occasional workarounds than a way to curate a media library.

The power of Emby is that you can throw in media files of all kinds and it automatically identifies these and retrieves metadata from providers to provide a rich experience.
But unless you lock any field, those are not a safe storage for your edits.

For example:

• You change your preferred language for a library to a different language
• Then, you run a metadata refresh
  => Your items will show in a different language
  => But not only "show" - the data in the Emby DB has been overwritten 

You can also choose the providers, specify their execution order, etc - with the same effect: the values in the Emby DB will be overwritten each time.

If you want to make edits, there are better options:

 In the Media File

There are tools which allow you to edit metadata in media files and Emby has an option to prioritize that data over others. This method is often used for audio files and related items.l

In Local separate Data

Like .nfo files. 

Directly with the providers

In most cases where Emby has loaded metadata for an item, it will also show links directly to the website of the metadata provider. If you see any errors or missing bits, it's best to create an account on this sites and submi1 you change there.

This way, other users will benefit from the improerd translations well.

Edited November 30, 2022 by softworkz

[jmos1277 9]

18 hours ago, softworkz said:

Okay, then you will definitely want to disable all savers. This will make sure that your nfo data won't be overwritten.

Now, if your intention would be to get all of the nfo data into Emby, you'D do like this:

• In libary options,
  • disable all remote (and other) metadata providers
  • except the nfo loaders
  • disable all savers
• Next, you might want to unlock your locked fields/properties - in case you want them to be updated
  An SQL statement might be the best way to do this - after opening t

• Finally, you perform a metadata refresh

• Once you have all the right data back in Emby Server, you might decide that from then on, you want Emby to be in charge again, to handle metadat.
  Then you would

  • Disable the loaders
    and

  • enable the savers

If you do so, the new data that Emby will discover will be written to the nfo files.

 

Wow @softworkz, excellent response.  Thanks for all the details and the thorough recommendation for a plan-of-attack.

I'm left with a couple of questions.

1) When I edit metadata in emby's webview, there are several locks (a lock on the "Title", the "Original title", "Sort title" fields, etc.). Additionally, there is a checkbox all the way at the bottom to "Lock this item to prevent future changes".  The bottommost checkbox sets the "lockdata" field in my .NFO files to true.  I ALWAYS set this checkbox for my Movies.
  a) What are the other field-specific locks for?  
  b) Are they setting something in the database, but not in my .NFO file?
  c) Do the field-specific locks inherit the "lock" status from the bottommost checkbox?

2) I don't have any "loader" settings in my library options.  However, I do have "Metadata Readers".  I assume this is the setting you refer to in your response above, is that correct?

3) Why do you suggest that I unlock my locked fields/properties?  If I'm pulling all metadata from my NFO files what good would unlocking fields do?  Would emby attempt to fill in missing metadata?  Where would emby get data from if I previously disabled all remote metadata providers?  It's not clear to me what affect unlocking my locked fields would have.

Thanks again!

 

As a side note (and to address your second post above) ... when I import new media into my library, I let emby populate the metadata using various metadata providers.  I examine and adjust that metadata as necessary and then lock the item to prevent future changes.  I utilize the NFO Savers to ensure that all my metadata is backed up along with my media.  Once upon a time I evaluated Plex and compared it to emby.  I MUCH preferred emby specifically because it gave me much more control over my metadata/artwork.  So, yeah, metadata providers are great.  But, being able to fine tune my artwork and metadata is one of my favorite features of emby!

 

Edited December 1, 2022 by jmos1277

[Happy2Play 9780]

SortName has to be locked as Emby by default will always utilize title.  You can see this by trying to edit sortname without locking it.

[jmos1277 9]

Just now, Happy2Play said:

SortName has to be locked as Emby by default will always utilize title.  You can see this by trying to edit sortname without locking it.

Interesting.  I have noticed that anytime I make a custom entry into the "Sort title" field, emby automatically locks the field.
Great info ... thanks!

[softworkz 5066]

45 minutes ago, jmos1277 said:

1) When I edit metadata in emby's webview, there are several locks (a lock on the "Title", the "Original title", "Sort title" fields, etc.). Additionally, there is a checkbox all the way at the bottom to "Lock this item to prevent future changes".  The bottommost checkbox sets the "lockdata" field in my .NFO files to true.  I ALWAYS set this checkbox for my Movies.

You shouldn't do this. Otherwise you won't get updated metadata. It's only meant to "protect" manual edits.

45 minutes ago, jmos1277 said:

a) What are the other field-specific locks for?  
  b) Are they setting something in the database, but not in my .NFO file?
  c) Do the field-specific locks inherit the "lock" status from the bottommost checkbox?

a) The general lock locks everything, the individual locks are locking individual fields only
b) All are setting something in the database - ONLY. Just when you have a saver enabled, things from the DB will be written to the NFO. 
c). When you set the general lock, you don't need to set any individual locks.
     Exception: It seems that it doesn't affect the sort name

45 minutes ago, jmos1277 said:

2) I don't have any "loader" settings in my library options.  However, I do have "Metadata Readers".  I assume this is the setting you refer to in your response above, is that correct?

Apologies, by "loaders" I meant "readers" in fact, yes.

45 minutes ago, jmos1277 said:

3) Why do you suggest that I unlock my locked fields/properties?  

I don't know what you want to do. Which data is supposed to be authoritative? Where is the source? Emby? Nfo? Or other metadata providers?

45 minutes ago, jmos1277 said:

 If I'm pulling all metadata from my NFO files what good would unlocking fields do?   It's not clear to me what affect unlocking my locked fields would have.

When fields are locked, they don't get the data from the NFO files. So when you say you your data source is NFO files, you MUST NOT lock the fields in Emby, because Emby won't pull the metadata from Nfo files for these fields, when they are locked.

45 minutes ago, jmos1277 said:

Would emby attempt to fill in missing metadata?  Where would emby get data from if I previously disabled all remote metadata providers?

Nfo files are just like another metadata provider. When you haven't enabled any remote providers, then Emby will only pull the data from the Nfo files, but this doesn't work when you lock the fields.

And when you say your metadata comes from the Nfo files, why do you enable the Nfo savers then? It doesn't make sense, for Emby to write to those files when these are the source of your metadata. Emby could just screw your data.

Edited December 1, 2022 by softworkz

[jmos1277 9]

3 minutes ago, softworkz said:

And when you say your metadata comes from the Nfo files, why do you enable the Nfo savers then? It doesn't make sense, for Emby to write to those files when these are the source of your metadata. Emby could just screw your data.

@softworkz, thanks again for your detailed response.  It definitely cleared up some of my questions.

I wanted to answer this question to clarify how I personally use emby and why I use NFO savers.  When I add a new MediaItem to my library I do use emby's metadata providers (e.g. TMDB, IMDB, TVDB, etc.) to retrieve metadata from remote services.  I use the NFO savers simply to create a backup of that metadata.  Occasionally, I will make customizations to the metadata ... like creating a custom "Sort title" ... or adding some custom text to the "Overview".  These changes are all written to the database AND backed up into my NFO files.  I also set the global "lock" checkbox.  This is also reflected in my NFO file as "<lockdata>true</lockdata>".  After I curate my metadata, I don't want emby to change it any further ... hence the lock.  Emby honors the "lockdata" field and has never altered my locked NFO files during any automated process.  However, I can still updated the NFO files by editing metadata in emby.  When I do ... emby will once again write those changes to the database and to my backup NFO files.

Now that I want to reset my emby database ... I'm VERY glad that I've backed up all my metadata in NFO files.  As per your suggestion, I can use use my NFO files containing my customized metadata to repopulate my database.  I have many thousands of MediaItems and I REALLY don't want to download fresh metadata from remote services that I once again will have to curate/update.

Once everything is back up and running, I can go back to retrieving metadata for new MediaItems from remote services.

Honestly, I thought everyone backed up their metadata into NFO files.  Backups are GOOD!

My process is probably a carryover from when I once used TheMetaBrowser to manage my metadata.  Back then, I had emby (MediaBrowser) just reading NFO files.  However, TheMetaBrowser project died right around the time emby developed a fully-fledged and quite capable built-in metadata editor.  But still ... backups are good!

I also save my MediaItem artwork in the MediaItem folder ... shhh!!