Jump to content

XML & NFO Questions


moviefan

Recommended Posts

moviefan

After the recent update to 3.3 wiped my entire library and rebuilt from scratch, I am set back in several very frustrating ways:

 

1) All of the edits to titles that I made directly in Emby have been removed.

 

2) All of the custom ratings that I configured have been removed 

 

3) All of the custom sorting that I configured has been removed

 

4) Emby is still refusing to read XML data for maybe 100 files properly and is just showing whatever it wants.

 

TBH, as 10+ year Emby user, who has spent probably over 100 hours customizing my library, and has donated more than I need to, this is infuriating.  I specifically have always chosen to save local metadata because of this exact fear - that Emby would get jacked and there would be no way of restoring.

 

First and foremost, I do regular backups of all of my system with versioning enabled.  Is there any file which I can replace prior to that upgrade date which would restore the changes that Emby destroyed?

 

Next, is the reason none of this information was saved properly because I am using XML instead of NFO or is all of the data listed above only stored in Emby's database and ready to be wiped at the next rushed software update?

 

If using NFO will somehow fix this, is there some mechanism to batch convert my collection to start using NFO with Emby?

 

Lastly, can anyone explain initial reasoning for this change?  Did it somehow help with server resources or scalability?  Compatibility with Kodi?  Or did you just decide cosmetically that you liked the NFO extension better?  We already converted once from mymovie.xml to movie.xml.  Starting to seem frivolous these changes without any good migration instructions for the longest users and supporters of Emby.

 

Signed,

Very disappointed and frustrated Emby user

Link to comment
Share on other sites

Nfo has been our default format for 3-4 years now. The rationale was to increase compatibility with other programs. For roughly the last 2 years Xml metadata support has been moved into a plugin. You are certainly still welcome to use the xml plugin, but our available time to spend testing it is limited, whereas NFO is tested on pretty much a daily basis.

Link to comment
Share on other sites

As far as xml not being read, it's really hard to speculate after the fact what might have happened. If you can reproduce it again then it will be easier to help you. I know for example that @@Happy2Play is still using xml without these problems.

Link to comment
Share on other sites

moviefan

The XML not being read is still happening.

 

I have approximately 100 titles where the data showing in Emby for things like date added is different than what is in the XML.

 

Not sure what you are asking me to reproduce.  Emby wiping out my DB again?  I dont think Happy2Play experienced the issue that I and several others experienced with the upgrade to 3.3.0.

 

What about all of the other questions regarding it not keeping the changes in local metadata?  I've read you say elsewhere that Emby can't write to XML at all anymore even with the plugin so I am guessing that it has only been storing all of my customizations in its database.

 

Is there any way to restore the database from an old file backup?

Link to comment
Share on other sites

 

 

I've read you say elsewhere that Emby can't write to XML at all anymore even with the plugin so I am guessing that it has only been storing all of my customizations in its database.

 

Well yes this part is correct. The plugin does not have saving support. It's just too much for us to maintain both at the same time. If there's a community member that might like to help out with the xml plugin, then it could be enhanced with new features.

Link to comment
Share on other sites

moviefan

So if you aren't going to truly support XML anymore, then surely you have to provide a way for the longest supported of Emby to switch to the new format without throwing out everything we have.

 

Emby has all of the information to create NFOs correcT?  Why can't I just enable an option to change to start using NFO and have Emby create them using all of the information from its DB?

 

Are the three data points I asked about above written directly to NFO files?  Would having NFO files have helped with any of the customizations that were lost?

 

Again, is there any way to restore my database by using the file before Emby so heartlessly destroyed it?

Link to comment
Share on other sites

AdrianW

This probably isn't going to help - but anyway...

 

I still use xml files stored locally with my video files, and I do customise the content - but I use MetaBrowser, so any customisations are stored in the local xml files rather than in Emby's database.

 

I've had large chunks of my library disappear when a NAS has been offline, and once the NAS is online again a re-scan brings everything back.

Link to comment
Share on other sites

Happy2Play

All customiztions done in Emby can be lost using xml format if there are DB issues, as Emby is xml read only.  So as @AdianW said you have to do all customiztions in another metadata editor if you are going to customize them.
 
 

Emby has all of the information to create NFOs correcT?  Why can't I just enable an option to change to start using NFO and have Emby create them using all of the information from its DB?
 

  
There is no way to transfer your custom metadata from the database to nfo files that I am aware of, as enabling nfo option applies to newly added media.  Sure you could Refresh metadata to create nfo files for existing media but you still loose already customized metadata as it pulls all new metadata.
  
 
Edit - If you want to change to nfo and keep your customization it will be some work on your part.  Assuming you are able to restore the database with customization.
 
Enable Metadata saver "NFO"  (Dashboard-Library-Metadata)
​Edit info on every movie and hit save (this will save whatever is in the database to nfo file)
 
This test was done with xml plugin and "Download artwork and metadata from the internet" disabled

 

 

Are the three data points I asked about above written directly to NFO files?  Would having NFO files have helped with any of the customizations that were lost?

 

Yes, custom Sort, custom Rating are written to nfo.  As for dateadded in my test it was the same as what was in the xml.
 
 

Again, is there any way to restore my database by using the file before Emby so heartlessly destroyed it?

 

 

Only way I could think of is restoring a previous version (not sure on non Windows setups on install locations, on Windows I would restore previous version of Emby-Server Folder, as there were so many changes between 3.2.7.0 and 3.3.0.0.
 
But the question on why the DB didn't survive the 3.3.0.0 upgrade would still be the big question.

 

 

 

Have you tried shutting down the server and restoring previous version of the library.db?

Edited by Happy2Play
Link to comment
Share on other sites

moviefan

All customiztions done in Emby can be lost using xml format if there are DB issues, as Emby is xml read only.  So as @AdianW said you have to do all customiztions in another metadata editor if you are going to customize them.

 

I guess this is the first thing I take issue with.  I get that Emby doesn't want to support two formats.  But first of all, I don't understand why the switch was necessary.  Seems superfluous and just a pain in the ass for the existing user base.  Second, if the switch WAS necessary, Emby should have provided a way for existing users to easily migrate.  One day I downloaded a new plugin with very little explanation as to why and it didn't even mention anything about not being able to write metadata anymore.  The whole transition was done horribly and it gives the impression that Emby doesn't care about its longest supporters.

 

 

Edit - If you want to change to nfo and keep your customization it will be some work on your part.  Assuming you are able to restore the database with customization.

 
Enable Metadata saver "NFO"  (Dashboard-Library-Metadata)
​Edit info on every movie and hit save (this will save whatever is in the database to nfo file)
 
This test was done with xml plugin and "Download artwork and metadata from the internet" disabled

 

Thank you for testing this or finding this information.  But there's no way I could do this.  I have over 4000 movies and 300 TV series.  Just clicking will take hours.  If it is possible to do it one by one why can't a global option to perform this same behavior be enabled so that I can just click once and have Emby go?  That would seem easy enough no?

 

 

Yes, custom Sort, custom Rating are written to nfo.  As for dateadded in my test it was the same as what was in the xml.

 

That is at least good to know.

 

 

But the question on why the DB didn't survive the 3.3.0.0 upgrade would still be the big question.

 

@@Luke appears to have figured this part out at least and issued a fix for 3.3.1.  Several other users reported the problem.  I asked Luke multiple times to pull the release because of the horrible impact it had upon me but he kept waiting for it to happen to more users even after he thought was might have caused it which seemed rather odd.  I understand that mindset for the beta or dev branches but potentially screwing over many users on the main branch and forcing them to rebuild their DB to prove his theory was disheartening.  Basically, I wonder if I should ever update this product again.  Just seems to bring problems and no improvements.  From my perspective there haven't been any improvements to code in the last couple of years except fixing the trailers plugin and fixing the trakt syncing watched status from my library.  I still can't stand the Genre view in the web app that they redesigned to copy Plex.  It looks so stupid having the two movies breadcrumb onto the next line with all of the black space when viewing from a standard 15" laptop monitor.  But apparently the designers think that people who use Emby all have 24"+ desktop screens so that is what this view is optimized for.

 

 

Have you tried shutting down the server and restoring previous version of the library.db?

 

No I haven't tried this.  I've seen Luke mention many times over the years that he doesn't recommend messing with the DB file and if I do this and it doesn't work then Emby will spend another 36 hours rebuilding the DB and my system will be down again.

Link to comment
Share on other sites

moviefan

This probably isn't going to help - but anyway...

 

I still use xml files stored locally with my video files, and I do customise the content - but I use MetaBrowser, so any customisations are stored in the local xml files rather than in Emby's database.

 

I've had large chunks of my library disappear when a NAS has been offline, and once the NAS is online again a re-scan brings everything back.

 

It doesn't really help my situation no but I appreciate the insight.

 

I guess I should have stayed with using MCM or Metabrowser or whatever to manage my metadata but when this feature first came out for Emby it actually worked.  It just managed the xml files and that was it.  Slowly over time they started pulling more and more stuff into their proprietary database and trying to make it hard to manage metadata locally.  Collections are a great example of this.  We are told over and over we shouldn't be doing the legacy collection format.  But the entire reason I have been adverse to switching to using Emby to manage them is exactly what happened here.  Emby releases an update to something and it can screw up all of my work I spent customizing my collections.  But Emby has made the legacy collection format less and less functional over time in favor of this much less stable way of doing it.

 

From my perspective, availability, stability, reliability are the most important aspects of this product.  And I would suspect that most users feel the same way.  People don't care about bells and whistles if core functionality is broken.  

  • Like 2
Link to comment
Share on other sites

We did do this in a way to allow you to migrate. If you had kept the server at default settings over the course of time it would have saved new nfo files.

 

Or, keep the server at default settings, then refresh all titles and you immediately get nfo. But I think what happened is you disabled nfo to try and stick to the older ways of doing things for as long as you possibly could. I'm sure we could have done a better job of making all of this more clear though.

Link to comment
Share on other sites

moviefan

I'm confused.  If migration was supported at some point why isn't it anymore?

 

I never explicitly changed any of those settings regarding NFO versus XML.  I honestly was shocked to see people talking about NFO in the forums one day and saying this was the new standard a while ago.

 

Can't you just enable an option that will do the refresh Happy2Play describes above to allow for a migration?  I don't see how suggesting to your users to manually click on each and every title in their collection, refreshing, and then clicking save is the only option.  Clearly the product is capable of doing this if it can be done by clicking 15,000 times.  Seems like something that would be easy to automate.

 

And yes, these notifications when you change things that only work for new users for the product should be handled better.  I continually feel like you don't care about your existing user base, and are only looking for more users.  I guess people with lifetime subscriptions don't count anymore since they don't bring in new revenue.  But up until I started to feel like I was outcast I would regularly donate money for no reason to you guys because I was happy with the product.  Things that are happening now are making me feel like I just threw my money away for no reason.

Link to comment
Share on other sites

 

 

I'm confused.  If migration was supported at some point why isn't it anymore?

 

It still is, there's just no "migrate" button. If you enable nfo saving, then nfo files will be saved gradually whenever media items get saved. That would be my recommended approach.

Link to comment
Share on other sites

And I'm sorry you feel the way you do, clearly that means this could have been communicated better. We mentioned this along the way both in the community and in the blog announcements for releases.

Link to comment
Share on other sites

abescalamis

I'm confused.  If migration was supported at some point why isn't it anymore?

 

I never explicitly changed any of those settings regarding NFO versus XML.  I honestly was shocked to see people talking about NFO in the forums one day and saying this was the new standard a while ago.

 

Can't you just enable an option that will do the refresh Happy2Play describes above to allow for a migration?  I don't see how suggesting to your users to manually click on each and every title in their collection, refreshing, and then clicking save is the only option.  Clearly the product is capable of doing this if it can be done by clicking 15,000 times.  Seems like something that would be easy to automate.

 

And yes, these notifications when you change things that only work for new users for the product should be handled better.  I continually feel like you don't care about your existing user base, and are only looking for more users.  I guess people with lifetime subscriptions don't count anymore since they don't bring in new revenue.  But up until I started to feel like I was outcast I would regularly donate money for no reason to you guys because I was happy with the product.  Things that are happening now are making me feel like I just threw my money away for no reason.

 

I could help you out, I went to you same thing a few years ago, I was not using emby at the time but then I realize that .NFO was being adopted and was being use more that .xml. after moving to NFO, I noticed more compatibility in different scenarios and less stressful.

 

It is very simple to transfer .xml to .nfo. you will need a combination of two programs for the best result. but firs copy all of the .xml just in case.

 

The two programs you need are Ember media Manager and Media Center Master.

 

The part that will take you time is setting the program Emby Media Manager because that will be the program that you will use to customize the metadata, scrap artwork add plots it will even download the trailers of a movie if you want.

 

the Media Center Master will be the program that will convert the xml to nfo.

 

 

Warning note: DO NOT TOUCH YOUR ENTIRE LIBRARY, JUST COPY ONE FILE OR TWO THEN YOU TEST AND MAKE SURE IT DID WHAT YOU WANTED. BECAUSE THIS PROGRAMS CAN RE-WRITE AND MESS UP YOUR ENTIRE LIBRARY IF NOT USE CORRECTLY.

 

Ember Media Manager is very powerful and can do everything you want, and because of that you need to go to the settings and adapt it to your needs, it will take you sometime to learn how to use it.

 

I could give you more detail instructions if you need, just let me know.

Edited by abescalamis
Link to comment
Share on other sites

One2Go

It doesn't really help my situation no but I appreciate the insight.

 

I guess I should have stayed with using MCM or Metabrowser or whatever to manage my metadata but when this feature first came out for Emby it actually worked.  It just managed the xml files and that was it.  Slowly over time they started pulling more and more stuff into their proprietary database and trying to make it hard to manage metadata locally.  Collections are a great example of this.  We are told over and over we shouldn't be doing the legacy collection format.  But the entire reason I have been adverse to switching to using Emby to manage them is exactly what happened here.  Emby releases an update to something and it can screw up all of my work I spent customizing my collections.  But Emby has made the legacy collection format less and less functional over time in favor of this much less stable way of doing it.

 

From my perspective, availability, stability, reliability are the most important aspects of this product.  And I would suspect that most users feel the same way.  People don't care about bells and whistles if core functionality is broken.  

I am glad this is being discussed here and I am also glad that this has not happened to me yet. Just like you I have spend many, many hours on customizing my media collection and just like you I have a large movie and TV show collection. Coming from the MB 2.6 crowd I wanted to keep the look and feel as close as possible. When I reached a level that I was happy with I stopped updating the server only to find out that changes had taken place that now generate error messages because it no longer could check for new releases. The only way to get to the latest server version was to download it from the website and install it over the present server version. The point is, if you stop updating you may find yourself also in a very difficult situation because internal changes may force you to spend time to update and make correction to your present meta data and setup.

 

I did what Adrian was doing because I was not comfortable with letting Emby handle all my meta data. I use MCM to get all the meta data in xml format, do my own customization of the xml file as well as customizing all the graphics, folder, backdrop, logo and ClearArt. So far I have had no problem with the Emby plugin reading the xml files. But also you may have to accept the fact that what you value highly, quality and customization is no longer a priority for the present users. Home Theater users are a minority now and I venture to say that Bells and Whistles matter, otherwise the Emby product would stagnate and the future would be bleak. Since this is not the case I trust that the creators of Emby chart a course that they see beneficial and we need to adapt or fit into that.

 

Collections. I feel for you here too. My preference is to have a folder structure with the top folder being the collection folder it has a folder.xml file and graphics representative of the collection. The subfolders have the individual folders containing the movies or the Blu-ray discs. My Star Wars collection has a subfolder for each Blu-ray movie disc as well as the Blu-ray Bonus discs all sorted by the SortTitle field. Since I use the Throgsoft themes I want in the video wall all Star Wars related items come under one folder that is sorted in the video wall in alphabetical order.

 

I am sure I could have done things differently possibly even more efficient, pleasing and useful. However I DO NOT want to challenge Murphy's law, when you make major changes in your underlying structure trust me Murphy always wins, this is part of life and human nature. I do not want my media content be Fouled Up Beyond Believe :D

 

I wish you all the best to get your media content back to the place where you will be spending more time enjoying it then fixing it.

Link to comment
Share on other sites

One2Go

the Media Center Master will be the program that will convert the xml to nfo.

 

While MCM can save in different formats would it not be a pain to run it against 4000 movies and 300 TV Shows or is there a way for MCM to read XML files and then save them into the NFO format? But I guess then you would not have perhaps certain tags or features that only exist in NFO and not in XML or vice versa. In addition there is the problem where the meta data either doesn't exist on the Internet or is very basic in content. I enlarged many of the descriptions to make them more informative then what you can scrape of the Internet.

Link to comment
Share on other sites

Happy2Play

I could help you out, I went to you same thing a few years ago, I was not using emby at the time but then I realize that .NFO was being adopted and was being use more that .xml. after moving to NFO, I noticed more compatibility in different scenarios and less stressful.

 

It is very simple to transfer .xml to .nfo. you will need a combination of two programs for the best result. but firs copy all of the .xml just in case.

 

The two programs you need are Ember media Manager and Media Center Master.

 

The part that will take you time is setting the program Emby Media Manager because that will be the program that you will use to customize the metadata, scrap artwork add plots it will even download the trailers of a movie if you want.

 

the Media Center Master will be the program that will convert the xml to nfo.

 

 

Warning note: DO NOT TOUCH YOUR ENTIRE LIBRARY, JUST COPY ONE FILE OR TWO THEN YOU TEST AND MAKE SURE IT DID WHAT YOU WANTED. BECAUSE THIS PROGRAMS CAN RE-WRITE AND MESS UP YOUR ENTIRE LIBRARY IF NOT USE CORRECTLY.

 

Ember Media Manager is very powerful and can do everything you want, and because of that you need to go to the settings and adapt it to your needs, it will take you sometime to learn how to use it.

 

I could give you more detail instructions if you need, just let me know.

 

The problem is not xml to nfo.  It is db to nfo as all customization is in the database. 

Link to comment
Share on other sites

Happy2Play

While MCM can save in different formats would it not be a pain to run it against 4000 movies and 300 TV Shows or is there a way for MCM to read XML files and then save them into the NFO format? But I guess then you would not have perhaps certain tags or features that only exist in NFO and not in XML or vice versa. In addition there is the problem where the meta data either doesn't exist on the Internet or is very basic in content. I enlarged many of the descriptions to make them more informative then what you can scrape of the Internet.

 

Yes it is a matter of enabling XMBC/Kodi nfo and clicking export as xml is MCM's base metadata.

 

This issues is still all customization that was done in Emby only resides in the database not xml metadata.

Link to comment
Share on other sites

abescalamis

While MCM can save in different formats would it not be a pain to run it against 4000 movies and 300 TV Shows or is there a way for MCM to read XML files and then save them into the NFO format? But I guess then you would not have perhaps certain tags or features that only exist in NFO and not in XML or vice versa. In addition there is the problem where the meta data either doesn't exist on the Internet or is very basic in content. I enlarged many of the descriptions to make them more informative then what you can scrape of the Internet.

 

Yes, MCM will read the data in the XML and create a NFO, running it against 4,000 movies and 300 TV shows should not take long because you are not scrapping anything or getting any info you will just create a NFO based on your XML.

 

First add the folders where your movies are stored.

 

Second go to settings and select this

 

5aa9a47936d8b_1.png

 

 

After this go to tools and select the "Create external Metadata for all files"

 

5aa9a53d7fe86_2.png

  • Like 1
Link to comment
Share on other sites

abescalamis

The problem is not xml to nfo.  It is db to nfo as all customization is in the database. 

 

 

I see now, the developer had to choose in between XML or NFO for the db. NFO will alway be prefer choice and will be for a lot of years I used the XML format back then but you have to move on to the times we live in, that is why I switched to NFO years ago.

Link to comment
Share on other sites

One2Go

This issues is still all customization that was done in Emby only resides in the database not xml metadata.

 

Can you give some examples as to customization that was done in Emby and now only resides in Emby's DB.

 

I can think of one item that I may be missing which is the watched status as well as the date that I watched it. Is that kept in NFO files?

 

Thanks for the suggestions about creating NFOs using MCM.

Link to comment
Share on other sites

I can think of one item that I may be missing which is the watched status as well as the date that I watched it. Is that kept in NFO files?

 

No because you can have multiple users on the system.

 

The issue only comes into play if NFO saving was disabled at the time that you made whatever edits or customizations to the metadata in Emby.  If that is the case, then those edits only go into the database and not any files.

Link to comment
Share on other sites

Happy2Play

Can you give some examples as to customization that was done in Emby and now only resides in Emby's DB.

 

I can think of one item that I may be missing which is the watched status as well as the date that I watched it. Is that kept in NFO files?

 

Thanks for the suggestions about creating NFOs using MCM.

 

As ebr said, if you are using xml metadata and edit anything via Emby it is only saved in the database as Emby can not save the edits to xml.  So all customization is lost if you have database issues or rebuild your system.

Edited by Happy2Play
Link to comment
Share on other sites

What you could do is just open the editor for a movie and click the save button. If you have nfo saving enabled, that will generate an nfo.

Link to comment
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now
×
×
  • Create New...