Metadata Questions

[NGNear 2]

Hey guys,

So, a couple of questions regarding metadata structure for my movie collection - I'll leave my TV and anime collections to one side for now.

First problem: Emby's movie scraper doesn't seem to be working for me.  By "not working", I mean that I can select a movie, even identify it correctly (I get an IMDB link which, when followed, leads to the correct movie), but when I "refresh metadata", regardless of the option I select, no data is shown from online sources.  Media info about the file (Video, Audio) is populated and written to the NFO (I have the Write Metadata to Media Folders option active), but nothing from "offboard".  The folder image is replaced with what appears to be a screengrab from the file, and no images appear to be downloaded either.  While the files are kept on a Linux machine, I'm sure the permissions are set correctly - the media info is written to the file without apparent problems.  This problem occurs with items with no existing metadata and with items with some existant data (see below).

At the moment, I have The Open Movie Database (1st) and TheMovieDb (2nd) selected as Metadata Downloaders, and Nfo as Metadata Saver.

 

Question: Any idea where to start troubleshooting this one?

 

Anyway, I'll be the first to admit that my collection has gained some... cruft... over the years - I've used a few metadata programs through the years, including MetaBrowser, MCM, MediaElch and a few others.  I'll not rehash the arguments over the death of movie/mymovies.xml direct support, I've got no problem with nfo files.  Here's my next problem, I've been adding my own tags to the metadata files over the years, in order to allow a few different scripts to run in the background - mostly backup stuff.  So I'd prefer not to just nuke my existing metdata files and allow Emby to replace them (and at the moment, that's not even an option, as you can see above).

The solution I came up with was to use hardlinks to link my movie.xml, mymovies.xml and <<moviename>>.nfo files, and allow Emby to read the now-existant nfo file and add to it.  Problem is, the xml "source" files have a range of keys that I think are now deprecated, and I'd like to figure out what Emby uses to create what it displays, and how they relate to older keys.
Examples: Previously used "Description" and "Overview" keys now seem to be "Plot"; "Tagline" now seems to be "Outline", etc

Questions: Does Emby honor existing keys when it opens an nfo file for a given movie? (I assume it does).  Is there a list of keys used/populated by Emby, particularly on the Movie item homepage?  Are there relational lists for translating an older key/copying it to what Emby uses, and heirarchial structure for which keys Emby copies from (I've seen my comments end up in a new "plot" key without me putting them there) or ignores and downloads as it sets up the nfo to it's requirements?

Next up, as part of the information I've been adding to the files is additional info in the main Overview/Plot paragraph(s) on the item page.  I've found that adding HTML-style "<p>" tags allows me to break the text into more readable pieces rather than a single run-on sentence, but this seems to work on a surprisingly inconsistent basis - some xml/nfo files with an existant Overview key including a couple of <p> tags correctly format the text, others with the same xml layout do not - when I open the xml/nfo file, the <p> tags I added in Notepad++ have been stripped out of the Overview key in the file.  I can add the line breaks through Emby by going to the "Edit Info" from the menu on the Item page, and add them directly to the Overview line - at which point they then get stripped anyway, presumably when the library is refreshed.

 

Question: Any idea why there is variability between the reaction to <p> tags in two different but same-structured nfo files?

[Deathsquirrel 741]

I'd suggest confirming your existing metadata is the issue.  Remove it.  Scan the content.  Does it identify correctly?

It may make sense to setup a small subset of a few movies in their own library before doing this.  That will keep tests fast.

[Luke 37065]

Hi there, for the refresh problem, please discuss an example and attach the information requested in how to report a problem. thanks !

 

Html content in overviews is not supported.

[Luke 37065]

 

 

Questions: Does Emby honor existing keys when it opens an nfo file for a given movie? (I assume it does).  Is there a list of keys used/populated by Emby, particularly on the Movie item homepage?  Are there relational lists for translating an older key/copying it to what Emby uses, and heirarchial structure for which keys Emby copies from (I've seen my comments end up in a new "plot" key without me putting them there) or ignores and downloads as it sets up the nfo to it's requirements?

 

yes, unknown tags are preserved as-is. Can you give an example of comments in plot? Thanks.

[NGNear 2]

Hey guys, thanks for the quick responses.

 

 

I'd suggest confirming your existing metadata is the issue.  Remove it.  Scan the content.  Does it identify correctly?

 

Short answer, Identify yes, Metadata Download no.  The problem I outline above occurs with both cleaned (i.e. no metadata at all) and previously-metadata'd movies.

 

Server log for the attempted refresh is attached to this post.  The movie in question was "Armour of God 2 - Operation Condor", but the problem is the same with movies that have english original titles as well.  Note that I've done a find-and-replace on the server's LAN WINS/DNS name, it now reads as <<emby_server_lan_name>> but is otherwise unchanged.

 

I've also attached a couple of my nfo files to this post, to provide examples of what I'm talking about.

 

The first metadata file is "2001 A Space Odyssey.nfo".  This file is a good example of:

- Formatting in the "Overview" key (see <p> tags on lines 239-241).  I'm also attaching two screenshots, 2001-A.jpg and 2001-B.jpg, which shows the Outline key in Emby's editor, complete with <p> tags, and the resulting formatting on the Movie page

- Copy of my comments into the "Plot" key ("BluRay, HD mkv and DVD Region 2")

 

The next metadata file is "American Beauty [DVD].nfo"

- This file actually still has the <p> tags in the Overview key, however the Movie page does not show the line breaks, and the Edit Info page does not show them in the Overview entry (I take it this is expected behaviour, from Luke's comment).

- This file also has a copy of my comments from the "Overview" key included in the "Plot" key.

embyserver.log

American Beauty DVD.nfo

2001 A Space Odyssey.nfo

[Luke 37065]

Your metadata contains a movie db id that is no longer valid:

2018-01-20 10:33:44.488 Info HttpClient: HttpClientManager GET: https://api.themoviedb.org/3/movie/234228?api_key=f6bd687ffa63cd282b6ff2c6877f2669&append_to_response=casts,releases,images,keywords,trailers&language=en&include_image_language=en,null
2018-01-20 10:33:45.029 Error HttpClient: Error ProtocolError getting response from https://api.themoviedb.org/3/movie/234228?api_key=f6bd687ffa63cd282b6ff2c6877f2669&append_to_response=casts,releases,images,keywords,trailers&language=en&include_image_language=en,null
	*** Error Report ***
	Version: 3.2.60.0
	Command line: /usr/lib/emby-server/bin/MediaBrowser.Server.Mono.exe -programdata /var/lib/emby-server -restartpath /usr/lib/emby-server/restart.sh
	Operating system: Unix 4.4.0.109
	64-Bit OS: True
	64-Bit Process: True
	User Interactive: False
	Mono: 4.6.2 (Stable 4.6.2.7/08fd525 Mon Nov 28 20:49:26 UTC 2016)
	Processor count: 8
	Program data path: /var/lib/emby-server
	Application directory: /usr/lib/emby-server/bin
	System.Net.WebException: The remote server returned an error: (404) Not Found.
	  at System.Net.HttpWebRequest.EndGetResponse (System.IAsyncResult asyncResult) [0x00064] in <bd46d4d4f7964dfa9beea098499ab597>:0 
	  at System.Threading.Tasks.TaskFactory`1[TResult].FromAsyncCoreLogic (System.IAsyncResult iar, System.Func`2[T,TResult] endFunction, System.Action`1[T] endAction, System.Threading.Tasks.Task`1[TResult] promise, System.Boolean requiresSynchronization) [0x00014] in <8f2c484307284b51944a1a13a14c0266>:0 
	System.Net.WebException
	  at System.Net.HttpWebRequest.EndGetResponse (System.IAsyncResult asyncResult) [0x00064] in <bd46d4d4f7964dfa9beea098499ab597>:0 
	  at System.Threading.Tasks.TaskFactory`1[TResult].FromAsyncCoreLogic (System.IAsyncResult iar, System.Func`2[T,TResult] endFunction, System.Action`1[T] endAction, System.Threading.Tasks.Task`1[TResult] promise, System.Boolean requiresSynchronization) [0x00014] in <8f2c484307284b51944a1a13a14c0266>:0 
	

[NGNear 2]

Hi Luke,

 

This movie is one which I nuked back to the original file and asked Emby to scrape; the TMDB ID in there is one provided by Emby.  I can repeat this process on another file if necessary.

 

I just removed all metadata from the Armour of God 2 folder again, so only the movie file remained, and then did a re-scan of the library (Server Home, Advanced, Scheduled Tasks, Scan Media Library).

 

This added an Armour of God 2.nfo file to the subfolder.  I entered the Movie Item, selected the three-dot menu icon, and selected the Refresh Metadata option.  I chose "Replace all Metadata", OKed it, and got a folder.jpg which is I think a screengrab from the video file, and the attached nfo file, which seems to have ID-ed the movie (now has both IMDB and TMDB keys), names (original Cantonese, Full Title, Sort Title) and media info, but not filled out any Plot or Outline detail.  See it attached to this message as "Operation Condor.nfo".

 

The Movie Item page is as per the screenshot.

 

 

I'm having trouble uploading the nfo file, not sure why.  It's included here behind spoiler tags as it's a little large.  Operation Condor.nfo (i.e. created by Emby):

 

 

<?xml version="1.0" encoding="utf-8" standalone="yes"?>
<movie>
  <plot />
  <outline />
  <lockdata>false</lockdata>
  <dateadded>2012-12-02 00:00:23</dateadded>
  <title>Armour of God II: Operation Condor</title>
  <originaltitle>飛鷹計劃</originaltitle>
  <sorttitle>Armour of God 2</sorttitle>
  <collectionnumber>146136</collectionnumber>
  <imdbid>tt0099558</imdbid>
  <tmdbid>10975</tmdbid>
  <runtime>92</runtime>
  <art>
    <poster>/mnt/8TB_D2/movies/Armour of God 2 - Operation Condor [DivX]/folder.jpg</poster>
  </art>
  <isuserfavorite>false</isuserfavorite>
  <userrating>10</userrating>
  <playcount>7</playcount>
  <watched>true</watched>
  <lastplayed>2016-04-30 15:58:18</lastplayed>
  <resume>
    <position>0</position>
    <total>5521.104</total>
  </resume>
  <id>tt0099558</id>
  <fileinfo>
    <streamdetails>
      <video>
        <codec>msmpeg4</codec>
        <micodec>msmpeg4</micodec>
        <bitrate>916279</bitrate>
        <width>512</width>
        <height>240</height>
        <framerate>23.976</framerate>
        <scantype>progressive</scantype>
        <default>False</default>
        <forced>False</forced>
        <duration>92</duration>
        <durationinseconds>5521</durationinseconds>
      </video>
      <audio>
        <codec>mp3</codec>
        <micodec>mp3</micodec>
        <bitrate>127704</bitrate>
        <scantype>progressive</scantype>
        <channels>2</channels>
        <samplingrate>44100</samplingrate>
        <default>False</default>
        <forced>False</forced>
      </audio>
    </streamdetails>
  </fileinfo>
</movie>

 

 

embyserver-2.log

[Luke 37065]

If you can reproduce on a brand new movie that would be the best way to troubleshoot. As it stands it looks to me like that old id must have been in your original nfo file and that's any the identify feature corrects it but the metadata refresh does not. Thanks.

[NGNear 2]

Duplicated with a completely fresh movie file.

 

Your Name.mp4 was added to a new folder called /Your Name [bluRay] and the library re-scanned as before.

 

I entered the Movie Item page, used the Identify option and confirmed the correct movie, then hit Refresh Metadata and Replace All Metadata (and replace images option) then ran.

 

The resulting files are: folder.jpg screengrab from the mp4, no images downloaded, and Your Name.nfo file, included below (won't let me attach, not sure why), with no Plot or Outline data.  Also attached is my server log for this attempt.

 

 

 

<?xml version="1.0" encoding="utf-8" standalone="yes"?>
<movie>
  <plot />
  <outline />
  <lockdata>false</lockdata>
  <dateadded>2018-01-03 21:19:41</dateadded>
  <title>Your Name [BluRay]</title>
  <imdbid>tt5311514</imdbid>
  <runtime>0</runtime>
  <art>
    <poster>/mnt/8TB_D2/movies/Your Name [BluRay]/folder.jpg</poster>
  </art>
  <isuserfavorite>false</isuserfavorite>
  <playcount>0</playcount>
  <watched>false</watched>
  <resume>
    <position>0</position>
    <total>8.452</total>
  </resume>
  <id>tt5311514</id>
  <fileinfo>
    <streamdetails>
      <video>
        <codec>h264</codec>
        <micodec>h264</micodec>
        <bitrate>699772</bitrate>
        <width>1280</width>
        <height>720</height>
        <aspect>16:9</aspect>
        <aspectratio>16:9</aspectratio>
        <framerate>29.97003</framerate>
        <language>und</language>
        <scantype>progressive</scantype>
        <default>True</default>
        <forced>False</forced>
        <duration>0</duration>
        <durationinseconds>8</durationinseconds>
      </video>
      <audio>
        <codec>aac</codec>
        <micodec>aac</micodec>
        <bitrate>125588</bitrate>
        <language>und</language>
        <scantype>progressive</scantype>
        <channels>2</channels>
        <samplingrate>44100</samplingrate>
        <default>True</default>
        <forced>False</forced>
      </audio>
    </streamdetails>
  </fileinfo>
</movie>

 

 

embyserver-3.log

[Happy2Play 8282]

Have you verified metadata and library options?  To me is sound like download options disable per library.

[NGNear 2]

Have you verified metadata and library options?  To me is sound like download options disable per library.

 

Son of a...  yeah, you were right.  I had selected Metadata Savers, Metadata Downloaders, Image Fetchers, Save Artwork and Metadata into Media Folders, Download Images In Advance correctly... but neglected to select "Download Artwork and Metadata from the Internet".

 

Sorry, guys.  ID-10-T problem here.

[NGNear 2]

Which is to say, I re-ran "Refresh Metadata" for Your Name and it downloaded the information correctly.  Interestingly, it didn't automatically replace the screengrabbed folder.jpg despite selecting "replace images" on the Metadata Refresh, but not a big deal.

[PenkethBoy 2063]

 

Next up, as part of the information I've been adding to the files is additional info in the main Overview/Plot paragraph(s) on the item page.  I've found that adding HTML-style "<p>" tags allows me to break the text into more readable pieces rather than a single run-on sentence, but this seems to work on a surprisingly inconsistent basis - some xml/nfo files with an existant Overview key including a couple of <p> tags correctly format the text, others with the same xml layout do not - when I open the xml/nfo file, the <p> tags I added in Notepad++ have been stripped out of the Overview key in the file.  I can add the line breaks through Emby by going to the "Edit Info" from the menu on the Item page, and add them directly to the Overview line - at which point they then get stripped anyway, presumably when the library is refreshed.

 

 

I use the "<br>" tag to do the same thing in the Overview/plot of Collections and they do not get stripped for me. Also "<b></b>" and "<u></u>" work as well for bold and underline - which i use for titles of the movies followed by their individual descriptions

 

e.g

 

from info page

 

i use two "<br>" tags to get a bit more separation on the text

[Luke 37065]

That is relying on unsupported behavior though and is not guaranteed to continue working.