Conversion from Collections XML to NFO format to capture <set> tag

[swagardson 1]

Hello,

I've been using Emby premium for a few years now and have generally had little-to-no issues, which has been great.

After upgrading to 4.7 (beta), I noticed items getting kicked out of my collections -- hundreds of items that I painstakingly put together. I understand that this is due to "better future compatibility" and standard format, but I do wish old collections would have remained untouched in a minor version update.

Anyways, I read these couple threads to attempt troubleshooting and playing around with various options:
https://emby.media/community/index.php?/topic/99087-latest-update-is-randomly-removing-tv-shows-from-collections-also-creating-new-collections/
https://emby.media/community/index.php?/topic/99945-films-and-tv-shows-randomly-keep-being-removed-from-collections/page/5/

And after multiple failures to have it resolved non-manually, I downgraded back to 4.5.3, which put everything back in its proper place.

Avoiding veering more off-topic, I'm wondering:

1. How are collections being defined in the most recent versions? Purely in the item's NFO file and no longer using `data/collections/*`?
2. If an individual item's NFO file is the new path forward, is there is a built-in Emby utility or general means to add <set> tags to an item based on each collection it is in?
3. Related to above, if multiple <set> tags are defined in an item's NFO file, will it show up in each collection it is defined in?
   1. Reason I ask is because I have a script which iterates through items in a collection and adds a <set> tag for each collection that item is in
4. Which options are most pertinent to accurately storing NFO metadata (e.g. "Metadata savers: [X] NFO" in a library's settings) AND reading that info in to properly populate collections?
5. Will the "Auto Box Set" plugin wreak havoc on random items if enabled? I noticed that the only collections being made were ones resulting from that plugin tagging the NFO files, while everything else was being kicked out.

I really do appreciate the time that everyone puts into the software and answering these questions.

Thanks for any information in advance.

[Abobader 3307]

Hello swagardson,

** This is an auto reply **

Please wait for someone from staff support or our members to reply to you.

It's recommended to provide more info, as it explain in this thread:

Thank you.

Emby Team

[Luke 39648]

@cayars

[Carlo 4546]

21 hours ago, swagardson said:

Avoiding veering more off-topic, I'm wondering:

1. How are collections being defined in the most recent versions? Purely in the item's NFO file and no longer using `data/collections/*`?
2. If an individual item's NFO file is the new path forward, is there is a built-in Emby utility or general means to add <set> tags to an item based on each collection it is in?
3. Related to above, if multiple <set> tags are defined in an item's NFO file, will it show up in each collection it is defined in?
   1. Reason I ask is because I have a script which iterates through items in a collection and adds a <set> tag for each collection that item is in
4. Which options are most pertinent to accurately storing NFO metadata (e.g. "Metadata savers: [X] NFO" in a library's settings) AND reading that info in to properly populate collections?
5. Will the "Auto Box Set" plugin wreak havoc on random items if enabled? I noticed that the only collections being made were ones resulting from that plugin tagging the NFO files, while everything else was being kicked out.

I really do appreciate the time that everyone puts into the software and answering these questions.

Thanks for any information in advance.

Did you restore the system back to 4.5.3 and not just install that version over top 4.6.4?  If you just installed 4.5.3 over top you are bound to have issues as the database was changed in 4.6.

Previously, collections were stored in XML files and should be written to the NFO file.  In the new version of Emby the XML files are no longer used and this information is now stored in the database itself and of course written to the NFO.

The BoxSet plugin is no longer used either as collection functionality is now built into each library where you can configure it.

How you proceed is going to depend on a few things.  Can you check a handful of your custom collection a pick out some of the movies in them and then look at the NFO files.
Do you see your custom collection in the NFO file?

If so that's good as it can be read back later to recreate the collection.  If it's not present and update to a newer version and full library scan will remove those collection. 
So we want to verify you have this info in the NFO.

Let's start with that.

[swagardson 1]

6 hours ago, cayars said:

Did you restore the system back to 4.5.3 and not just install that version over top 4.6.4?  If you just installed 4.5.3 over top you are bound to have issues as the database was changed in 4.6.

Hey, Cayars. I did downgrade on top, which I know will cause potential incompatibility (grievances about non-semantic versioning aside), but the collections issue was my main concern. I'm looking to resolve the issue now so I can upgrade again.
 

6 hours ago, cayars said:

Previously, collections were stored in XML files and should be written to the NFO file.  In the new version of Emby the XML files are no longer used and this information is now stored in the database itself and of course written to the NFO.

The BoxSet plugin is no longer used either as collection functionality is now built into each library where you can configure it.

That's all good and understandable. Thank you for the info.

6 hours ago, cayars said:

How you proceed is going to depend on a few things.  Can you check a handful of your custom collection a pick out some of the movies in them and then look at the NFO files.
Do you see your custom collection in the NFO file?

If so that's good as it can be read back later to recreate the collection.  If it's not present and update to a newer version and full library scan will remove those collection.

So that's my key issue. The NFO files don't have any reference to my custom collections, so the server kicks them out little by little over time.

For clarification:

- What component or setting of Emby should have it writing the information to the NFO files automatically?
- Can one item (NFO file) be a part of multiple "<set>"s (AKA collections)?
- Are collections automatically made when reading from the NFO file's <set> tags?
- How important is the database in determining what is a collection item and what is not? Is the NFO file populating the database in this regard?

Thanks again for your time in advance.

[Carlo 4546]

He's what I would try. Pick a movie that is in a collection that presently does not have that info in the NFO.

In Emby do a mete-refresh on just that movie.  Check the NFO.  Is the collection info there now?
If so you're probably in good shape.  Try a few more.  If what you test all get this info written to the NFOs you should be safe to do a library wide meta-refresh.

Let us know how this works.
If it didn't work, try another test on an individual movie or two.  Turn of NFO reading in the library then perform the above.
Any difference?
 

[Luke 39648]

Quote

- Can one item (NFO file) be a part of multiple "<set>"s (AKA collections)?
- Are collections automatically made when reading from the NFO file's <set> tags?

Yes to both.

[swagardson 1]

14 hours ago, Luke said:

Yes to both.

Awesome. Thank you for confirming.

22 hours ago, cayars said:

He's what I would try. Pick a movie that is in a collection that presently does not have that info in the NFO.

In Emby do a mete-refresh on just that movie.  Check the NFO.  Is the collection info there now?
If so you're probably in good shape.  Try a few more.  If what you test all get this info written to the NFOs you should be safe to do a library wide meta-refresh.

Let us know how this works.
If it didn't work, try another test on an individual movie or two.  Turn of NFO reading in the library then perform the above.
Any difference?
 

Didn't seem to work, unfortunately.

Oddly enough I don't seem to have the option to turn off NFO reading for the library (even with advanced settings enabled), but I do remember that setting being there before. I only have the "Metadata savers" section. If this is a side-effect of the on-top downgrade, I apologize.

[Carlo 4546]

Go to Library menu in admin and check the type being shown for you libs.  For example does you movie library use type Movie or Mixed Content?
This is where it will show that.

If the library is setup as the wrong type your options will be different and of course scanning will be different as well.  If this is set wrong it can be correctly but let us know.

[swagardson 1]

Thanks for the quick response.

Everything should be set correctly. Here's a sample screenshot of my libraries.

For reference, everything we've been doing so far I've been relegating only to the "Movies" library, as that's the only library I have collections in.

[Luke 39648]

Is there still an issue at this point?

[swagardson 1]

Yes, there is. By "everything should be set correctly", I meant that all of my videos are categorized correctly, as they've always been. Not sure why some options are not showing up.

I'm awaiting next steps, or being told that I should run my custom script.

[Luke 39648]

What options exactly? The option to turn off metadata readers doesn't exist in 4.5.

[swagardson 1]

Hmmm, okay. Then we should be fine then in that regard.

I'm ready to try anything else that may work.

[Luke 39648]

Are you still on 4.5.4?

[swagardson 1]

Yup. I've been in the same state since the beginning of the thread. I will upgrade again eventually, but I want to try and upgrade legitimately first.

[Luke 39648]

Ok. What are you looking to do before you update?

[swagardson 1]

I'm confused. Are we not still trying to resolve why my NFO metadata does not get updated with my collection info (AKA the <set> tags), which then starts slowly kicking my items out of my collections upon upgrade to >= 4.6?

[Luke 39648]

Try refreshing the metadata on a title and see if that updates the nfo file. Please see if that helps. Thanks.

[swagardson 1]

22 hours ago, swagardson said:

Didn't seem to work, unfortunately.

I already did this when prompted awhile ago. I'm even more confused now. Lol.

Is @cayars not helping any more?

[Luke 39648]

He is, but the instructions I gave you are the right ones. If that's not happening, then we need to look at why, e.g. your library options, server log from an example, etc.

 

[Carlo 4546]

30 minutes ago, swagardson said:

Is @cayars not helping any more?

Still here but Luke jumped in and I didn't want to distract or make what he was asking/saying confusing.

[swagardson 1]

Here are the server logs, with the last item being the failed refresh, which could be the issue with the NFO file not populating since the transaction gets cancelled overall.

embyserver.txt

Unrelated, but there's also a lot of UPnP/SSDP socket errors that populate the majority of the log. If you know what the issue with that may be, let me know. If it's not simple, I can open another topic.

[Luke 39648]

19 hours ago, swagardson said:

Here are the server logs, with the last item being the failed refresh, which could be the issue with the NFO file not populating since the transaction gets cancelled overall.

embyserver.txt 24.84 MB · 1 download

Unrelated, but there's also a lot of UPnP/SSDP socket errors that populate the majority of the log. If you know what the issue with that may be, let me know. If it's not simple, I can open another topic.

It looks like your connection to MovieDb is timing out. Typical causes of this are firewall, security software, or VPN. Can you please look into that? Thanks.

[Carlo 4546]

23 hours ago, swagardson said:

Here are the server logs, with the last item being the failed refresh, which could be the issue with the NFO file not populating since the transaction gets cancelled overall.

embyserver.txt 24.84 MB · 2 downloads

Unrelated, but there's also a lot of UPnP/SSDP socket errors that populate the majority of the log. If you know what the issue with that may be, let me know. If it's not simple, I can open another topic.

Try turning off DLNA in Emby menu.