4.6 - How are Collections stored and other changes?

[Diedrich 355]

WooHoo! 4.6 is out! So many awesome things with this new version. Thank you to all the devs for all the hard work. The questions:

What are the changes for Collections?
How were my current 4.5 Collections migrated?
How are Collections stored / and where are they stored?
How do I migrate Collections when moving to a new OS/environment?

Anything else we need to know about the new Collections rewrite?

Again, thank you so much for all the hard work!

 

 

 

 

 

[crusher11 854]

Collections are now in metadata/collections, but it's just an NFO file with the collection name, overview etc and any collection artwork.

Items are linked to collections via the NFO file for each item, there's now a <set> tag which contains either the TMDB ID of the collection, or the name of the collection, depending on whether it's a custom or automatic collection.

 

[Carlo 4330]

In addition to what crusher11 just mentioned, this info in now in the database vs being read from XML files.

Previously the XML file had the path hard coded to the media which meant if you ever converted it (new extension) or moved it, the collection was broken until you manually edited the collection XML file. You can now move any of these files just like other media and after a scan or refresh will get updated and still be part of any collections.

During the update the old style XML files were read in and converted to the new/better format as a one-time only job.

Migrations are much easier now because this info is much more "portable" and doesn't rely or specific media paths in the XML file.

Hope that helps

[Diedrich 355]

Thank you very much to both of you, I appreciate the information. I think that answered all my questions.

[blade005 172]

8 hours ago, cayars said:

In addition to what crusher11 just mentioned, this info in now in the database vs being read from XML files.

Previously the XML file had the path hard coded to the media which meant if you ever converted it (new extension) or moved it, the collection was broken until you manually edited the collection XML file. You can now move any of these files just like other media and after a scan or refresh will get updated and still be part of any collections.

During the update the old style XML files were read in and converted to the new/better format as a one-time only job.

Migrations are much easier now because this info is much more "portable" and doesn't rely or specific media paths in the XML file.

Hope that helps

Should there be a folder created for each COLLECTION item under programdata/metadata/collections with this one-time only job?

I ask because my 396 COLLECTION items are in this new collection library after the update. So it does seem like everything from the previous collections folder did get integrated into the database. However, when I look in the metadata/collections folder I currently see six (6) folder entries where I would expect to see 396. And of those 6 folders, only 4 had a collection.nfo file in them. I have run a full SCAN LIBRARY a few times since the update. Is there anything else that should be done in order to get all items updated into this new folder location? Without that, any future migration to a new hardware server would require a complete rebuild of COLLECTION items. Would prefer not to have to do that.

Thanks.

[Diedrich 355]

@cayars@Luke

v4.6.0.50, logs attached

I rebooted to accept the update from 4.5 and then manually ran a library scan. I then posted the question above about the Collections. Over 3 hours after the first reboot I do not see the conversion from the old collections. I rebooted and ran the scan again. As before, the scan hangs at 97.6% for a very long time (10 minutes) to now get to 97.7%. The scan is now complete and the collections haven't converted yet. Do I need to wait longer? Is this conversion not firing or is it breaking?

embyserver.txt

[Luke 37095]

1 minute ago, Diedrich said:

@cayars@Luke

v4.6.0.50, logs attached

I rebooted to accept the update from 4.5 and then manually ran a library scan. I then posted the question above about the Collections. Over 3 hours after the first reboot I do not see the conversion from the old collections. I rebooted and ran the scan again. As before, the scan hangs at 97.6% for a very long time (10 minutes) to now get to 97.7%. The scan is now complete and the collections haven't converted yet. Do I need to wait longer? Is this conversion not firing or is it breaking?

embyserver.txt 3.68 MB · 1 download

Hi, I would just be patient and allow the scan to finish.

[Diedrich 355]

1 minute ago, Luke said:

Hi, I would just be patient and allow the scan to finish.

The manual scan did finish, twice. No conversions yet.

Edited May 20, 2021 by Diedrich

[Luke 37095]

Just now, Diedrich said:

The scan did finish, twice. No conversions yet.

What does "no conversions" mean?

[Happy2Play 8296]

Are you referring to information moving from data\collections to metadata\collections?

[Carlo 4330]

5 minutes ago, Diedrich said:

The manual scan did finish, twice. No conversions yet.

Do you have collections?
Do they work?

If so the upgrade was successful and your collections were migrated properly.
You may need to redo some of your images however.

[Diedrich 355]

6 minutes ago, Happy2Play said:

Are you referring to information moving from data\collections to metadata\collections?

Yes. I still see folders in data/collections along with the images and xml files for each collection. I only see one folder in metadata/collections for a collection that was created after the update.

5 minutes ago, cayars said:

Do you have collections?
Do they work?

If so the upgrade was successful and your collections were migrated properly.
You may need to redo some of your images however.

Collections work. Images appear as expected. I'm still dubious that there was a conversion since I still have xml files in the data folder.

[Luke 37095]

You're fine then.

[blade005 172]

10 minutes ago, Happy2Play said:

Are you referring to information moving from data\collections to metadata\collections?

Yes, from my previous post. All of the collection XML collection information and image content that was in data\collections has not moved over to metadata\collections.

6 minutes ago, cayars said:

Do you have collections?
Do they work?

If so the upgrade was successful and your collections were migrated properly.
You may need to redo some of your images however.

Yes, I do have collections. (It created a new collection library entry, while leaving my previous one there as well.)

Yes, they all seem to work. Although entering each collection requires a wait of anywhere from 5 to 20 seconds before any items are displayed.

But, as asked previously, should we see a folder entry for each collection item in the new location of metadata\collections ?  If the answer is yes, then I have an issue since 99.5% of mine are not populated in the new location.

[Happy2Play 8296]

Don't believe there is a migration from folder to folder as I only get info in metadata\collection by refreshing said collection.

[Diedrich 355]

4 minutes ago, blade005 said:

But, as asked previously, should we see a folder entry for each collection item in the new location of metadata\collections ?  If the answer is yes, then I have an issue since 99.5% of mine are not populated in the new location.

Same concern. Same question.

Except.... So how are we supposed to migrate the collections library? Copy the data/collections or metadata/collections since there is still xml data in data/collections?

Edited May 20, 2021 by Diedrich

[blade005 172]

12 minutes ago, Diedrich said:

Yes. I still see folders in data/collections along with the images and xml files for each collection. I only see one folder in metadata/collections for a collection that was created after the update.

Collections work. Images appear as expected. I'm still dubious that there was a conversion since I still have xml files in the data folder.

I'm with you there. All of my XML collection folders (396) are still in the previous location of data/collections. This was convenient if you installed a new server as you could move this folder over and after a library scan your collections were as before.

 

After this update and a successful library scan this new location of metadata/collections has six (6) folder entries and only four (4) of these have a collections.nfo file and none of them have the image information that was in the previous folders. Not sure how these six (6) showed up and not the other 390 and why only four (4) of them have a collection.nfo file in them.

 

The collections are all there after the update, but it appears that they are integrated in the database, but there is no migratable folder structure for future use. Bummer.

Edited May 20, 2021 by blade005

[blade005 172]

4 minutes ago, Happy2Play said:

Don't believe there is a migration from folder to folder as I only get info in metadata\collection by refreshing said collection.

Do you mean a refresh of an individual collection item? If yes, YIKES!! I would have to hit the REFRESH button 396 times to get all of my collection items updated in the new location!!!

Edited May 20, 2021 by blade005

[Luke 37095]

5 minutes ago, blade005 said:

there is no migratable folder structure for future use. Bummer.

Your movie nfo files, if you use nfo metadata.

[Diedrich 355]

6 minutes ago, Happy2Play said:

Don't believe there is a migration from folder to folder as I only get info in metadata\collection by refreshing said collection.

Ah-ha! That did it! Thanks! "Refresh metadata" worked to "convert" (move the collection from data/collections to metadata/collections. Thanks!

[blade005 172]

5 minutes ago, Diedrich said:

Ah-ha! That did it! Thanks! "Refresh metadata" worked to "convert" (move the collection from data/collections to metadata/collections. Thanks!

Is that for an individual item? The new collection library that was created does not have a REFRESH METADATA or SCAN LIBRARY FILES like the previous one does.

[Luke 37095]

Just now, blade005 said:

Is that for an individual item? The new collection library that was created does not have a REFRESH METADATA or SCAN LIBRARY FILES like the previous one does.

Because it's not a library. It's just a virtual grouping.

 

8 minutes ago, Luke said:

Your movie nfo files, if you use nfo metadata.

 

[blade005 172]

6 minutes ago, Luke said:

Because it's not a library. It's just a virtual grouping.

 

 

Okay. Got it. Cool if the movie.nfo files will travel with Collection information for a migration.

I did refresh a few individual collection items and that did create a folder in the new location with a collection.nfo file. But, no image data for the collection items. Where is that stored? They are showing up for the collections, but I'm not sure where they are being stored.

[Diedrich 355]

7 minutes ago, blade005 said:

Is that for an individual item? The new collection library that was created does not have a REFRESH METADATA or SCAN LIBRARY FILES like the previous one does.

I did the refresh metadata for each collection as seen here:

 

However, I did not get the poster images copied over from the data/collections folders to the new location of /metadata/collections, only the nfo files were created for each collection. For completeness, I also did a "Scan Library Files" for each collection just to be sure.

[Diedrich 355]

1 minute ago, blade005 said:

Okay. Got it. Cool if the movie.nfo files will travel with Collection information for a migration.

I did refresh a few individual collection items and that did create a folder in the new location with a collection.nfo file. But, no image data for the collection items. Where is that stored? They are showing up for the collections, but I'm not sure where they are being stored.

Mine didn't migrate either, they are still in the data/collections folders.