Question about server behavior on remove libraries and re-adding them originating from a different path

[djhifi 55]

I am in the process of reformulating some of my internal HDD volumes on my NAS.

Those HDDs have "Movies" and "TV Shows" with .nfos stored with media but the artwork is scrapped and stored on a centralized folder rather than with media (like the .NFOs are).

Those .NFOs (stored with media) are pointing to that said centralized artwork path I've defined (exemple below).

 

If I remove the library (on EMBY) that I need to "destroy" the HDD volume, would the server remove the individual artwork related to those IDs on that said centralized path?

 

I ask this because the other way around (destroying a volume with the server uninstalled) it's easier and I've already tested and works (works = library picks up what was defined pre-destruction, including artwork that I manually defined).

 

Using this method:

 

If I backup the centralized metadata folder, uninstall the server (which I did, to move it to SSD) and then move said folder back to the path the .nfos are pointing to, it works to a point. It picked everything up despite the media files (movies and shows) being on a dif. path. But ignores the images. How does it handle this then? Since those are written on the nfo's. Overwrites?

 

 

 

Having the server installed and removing a library is something diferent to my eyes so:

 

Admiting it does remove such images, I would then backup the centralized metadata folder again and then copy and overwrite the libraries that I wouldn't touch.

Is this viable or would the server atribute a dif. "itemid" since I'm actually modifying libraries with it (the server) installed and running (as opposed to what I did above) ? edit: it seems the server ignores the centralized path and only reads .nfos if I re-add the library on a dif. path

 

This is an example of an .nfo from one episode of Walking Dead that was on one of the volumes which I destroyed (after uninstalling the server as well) and then changed that TV Show to another volume and reinstalled the server:

 

**************

<episodedetails>
  <plot>The Survivors encounter unfamiliar faces outside the safety of their community's walls and must decide whether or not this new group can be trusted.</plot>
  <outline />
  <lockdata>false</lockdata>
  <dateadded>2018-11-12 21:02:00</dateadded>
  <title>Who Are You Now</title>
  <director>Larry Teng</director>
  <rating>7</rating>
  <year>2018</year>
  <imdbid>tt8250818</imdbid>
  <tvdbid>6847067</tvdbid>
  <runtime>54</runtime>
  <art>
    <poster>/volume6/Emby/Emby Stuff/metadata/library/91/913fcf13e307e8c6efd350e228266f7a/poster.jpg</poster>

*************

 

In the "dateadded" field you can see that the date relates to the time it was added to the library BEFORE I uninstalled the server (and the TV Show was on the still-not-destroyed volume).

You can also see the "art" field is pointing to the centralized metadata path where I store all artwork.

 

So again, How does it handle this ? Since those are written on the nfo's. Overwrites them, in case I back up said folder and restore it prior to making a library scan on "new" items? or?

 

- - - -

 

@@Happy2Play said that one of the possible outcomes could be that when I added back that library originating from a diferent path and altough I had the .nfos stored with media and pointing to the centralized artwork, the server would ignore and rescrape everything again as if it was new content (with the added pain in the a** of having to go through one by one to check if all are 100%).

 

Im posting this public to hear from more people.

Edited November 22, 2018 by djhifi

[Happy2Play 8356]

@@Luke

 

From a centralized image perspective, if a library is removed and readded with new path and images are save in the nfo with a no longer valid itemid will they eventually get overwritten by a item taking that itemid?

 

Or will this not be a issues as long as this original database exists.

Edited November 22, 2018 by Happy2Play

[djhifi 55]

@@Luke

 

From a centralized image perspective, if a library is removed and readded with new path and images are save in the nfo with a no longer valid itemid will they eventually get overwritten by a item taking that itemid?

 

Or will this not be a issues as long as this original database exists.

 

I think I have to take back half of what I said about the sucessful experience related above when I uninstalled the server and backed up the artwork centralized folder.

(I edited the first post, but still has the fundamental question)

 

It seems the server is indeed displaying all the correct movies and tv shows (even the ones that were badly scrapped and I manually edited) but it's not choosing the artwork that is on the centralized metadata folder which I backed up and restored.

 

So in theory it seems that it's Reading the .nfos so the movies that were being badly scrapped are correct because the nfo's are there despite the path is diferent but regarding images it seems the server is ignoring what's in the .nfo (the path were the image is stored as the exemple of my first post).

 

So maybe you're right and it's attributing a new itemID even on a new installation of the server. So what's the usefulness of having the centralized artwork path (with the artwork ID) stored with the .NFO for file-moving purposes? 

And also, how does the server handle the fact that the centralized metadata path is written on .NFOs that the server reads? It overwrites them? (see the example on first post)

 

Im currently running a test on a movie.

 

I have deleted it's artwork from the metadata folder and the server is currently running a scan on the new HDD i've added to the library.

I can live with the fact of not having to manually identify movies that were badly scrapped and only needing to change their images but then again, what happens with the images stored in the said centralized path and written on the .NFOs?

 

Unrelated (or not):

 

Is something bugged with "sort by date added" ? I think these new movies im -readding are not showing up when I select that view.

They show up with "sort by title" view though.

 

Since the "total number" got stripped from the interface I can't really confirm this. I will install the statistics plugin to see if all is fine.

Edited November 22, 2018 by djhifi

[Luke 37248]

@@Luke

 

From a centralized image perspective, if a library is removed and readded with new path and images are save in the nfo with a no longer valid itemid will they eventually get overwritten by a item taking that itemid?

 

Or will this not be a issues as long as this original database exists.

 

Id's are never re-used, so no.

[Luke 37248]

PS: Is something bugged with "date added" ? I think these new movies are def. not showing up when I select that view.

 

No, it's not. Explore your date added values for your titles. Also explore the user settings that determine what is displayed in latest items.

[djhifi 55]

Id's are never re-used, so no.

 

Then in the event I remove a library and re-add it how does the server handle the fact that the centralized metadata path (for artwork) is written on .NFOs with the obsolete itemID link to the folder previously created (and still existing on the metadata path? It overwrites on the .NFO with the new one and deletes locally the "old" one on the library scan since the IDs are not re-used? Deletes those folders only and does nothing at the .NFO level? Or something else?

 

example:

 

<episodedetails>  

(…)

  <art>

    <poster>/volume6/Emby/Emby Stuff/metadata/library/91/913fcf13e307e8c6efd350e228266f7a/poster.jpg</poster>

 

The above is on a Movie.nfo that gets removed. I then re-add said movie or entire collection. That folder and file still exist on my manually-configured centralized metadata path for all the artwork.

 

What happens during the library scan on these types of items that have .nfos using an older ID (that still exists physically on another part of the server)?

 

 

 

That's my original question above anything else. Because I cannot seem to understand all the expected behaviors in these events.

Edited November 22, 2018 by djhifi

[djhifi 55]

No, it's not. Explore your date added values for your titles. Also explore the user settings that determine what is displayed in latest items.

 

I'm in the process of scanning a new library that got moved from a raid that I unmounted. But yesterday when I added another new library in the same conditions it was showing the new items getting added in the correct order (but still blank in terms of artwork) and then when it finished said scan, the only way I could find the movies would be via search or sort by title. The movies disappeared from the correct order of date added into oblivion on that view mode.

But anyway what do you mean by "values" and "user settings that determine what is displayed" ? I have got it defined on the dashboard to be "use date scanned into the library" since always.

 

Im talking within the library of "Movies". "sort by -> date added". Its not displaying movies that were in the library but that I removed and re-added on a dif. path.

 

All those movies have the "dateadded" field in the .nfo written from earlier date, of course, since they WERE in my Library, but don't even even seem to be showing in the end of the list of "date added" if I sort descending/ascending. Or I'm missing something which I'm unaware or it's because a new scan is in progress. But yesterday nothing was in progress and the behavior was the same.

 

As far as I know, even if the items existed in the database and then got removed but I kept all the .nfos, adding them back would place them first in line for date added sort orders. And it doesn't seem to be happening.

On TV Shows, for example, only "Preacher" of all the new TV Shows I've removed and added back is showing in the "Latest TV Shows" in homescreen but is not replicating that behavior if I open "TV Shows" library itself. It disappears when it should be there in the first page since it was one of the last added. You get the idea? Something wierd is going on both with Movies and TV Shows that I've removed and that I'm now adding back on a diferent path.

 

Anyway I will let this new scan finish to look that up. I thought it could be related to the fact I'm adding a new path but the .nfo's on that path are pointing to the artwork IDs in the centralized folder that still exist but you've confirmed new items change IDs so I don't know (again related related to the post above).

 

On the other hand, If I add a new movie that I just downloaded from anywhere, everything goes fine and it goes to it's correct place in date added as being the last item, since the server has to do everything from scratch on that media file.

 

Do I need to refresh per-library choosing "search for missing metadata" and untick "replace images" when the .NFOs of "new scans" exist or something to make this work? 

Edited November 22, 2018 by djhifi

[Luke 37248]

if you remove and re-add the libraries there is no guarantee you will get the same locations in the centralized metadata folder. That's why we recommend local metadata.

[djhifi 55]

if you remove and re-add the libraries there is no guarantee you will get the same locations in the centralized metadata folder. That's why we recommend local metadata.

 

Ups, my bad above. From my past testings, removing and re-adding libraries as long as their path is the same, re-used what was downloaded to the centralized metadata folder and written in the .NFOs. I've done it 2 times and everything remained untouched. Even though it's not an exact science as you say and I can't guarantee 100%, I took not of some files that I manually changed their artworks before removing the library and after re-adding it the artwork was what I manually changed.

 

My question relates to libraries re-added from diferent path than it was before. 

 

I understand that, by design, the server first allocates in the .NFOs the URL and only then downloads it. But how does it deal with .NFOs that have the path already set and pointing to an existing ID, since the files change itemIDs if they are on a diferent path how is this deal with?  Gets wiped, overwritten, etc?

 

Because that's what I'm doing right now on several HDDs. I'm changing the way they are on my NAS so I have to remove and re-add some libraries that now show as being on a dif. path.

 

Altough I understand that it's much more hassle-free storing both .NFOs and artwork with media files, I prefer just to have the .NFOs and then have all the artwork on a centralized folder in the SSD. I avoid having the HDDs waking up all the time. 

Edited November 22, 2018 by djhifi

[Luke 37248]

The server never reads image paths from nfo files. We only write if (if you want), so that Kodi can use it.

[djhifi 55]

The server never reads image paths from nfo files. We only write if (if you want), so that Kodi can use it.

 

Yes, I'm aware of that. I've selected that option because of artwork caching with the Emby for Kodi plugin.

 

I'm not explaining what I wish to know very well.

 

By design this is what happens:

 

Imagine "Movie X" gets added + scrapped.

The server writes the path to the .nfo.

First it writes the url and then it replaces it with the path of the corresponding file it downloads (poster, backdrop, whatever) to the centralized location in my case.

Folders containing that said artwork are now created on a per-movie basis

 

Example: 

 

 

  <art>

    <poster>/volume6/Emby/Emby Stuff/metadata/library/65/652a643e5619ac6f8c20c47ea03d0b6b/poster.jpg</poster>

    <fanart>/volume6/Emby/Emby Stuff/metadata/library/65/652a643e5619ac6f8c20c47ea03d0b6b/backdrop.jpg</fanart>

  </art>

 

In bold/blue I believe it's the itemID, right? (that by the way, also gets referenced on the library.db sqlite tables, correct?)

 

---------------------

 

Ok, so since it is storing all the artwork on a centralized location, when I remove a library or a movie and add it back from a diferent path, what happens to THAT folder (above) containing the artwork that the server created "before" I removed the movie/library?

 

Do these folders get overwritten or the centralized folder on my NAS will grow and not manage unused itemID folders/artwork?

 

I ask this since you said the server never re-uses ID. So I wonder, it creates new folders for the artwork. But the folder was already there, when the item had a diferent itemID.

 

So I was just wanting to know what happens to that "old" folder and all the "old" artwork.

Edited November 22, 2018 by djhifi

[Luke 37248]

Those old folders become obsolete and will be cleaned up.

[djhifi 55]

Those old folders become obsolete and will be cleaned up.

 

ahhhh! Finally! Thanks.

 

It does so on each library scan and also metada data refresh or with the "remove old sync data" task?

 

And also, Imagine I enable "Save artwork into media folders" would it be retroactive or only for new content?

 

I'm wondering if I enable that option and run a library scan, if the server copies over the centralized metadata path to the individual media folders?

 

If not, would be a nice feature request.

 

Something like implementing a hidden-option after "save artwork into media folders" that would make the server read the .nfo path and backing it up or simply copying it over to the media folders.

 

what do you think?

Edited November 22, 2018 by djhifi

[Happy2Play 8356]

ahhhh! Finally! Thanks.

 

It does so on each library scan and also metada data refresh or with the "remove old sync data" task?

 

And also, Imagine I enable "Save artwork into media folders" would it be retroactive or only for new content?

 

I'm wondering if I enable that option and run a library scan, if the server copies over the centralized metadata path to the individual media folders?

 

If not, would be a nice feature request.

 

Something like implementing a hidden-option after "save artwork into media folders" that would make the server read the .nfo path and backing it up or simply copying it over to the media folders.

 

what do you think?

 

Only on New content.

[djhifi 55]

Only on New content.

 

Thanks.

Damn

The feature I suggested, is something too out of reach to think of?

[Luke 37248]

I think there might be an old feature request already to support reading paths from nfo files.

[djhifi 55]

I think there might be an old feature request already to support reading paths from nfo files.

 

Yes. During my fórum searches I've found a very angry user asking for a bulk backup implementation on the backup/restore plugin and you answering it would get big, etc etc.

 

If you're not referring that thread then I don't know.

 

But I wasn't asking for something like that. Just the ability to read from the paths in the .NFOs using the itemID stored within SQL, comparing them to the 32-bit strings that give name to individual folders and  then transitioning that artwork to the respective media files folders if the user selected such option in the dashboard. Something like a hidden switch.

 

I would like to make the change from centralized artwork to stored with media files after all this talk. But I have over 6000 Movies and many TV Shows and the fact that it's only valid for new content turns me off

 

Doable? Should I open a specific request thread just for the record?

Edited November 23, 2018 by djhifi

[Luke 37248]

Sure, you can open a feature request topic. Thanks.

[djhifi 55]

Sure, you can open a feature request topic. Thanks.

 

Done: https://emby.media/community/index.php?/topic/65645-ability-to-enable-the-server-to-read-arwork-path-files-from-nfos-and-copying-them-over-media-files/

 

You can close this one if you'd like. It's purpose is now fully exhausted.

 

Thanks.

Edited November 23, 2018 by djhifi