Locally cache metadata

[GrimReaper 3331]

1 minute ago, xe` said:

@GrimReaper76 thank you. This matches exactly what I detected as unexpected behaviour and assumed was a bug.

I am going to have to radically re think how I use Emby as this hurts a lot. I cant have clients that are simply browsing spinning up an arrays of disks to retrieve a few of images.

I also wondered why were we ever forced to choose, in my ideal world it would be stored in BOTH locations (HDD space is really cheap nowdays), however, only way I cold sort that is to run two instances of Emby server with different tasks: my "main" server which is built for speed - database only, nvme SSD, cache and data folders junctioned on RAMdisk, and my "secondary" server which uses same Libraries with artwork saving to media folders. Best of both worlds. Downside is that for any manually curated item you need to do it on both servers but since i seldom do that, works well enough for me. 

Cheers

[xe` 42]

Assuming I am not happy for users to cause arbitrary disk spins or bypass the fast local metadata folder to call art from remote slow/links disks simply by casually scrolling in Emby (which no one should be OK with TBH) I am left with two options:

• Stop using in place fanart completely
• Minimise the impact by forcing all in place art onto a single remote NAS side helper disk

The second option is possible for me but probably not for most users and it really just does "badly" what the Emby `metadata` folder should be doing.

 

To the casual observer this long thread may all seem like an edge case but essentially what we are saying is anyone that uses in place fanart does so at a hidden unavoidable performance cost which is funnily what many will be using in place fanart to try and avoid in the first .

I strongly recommend this is addressed at the architectural level.

[PenkethBoy 2063]

what do you mean by this?

essentially what we are saying is anyone that uses in place fanart does so at a hidden unavoidable performance cost

[xe` 42]

8 minutes ago, PenkethBoy said:

what do you mean by this?

essentially what we are saying is anyone that uses in place fanart does so at a hidden unavoidable performance cost

Unlike metadata scraped from the internet should you choose to use metadata stored with the video files when content in the Emby cache expires it will rebuild the cache by pulling it from metadata stored with the video files.

For most people this will be slower than rebuilding from the local Emby metadata folder as metadata stored with the video files is typically on remote NAS server, on slower bulk storage disks or spun down.

[PenkethBoy 2063]

well you have the choice to not store with your media - and all images will be in the local metadata folder

or dont spin down your disks - its not saving you much power cost and HDD are designed to be on all the time - spinning up and down actually wears them out quicker as its the highest strain on the mechanical parts

[xe` 42]

I am not sure making huge sweeping statements about hardware helps us conclude this. I appreciate and agree that what you say is often true but it is not always true and it is generally never a good idea for software design to assume everyone has the same hardware design as you are used to.

 

It boils down to this basic, in my opinion, incorrect architecture assumption:

When Emby scrapes metadata from the internet it preserves a duplicate copy in the local `metadata` folder because online sources are unreliable, slow and expensive.

When Emby scrapes metadata from images saved beside the media files it does not duplicate a copy locally because it assumes this data is already reliable, fast and cheap. This is a demonstrably bad assumption especially at scale.

 

 

 

[PenkethBoy 2063]

i'm not the one making huge sweeping assumptions

you are misleading people with some of your statements

[CvH 14]

So "the cache" is actually a big performance hit (besides useless hdd spin up, delay through processing etc) in basically every realistic usage scenario if the metadata is besides the actual media?

I can't really make up a scenario (besides ultra power user or tiny collections) where the cache improves here anything. I am also not sure why the cache get dropped every 30 days, makes no sense at all. Kodi for example drops the cache if I force it or it detects a change at the source for the cache. Besides that the cache gets never dropped because it would be, surprise, a major performance hit when scrolling through the collection.

There is likely a reason (i have no idea) why the cache is dropped if you get the data from the internet, but there is no real reason to drop it at all if all your media and metadata is local.

At kodi I cache a single time, done. A emby I need to cache every time I access something, sounds broken if you ask me.

And to be honest 30 days is not enough to ensure you click through the entire collection just to keep the cache.

[GrimReaper 3331]

I feel this discussion has become quite convoluted, with different views of how the cache actually works. So, for sake of good order, I'll throw my understanding and hoping some of the devs might shed some light if incorrect. 

Cache is used to hold prerendered artwork (with prerequisite of it being requested in the first place, i.e. library browsing) for serving different clients, regardless whether that artwork is stored locally (database only, metadata folder) or with media. So I absolutely disagree that it doesn't improve anything, in contrary, it speeds up browsing considerably, with lesser gains for local metadata and huge gains with art stored with media. Without it, art from media folders would have to be pulled each and every time it is requested. Issue arises once that cache is dumped, whether by intentional deletion or expiration, and needs to be rebuilt. That goes fast enough for items in local metadata folder, and for art in media folders it has to be pulled from there as there is nowhere else to pull it from. Hence HDD spins and slow rendering times. Still better than doing it every time needed. 

As @Luke already stated, those 30-day dumps are there to prevent it imploding on itself by growing ever larger. Not all of us use the server in the same way, I personally have no issue with it growing as much as it needs to. But i don't do LiveTV recordings, no home videos, no pictures etc. I understand that lots of ppl do and do a lot of deleting also. Those deleted items should not be referenced in cache, hence the dumps.

And as stated above, I would really prefer metadata folder to store ALL artwork and copies to be saved in media folders, with serving logic being Cache>(if art unavailable) fallback to Metadata folder>(if art unavailable) fallback to Media folders. 

Cheers

 

 

[Alfsvag 2]

 

As I have tons of custom .nfo files from a Kodi migration I want to reuse them. However, as I don't want my drives to spin up when accessing metadata and the cache currently can't hold it all,  as a temporary world around, would it be possible to disable the .nfo metadata so that a metadata database gets created on my SSD that scraps the pre-existing metadata from my .nfo files?

Or would the .nfo files be ignored? 

Edited November 26, 2020 by Alfsvag
Clarification

[GrimReaper 3331]

1 minute ago, Alfsvag said:

As a temporary workaround, wouldn't it be possible to first save all the metadata as .nfos with the media and then disable .nfo metadata so that a metadata database gets created on e.i. a SSD? Assuming that the new database would scrap all the info from the previously created .nfos. Not ideal of course as metadata added after this wouldn't be stored as .nfos, but would this work?

AFAIK database would be created regardless of your saving to .nfos or not, they are there only for Kodi (or other apps) compatibility. So you might as well leave it enabled as it'll make no difference on subsequent server behaviour. 

[Alfsvag 2]

2 minutes ago, GrimReaper76 said:

AFAIK database would be created regardless of your saving to .nfos or not, they are there only for Kodi (or other apps) compatibility. So you might as well leave it enabled as it'll make no difference on subsequent server behaviour. 

Wait, but isn't the whole reason why the drives spin up to access metadata stored there? Something that wouldn't happen if there were no metadata there which would be the case if .nfo files were disabled and all metadata were stored somewhere else in a database?

[GrimReaper 3331]

Just now, Alfsvag said:

Wait, but isn't the whole reason why the drives spin up to access metadata stored there? Something that wouldn't happen if there were no metadata there which would be the case if .nfo files were disabled and all metadata were stored somewhere else in a database?

They spin up to access artwork. Images. 

[Alfsvag 2]

6 minutes ago, GrimReaper76 said:

They spin up to access artwork. Images. 

Yes, but AFAIK those are part of the metadata that can also be saved elsewhere and not with the media.

 

edit: Should probably have clarified that I have that data too from my previous setup, not just the .nfo files. So the idea was simply to have all that scraped and stored elsewhere.

Edited November 26, 2020 by Alfsvag

[GrimReaper 3331]

Just now, Alfsvag said:

Yes, but AFAIK those are part of the metadata that can also be saved elsewhere and not with the media.

Yes, it is separate option under Library settings. .nfos gets read only upon Library (re)build and, if I recall correctly, gets precedence over internet scraped data. Afterwards, it's use is practically nonexistent for current setup. I think it does get written occasionally by Emby on metadata change, again, for compatibility only. 

[roaku 797]

11 minutes ago, GrimReaper76 said:

AFAIK database would be created regardless of your saving to .nfos or not, they are there only for Kodi (or other apps) compatibility. So you might as well leave it enabled as it'll make no difference on subsequent server behaviour. 

If you directly edit an nfo file, that change gets propagated to the db on the next library scan.

That suggests they serve a larger role than just compatibility and the drive they're on will be accessed.

[Alfsvag 2]

5 minutes ago, GrimReaper76 said:

Yes, it is separate option under Library settings. .nfos gets read only upon Library (re)build and, if I recall correctly, gets precedence over internet scraped data. Afterwards, it's use is practically nonexistent for current setup. I think it does get written occasionally by Emby on metadata change, again, for compatibility only. 

I suppose that would work for me then as my drives that are filled up will very rarely get altered so if I can scrap the whole content of those drives those should no longer spin up when accessing the metadata and artwork!

However, I don't think the .nfo files would be maintained anymore when storing to .nfo is disabled. So not ideal, but that's why I frased it as a temporary solution.

Edited November 26, 2020 by Alfsvag

[GrimReaper 3331]

Just now, roaku said:

If you directly edit an nfo file, that change gets propagated to the db on the next library scan.

That suggests they serve a larger role than just compatibility and the drive they're on will be accessed.

Which is basically same as metadata change, database gets updated and i don't think it has any use afterwards.

[Alfsvag 2]

A final question on this matter, when emby finds a .nfo file does it still check scrapers for additional info and if so, can that be disabled when a .nfo file is detected? (Kodi have such a feature)

[xe` 42]

7 minutes ago, roaku said:

If you directly edit an nfo file, that change gets propagated to the db on the next library scan.

 

fwiw I believe that is true of XML nfos and not URL nfos

[roaku 797]

4 minutes ago, xe` said:

fwiw I believe that is true of XML nfos and not URL nfos

I only have experience with XML based, so let's limit what I said to that format.

[GrimReaper 3331]

18 minutes ago, roaku said:

If you directly edit an nfo file, that change gets propagated to the db on the next library scan.

That suggests they serve a larger role than just compatibility and the drive they're on will be accessed.

 

On 1/30/2019 at 7:40 AM, Luke said:

You would simply turn off nfo metadata saving. The downside of this though that if you ever use the emby metadata editor to make changes, they will not be written to the nfo file, and they could end up eventually getting lost altogether the next time the server re-imports data from the nfo. That could happen if the nfo ever changes or if you refresh metadata manually on something.

 

[GrimReaper 3331]

16 minutes ago, Alfsvag said:

A final question on this matter, when emby finds a .nfo file does it still check scrapers for additional info and if so, can that be disabled when a .nfo file is detected? (Kodi have such a feature)

 

[xe` 42]

It is worth noting in the context of this thread that if you create a URL only nfo (so this is a flat txt file called tvshow.nfo or movie.nfo) with only the IMDB or tvdb link in it then Emby will use this as an anchor.

What does this mean? It means that instead of guessing what the movie is it will use the id you gave it but in all other ways it will act like a normal non nfo internet metadata lookup. If you dont have artwork saved next to the movie you wont suffer from the issues we discussed in this thread. Current this is the best compromise as it allows you to get 100% perfect matching for ever more. Kodi also supports this as well as TMM. None of the tools can make this format though they all want to use XML.

[Carlo 4331]

4 hours ago, xe` said:

I am not sure making huge sweeping statements about hardware helps us conclude this. I appreciate and agree that what you say is often true but it is not always true and it is generally never a good idea for software design to assume everyone has the same hardware design as you are used to.

 

It boils down to this basic, in my opinion, incorrect architecture assumption:

When Emby scrapes metadata from the internet it preserves a duplicate copy in the local `metadata` folder because online sources are unreliable, slow and expensive.

When Emby scrapes metadata from images saved beside the media files it does not duplicate a copy locally because it assumes this data is already reliable, fast and cheap. This is a demonstrably bad assumption especially at scale.

This has been proven over the years by published documentation from many sources including data centers. Dell, Amazon, IBM, Microsoft have all done testing and evaluation of HDD disc in various power managements strategies.  Overall costs are lower letting them spin full time.  The minimal cost saved in electricity is outweighed by more often replacements.

Now if you have a PC that's only used once a day or something it's likely a different story but for discs being used a few times a day it's concluded it's better to leave them constantly spinning.

3 hours ago, CvH said:

So "the cache" is actually a big performance hit (besides useless hdd spin up, delay through processing etc) in basically every realistic usage scenario if the metadata is besides the actual media?

False.  The whole point of the cache is to IMPROVE loading speed and it certainly does NOT cause a performance hit. You're making assumptions over and over of how you think things work but they don't work the way you think.

56 minutes ago, xe` said:

It is worth noting in the context of this thread that if you create a URL only nfo (so this is a flat txt file called tvshow.nfo or movie.nfo) with only the IMDB or tvdb link in it then Emby will use this as an anchor.

Emby will use the NFO for ID purposes then follows it's normal rules depending if you have graphics already on disc named properly.  It also depends on your meta-data settings what it uses to fetch missing info.