Jump to content

Locally cache metadata


Recommended Posts

Is the option to follow the three tier cache>metadata>remote-share-art scheme that online scraping uses being considered at all for offline scraped data as discussed here?

 

If not, I am not sure there is any value in hammering out any more details or symptoms of the existing two tier `cache>remote-share-art` system here when the core architecture wont do whats needed.

Better we expend effort looking at ways to work around the shortcoming for those that feel it worth the effort. I have some ideas but for most, if they manage to wade through this "wall of words", the best solution would be to simply not use offline art scraping at all with Emby and use URL only nfos as a middle ground.

Link to post
Share on other sites
cayars

I don't understand what the 3 tier thing is you're referring to.

There is nothing wrong with Emby but you need to set it up for YOUR environment. If you don't power down your disc and they are local then you can store your graphics with the media.

If you use remotely mounted discs or let them spin down then DO NOT store your graphics on the disc but allow Emby to manage this for you in the Meta-Data folder that you can put where ever you want it to be on any disc.

You can have the cache and meta-data folders on the same drive or different depending on your needs.

What doesn't make sense is putting the graphics on a disc you know won't be quickly available.

Right now you're trying to fight the system and you're loosing.

Link to post
Share on other sites

 

8 minutes ago, cayars said:

I don't understand what the 3 tier thing is you're referring to.

I will do my best to explain.

When scraping art metadata from an internet source Emby employs a 3 tier system "internet > metadata folder in Emby install > cache folder in Emby install"

When scraping art metadata from non internet source e.g. images beside media, Emby employs a 2 tier system " internet> cache folder in Emby install"

I am asking if Emby plans to Emby employ the 3 tier version for both.

 

I appreciate its much easier for you to just to ask people to work around it outside of Emby but can we (and I mean this with all humility)  just for a moment accept that perhaps you dont fully understand how every storage design in the world works and that there may be architectures and technologies you haven't been exposed to where functions other than Emby access the same storage especially at scale. If we can accept this we can quickly realise that just changing everything to suit just Emby is not always an option.

I really do welcome the advice and insights but it feels like you have decided already this is how it all works for you so thats how it must work for everyone.

 

Link to post
Share on other sites
cayars

No that is not how it works.

A) use the graphics in the media folder if existing
B) Pull the artwork and put it EITHER with your media or in the meta-data folder. It can either pull the graphics at time of media discovery OR only on first use (configurable per library). FULL STOP

When Emby apps display graphics for any media item it gets pulled from the cache.  If the data is not already rendered in the cache it get's pulled from meta-data folder, rendered, saved in cache location and returned to the app.

So if you want to have discs that spin down when not in use BUT want the server to be able to display them you can't have the graphics on a disc not spinning (doesn't make sense) so you NEVER place graphics with media.  You set your Emby meta-data & Cache folders on a disc that doesn't sleep.  DONE

What you can't do is have the SOURCE of the graphics on a disc not spinning and expect to be able to load them as that doesn't make much sense. If you want to speed things up on your system turn on for each library the download in advance graphics option, do not store graphics with media.  Then delete all graphics from the library mount points and refresh the meta-data.  DONE

 

  • Like 1
Link to post
Share on other sites

 

13 minutes ago, cayars said:

No that is not how it works.

....

When Emby apps display graphics for any media item it gets pulled from the cache.  If the data is not already rendered in the cache it get's pulled from meta-data folder, rendered, saved in cache location and returned to the app.

 

If this is the case then there is a bug because it can be demonstrated that once the cache purge happens, if you have art beside the media, it will bypass the metadata folder and re-cache it direct from the original art beside the media.

This has been observed previously as seen by this comment:

On 11/25/2020 at 4:57 PM, GrimReaper76 said:

It does use it for the Libraries that don't store artwork in media folders. For those that do, it doesn't because that art is not present in metadata folder. 

...

and consideration that it is not clear cut given by this post

On 11/25/2020 at 12:55 PM, Luke said:

It's possibly on demand.

It is also worth pointing out that Emby is just one app in a media centre ecosystem and we have be be careful when suggesting solutions like "dont ever have art beside media"  without realising that will have an impact elsewhere.

I am pretty much reserved to the fact that Emby wont entertain any enhancements here and thats ok but the previous posts brings us full circle to suggest it may still actually be a bug (although I dont honstly think so I think is it more likely the last post claiming the 2 tier model is wrong was in error).

Link to post
Share on other sites
cayars
22 minutes ago, xe` said:

 

If this is the case then there is a bug because it can be demonstrated that once the cache purge happens, if you have art beside the media, it will bypass the metadata folder and re-cache it direct from the original art beside the media.

This is what you're not following.  Meta-Data folder is where ever the graphics resides.  It can be in the CENTRAL meta-data folder OR with the media.  It can be different for EVERY piece of media you have on your system.  If you have graphics with your media THAT FOLDER IS THE META-DATA folder for that media.

To complicate this a piece of media can have graphics stored in both the central and media folder.  If for example you use a 3rd party app that downloads some but not all the graphics Emby uses then Emby will use what you provide from the media folder BUT depending on your library settings download other graphics and store in according to your settings.

So again you adjust your libraries to NOT place graphics with media.  You manually remove graphics from the media libraries and rescan/refresh libraries to force it to put the graphics in the system defined meta-data folders.

You're overthinking this.  It is very clear cut how it works.

  • Like 1
Link to post
Share on other sites
PenkethBoy

Yep - there is no bug that i see and emby works the same way as @cayarshas explained

The simple issue here is you have a fixation with HDD spinning up to provide data - get passed that issue and you would not care about where the data comes from

Insist that you have disks spin down and .......

  • Like 1
Link to post
Share on other sites

OK everyone thanks for the replies. this confirms that Emby uses a two different tiering models as I posted last (and was subsequently incorrectly quoted as not true).

 

The subtly of my points is now getting lost, with support replies skimming my posts and considering only very basic old and small scale storage designs which admittedly represent the majority of current users and what they will be used to. When seen in that context my observations on Emby back-end storage assumptions, whilst still 100% true, comes at such a small cost as it is easy to waive off as minuita.

@Luke If nothing else comes of this long thread please ponder this quote and what it really means next time you are thinking about this subsystem.

19 hours ago, xe` said:

It boils down to this basic, in my opinion, incorrect architecture assumption:

When Emby scrapes metadata from the internet it preserves a duplicate copy in the local `metadata` folder because online sources are unreliable, slow and expensive.

When Emby scrapes metadata from images saved beside the media files it does not duplicate a copy locally because it assumes this data is already reliable, fast and cheap. This is a demonstrably bad assumption especially at scale.

 

Mentioning disk spin down didnt help the matter as whilst I hoped it would give an understandable example at small scale for demonstration purposes it unfortunately backfired locking everyone into a self affirming way of thinking that their idea of storage design is the only way ... and not what I hoped... to show that there are other factors to take into account at scale.

I thank everyone for all the advice and whilst it did get quite passionate I certainly know more now about how Emby works than I did when I started.

I will leave it there but if you would like to expand your knowledge and have a sneak peak at the future have a look into what vendors are doing with on-prem cold storage and related tiering technologies. Like F1 it will trickle down soon enough to the prosumer.

Good luck.

Link to post
Share on other sites
PenkethBoy
When Emby scrapes metadata from images saved beside the media files it does not duplicate a copy locally because it assumes this data is already reliable, fast and cheap. This is a demonstrably bad assumption especially at scale.

This is a very valid reason - and nothing to do with scale

only time this might be valid if you are using cloud storage as your primary location - but other options are available to make this a minimal issue.

Link to post
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now
×
×
  • Create New...