Jump to content

Locally cache metadata


Recommended Posts

Crusher21

My library spans across multiple drives, and as I'm scrolling through my library, much of the metadata (such as the movie/TV show artwork) will be blank for about 20 seconds while it waits for my drives to spin up. I know that the metadata is stored locally on the respective drives, but is there a way to make Emby store a copy locally on the server so that everything is always loaded when sifting through my libraries?  

Link to post
Share on other sites
cayars

Hi, You have a few options.  First you can store the metadata in a specific location set by the meta-data field found at LIBRARY-ADVANCED, but this is only used if you don't turn on the option in each library to store the info with the media.

However, Emby uses a CACHE to hold rendered images which you can set the location in the SETTINGS menu.

One of the things you can do is pre-populate the cache and force it to render the images. The manual way for example is load up your movies and then scroll page by page until you've viewed every movie.  Same for TV Shows.

That will make the main navigational images cached and make viewing your libraries much easier.  However when you get to detail screens or TV Show season the particular drive will need to spin up.

Personally I don't allow my drives to spin down as I find they die quicker that way especially with multiple drives in a cabinet and the spin up/spin down causes vibrations that don't happen when they constantly run.  I don't worry about the cost to run those drives because it's actually not that much and cheaper then the replacement costs overall.  So for me it's win/win leaving them running as everything is quicker.

But that's just me.

Link to post
Share on other sites
On 11/19/2020 at 5:00 PM, cayars said:

Hi, You have a few options.  First you can store the metadata in a specific location set by the meta-data field found at LIBRARY-ADVANCED, but this is only used if you don't turn on the option in each library to store the info with the media.

However, Emby uses a CACHE to hold rendered images which you can set the location in the SETTINGS menu.

One of the things you can do is pre-populate the cache and force it to render the images. The manual way for example is load up your movies and then scroll page by page until you've viewed every movie.  Same for TV Shows.

That will make the main navigational images cached and make viewing your libraries much easier.  However when you get to detail screens or TV Show season the particular drive will need to spin up.

Personally I don't allow my drives to spin down as I find they die quicker that way especially with multiple drives in a cabinet and the spin up/spin down causes vibrations that don't happen when they constantly run.  I don't worry about the cost to run those drives because it's actually not that much and cheaper then the replacement costs overall.  So for me it's win/win leaving them running as everything is quicker.

But that's just me.

I appreciate the reply! I do indeed have it set to hold the metadata in the same location as the media, so I know this is largely the reason for the problem. Is there a way to increase the maximum size of the cache? Perhaps that would be a workaround.

Out of curiosity, how do you prevent your drives from spinning down? 

I'm in an interesting situation where I find that external drives are much cheaper per terabyte, so I have a herd of external drives. I know that I can rip them from their enclosures, but I prefer not to do that until the warranty period is over. In terms of vibrations, are you aware of any sort of physical apparatus that one can purchase to dampen vibrations? Also, do you know if there are any server racks on the market that allow you to insert a sort of "shelf"? The idea being that I need a flat surface for external hard drives that looks decently professional. Sorry I know this is quite off-topic

Link to post
Share on other sites

There is no limit of the cache.  It will be what ever size it needs to be to hold the images of your system.

Typically settings in the OS you are using or on the drive itself (some drives).
I also run a program on Windows called KeepAliveHD which can read and/or write to a drive every X period of time therefore keeping the drive from going to sleep.

I try to avoid external drives but instead use higher quality "internal/NAS" drives but these days the differences in drives is not as much as it once was. A good case IMHO is the best way to avoid vibrations.

  • Like 1
Link to post
Share on other sites
On 11/19/2020 at 10:00 PM, cayars said:

...

However, Emby uses a CACHE to hold rendered images which you can set the location in the SETTINGS menu.

One of the things you can do is pre-populate the cache and force it to render the images. The manual way for example is load up your movies and then scroll page by page until you've viewed every movie.  Same for TV Shows.

That will make the main navigational images cached and make viewing your libraries much easier.  However when you get to detail screens or TV Show season the particular drive will need to spin up.

...

Can I just ask for some clarification on this.

The way I interpret you advice is that if you set the following to "on":

Quote

 

Download images in advance

By default, most images are only downloaded when requested by an Emby app. Enable this option to download all images in advance, as new media is imported. This may cause significantly longer library scans.

 

 
 
and the following to "Never"
 
Quote

Generate video preview thumbnails:Video preview thumbnails provide live updates while seeking in supported apps. Thumbnail generation may take a long time, cause high CPU usage and consume additional disk space.

 

and manually page through the main views for movies and tvshows then Emby will cache your artwork and not read them on the fly when needed allowing drives to stay spun down.
 
I have tested this and it does not seem to actually work.
 
Only one of my sources uses local fanart and I can clearly see movies art from that source visually slow to load, hear HDD spin ups and logs show some art taking 10+ seconds until drives spin up.
Link to post
Share on other sites
PenkethBoy

remember the files cached expire every 30 days - so you have a rolling cache update happening a lot of the time

and no you cant change the cache expiry period

 

  • Like 1
Link to post
Share on other sites

Thanks for the reply. I am confident it is not that as the movies view is used extensively by the family.

It is easy enough... if not very laborious... to replicate and test. I am happy to do this if what I am describing isnt expected behavior.

Edit: can i also clarify "remember the files cached expire every 30 days". Does this expiration scheme apply to all cached metadata art or only that which comes from a local source proving artwork as image saved beside the content?

Edited by xe`
Link to post
Share on other sites
14 minutes ago, PenkethBoy said:

remember the files cached expire every 30 days

what is the purpose that the cache is redone every 30 days ? 
the typical data is likely never change at all and very less likely in a 30day span

my "old" data for example is static for now 5+ years

recaching is nice and all but at big collections its a real pain

  • Like 1
Link to post
Share on other sites

You really shouldn't notice this happening if you have the job set to run at night. By doing a constant recache ever 30 days it keeps you data from imploding on itself ever growing larger and larger.  People have shows movies they add watch and remove from the system.  If you have live tv setup you get a lot of graphics from the guide that you don't need kept for ever, etc

Link to post
Share on other sites
8 minutes ago, cayars said:

You really shouldn't notice this happening if you have the job set to run at night. By doing a constant recache ever 30 days it keeps you data from imploding on itself ever growing larger and larger.  People have shows movies they add watch and remove from the system.  If you have live tv setup you get a lot of graphics from the guide that you don't need kept for ever, etc

By job do you mean the setting?

Quote

Automatically refresh metadata from the internet:Enabling this option may result in significantly longer library scans.

 

Link to post
Share on other sites
Happy2Play
3 minutes ago, xe` said:

By job do you mean the setting?

 

Sure have a looked at ScheduledTaskService in the API.

Spoiler

{
    "Name": "Cache file cleanup",
    "State": "Idle",
    "Id": "241d4fcb19a1d557ee62428e411da609",
    "LastExecutionResult": {
      "StartTimeUtc": "2020-11-24T03:35:44.6706465Z",
      "EndTimeUtc": "2020-11-24T03:35:46.0827571Z",
      "Status": "Completed",
      "Name": "Cache file cleanup",
      "Key": "DeleteCacheFiles",
      "Id": "241d4fcb19a1d557ee62428e411da609"
    },
    "Triggers": [
      {
        "Type": "IntervalTrigger",
        "IntervalTicks": 864000000000
      }
    ],
    "Description": "Deletes cache files no longer needed by the system",
    "Category": "Maintenance",
    "IsHidden": true,
    "Key": "DeleteCacheFiles"
  },

 

Link to post
Share on other sites

OK will dig into this some more tomorrow as I am getting quite confused. For example the setting `Automatically refresh metadata from the internet` doesnt immediately feel intuitively relevant to scraping local non internet art stored along side the media files.

Also its not obvious if the trigger for a cache being old is based on last access time i.e. when a user requests the image, last update time i.e. when the upstream metadata provider was last updated or some combination of both.

I also keep talking about art and images but "Deletes cache files no longer needed by the system" might also refer to xml from the metadata provider.

Link to post
Share on other sites
Happy2Play
3 minutes ago, xe` said:

I also keep talking about art and images but "Deletes cache files no longer needed by the system" might also refer to xml from the metadata provider.

I don't know all the details, but anything in the cache folder over 30 days old gets purged is my understanding.

  • Like 1
Link to post
Share on other sites

First a thanks for everyone so far, this is a longer post but it contains some good information.

 

Much of the confusion and responses have been been based around the use of the term `cache`. In most? other multimedia projects the art `cache` is the location where the system holds all the art. This is not true in Emby where the cache is a separate dedicated location to keep an ephemeral 30 rolling day pot of data. In Emby terms the scraped cache is stored in the `metadata` folder and does not expire out automatically.

Since by default the cache folder is on the same storage media as the metadata it implies it contains further optimised content such as previews (as there can be no speed advantage between access time to cache and metadata on same disk). The other option is it is an optional framework to move and store this data on a faster disk for power users.

The logical conclusion then is that Emby refers to the cache when a client requests a screen assets (other than a video) and should that not be available it pulls from metadata, delivers to the client and also populates the cache to be ready to "accelerate" the next time it is requested.

To test this I manually paged the main movie and TV views and left my NAS to spin down disks over night. The next day I accessed these views in Emby again and watched disk spin status. None spun up meaning all data was served from cache or metadata and that original art on disk wasn't accessed. This is expected behaviour.

I repeated the experiment from a system which has not seen access Emby before and again no disk spin up. This is expected behaviour and rules out browser cache.

 

So where does this leave me?...

My current theory is that since 99% of our family use of Emby is via the `Emby for Kodi` addon this extra layer of client side caching is causing Emby to expire out its own cache (i.e. even though the content is requested often as Kodi is serving it locally Emby does not see this and thinks it is all low value and expires it).

This elegantly explains the expiration but it doesn't explain why once the cache is busted Emby spins up disks to rebuild it later. In my understanding this rebuild should use the metadata folder data and not the original art stored beside the video. This is not expected behaviour and is either a bug or a some special case for this kind of art?

The other very confusing thing is that even now the number of cached jpgs in the `cache` folder is lower than the number of covers in the main tv and movie views. This appears to mean that in some conditions cache is never populated which seems especially weird.

 

Link to post
Share on other sites
cayars

There still is a speed advantage of the cache vs metadata directories as these are already rendered to size for the app.

If you set Emby to store the graphics with the media itself then these aren't stored in the metadata folder as you told it not to.  The graphics will then be fetched from the disc holding the media.

The cache is built on use.  So if you've never viewed a media object before it won't be in the cache.

Link to post
Share on other sites
pwhodges
5 hours ago, xe` said:

Much of the confusion and responses have been been based around the use of the term `cache`. In most? other multimedia projects the art `cache` is the location where the system holds all the art. This is not true in Emby where the cache is a separate dedicated location to keep an ephemeral 30 rolling day pot of data.

As a computer term, "cache" means, and always has meant, temporary storage used to improve performance.  It should not be used to refer to permanent storage, and if other projects are doing that they are misusing the term and liable to cause confusion.

Paul

Edited by pwhodges
  • Like 1
Link to post
Share on other sites
58 minutes ago, Luke said:

It's possibly on demand.

Its very hard for me to debug further but all indications are there is something "different" happening when you store artwork next to the media vs rely solely on online scrapers.

Certainly when the cache expires the expected behaviour would be for it to fall back to using the metadata folder rather than the original artwork (or the net result will be a hidden but sizeable and unnecessary hit on disk IO).

Happy to test further if there are any tests I can do but I have run out of ideas that add any more useful detail.

Link to post
Share on other sites

No sorry. Let me summarize since there has been a lot of words to get to this point. Please remember this is anecdotal via NAS IO observation and feedback here only.

 

On a rolling 30 day cycle, based on last access time, all cache items expire. When a cache item expires Emby will refer to its metadata folder the next time a user asks for an asset e.g cover. It will also rebuild that cache item. This all works perfectly when you use online metadata as source.

However when you keep art beside your media (rather than using only online scraper) it seems it no longer works this way. In this scenario it Emby appears will bypass the metadata folder and access the original art directly to refresh the cache on demand.

As I have a lot of disks in an spun down state it is easy for me hear the impact of this in relation to the slow loading of covers in Emby that are using this type of "beside the media" metadata (while all around it the covers that are not of this type load instantly).

 

 

Link to post
Share on other sites
cayars

Hi, Not from my experience, Emby will use the artwork beside your media if present.

I don't think you quite understand how the caching works as your last statement wouldn't have been said.  With looking at the cache folder to see what's there or not there is no way for you to know why it's slow vs fast.  Chances are the image isn't in the cache and that's why it has to wait for the disc to spin up to pull it.  Images in the cache will of course load very quickly and images in Meta-data folder (for those not stored with media) will assuming the disc is already spinning load faster but still need to be rendered and save to cache.

Link to post
Share on other sites

Let me a question to clarify if I am misunderstanding.

When an image is not in the cache should or should it not use the metadata folder to refresh the cache.

Link to post
Share on other sites
GrimReaper76
4 minutes ago, xe` said:

Let me a question to clarify if I am misunderstanding.

When an image is not in the cache should or should it not use the metadata folder to refresh the cache.

It does use it for the Libraries that don't store artwork in media folders. For those that do, it doesn't because that art is not present in metadata folder. 

There's an open similar FR, it has gained some slight traction but remains to be seen whether it'll catch devs' attention. 

 

  • Like 2
Link to post
Share on other sites

@GrimReaper76 thank you. This matches exactly what I detected as unexpected behaviour and assumed was a bug.

I am going to have to radically re think how I use Emby as this hurts a lot. I cant have clients that are simply browsing spinning up an arrays of disks to retrieve a few of images.

Link to post
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now
×
×
  • Create New...