FR: Add image cache to local drive

[aspdend 174]

Just looking at the forum and the note from Luke about starting a feature request on one of the threads and as I was thinking about this recently anyway, thought I'd throw my hat in the ring.

 

I use UnRAID to store my data and I store all my metadata , images etc with my media files. I have been toying with the idea of storing the data in one central location so that HDD's don't need to spin up when I'm just browsing my library. Instead of this, is it possible to implement a feature (switched off by default) that allows a mirrored set of data to be stored on a different drive...

[dcook 265]

You can define the cache folder location already, if you look under the Path's section of the dashboard.

For example mine is: 

 

Cache:

C:\Users\svc_emby\AppData\Roaming\Emby-Server\cache

[Luke 37007]

You can define the cache folder location already, if you look under the Path's section of the dashboard.

For example mine is: 

 

Cache:

C:\Users\svc_emby\AppData\Roaming\Emby-Server\cache

 

That only moves the existing cache. What he's asking for is that we add more data to the cache so that we can reduce access to your media folders.

[dcook 265]

Well if you also define the metadata folder then the only reason to access media folder is to actually start playing the requested video.

 

At least that is how mine is setup, is that not the preferred setup?

 

This way it keeps your media library clean so it does not get filled up with small files that just take up whole blocks of storage, you can have your metadata or cache data on a separate fast SSD drive, and then keep your Media Library lives on slower storage or a NAS

 

With the current options of choosing to save metadata with library or save in its own location, I don't see the point in having another 2nd copy of the same metadata or cached data.  

 

With my metadata and cache folders together being almost 25GB in size I would not want to be forced to store this data in a 2nd location for no reason.

Edited May 16, 2016 by dcook

[Waseh 4]

This would be amazing - I was actually under the impression that the data/artwork cached in the emby metadata/library folder was enough to keep my drives from spinning up, which was why i decided to save it with the media, but this thread made me realise that this was the culprit to slow loading of artwork in the webclient and android app. The disc with the relevant metadata has to spin up first.
My original asumption was based on this thread: http://emby.media/community/index.php?/topic/31747-moving-metadata-out-of-media-library-location/
 

[dcook 265]

This would be amazing - I was actually under the impression that the data/artwork cached in the emby metadata/library folder was enough to keep my drives from spinning up, which was why i decided to save it with the media, but this thread made me realise that this was the culprit to slow loading of artwork in the webclient and android app. The disc with the relevant metadata has to spin up first.

My original asumption was based on this thread: http://emby.media/community/index.php?/topic/31747-moving-metadata-out-of-media-library-location/

 

 

Waseh,

 

If you set your metadata to store not in the media library your library does not have to spin up until you actually try to play a video.  

 

It sounds like he wants a 2nd copy of the same metadata or cached data, which I still don't understand the point when you can already define where it is to be stored.

[Waseh 4]

I know (and i originally had it like that) but i like keeping the metadata with the media - It's more manageable like that i think, and it makes sure that if anything were to happen to my emby installation the artwork i chose would be unaffected.

Edited May 16, 2016 by Waseh

[dcook 265]

I would argue the opposite that it is less manageable, if you keep your media library clean, and all your metadata on its own SSD, you have better performance and more easy to manage as its all in one place.

 

Regardless, I don't agree that we should store a 2nd copy of the metadata or cache data.

 

Perhaps that could be a EMBY Plugin, but I don't think it should be in the core.

Edited May 16, 2016 by dcook

[Happy2Play 8242]

In the end it is about whether your drives are on 24/7 or spin down.  That factor will determine the performance issue you are talking about.  I guess I have never seen it do my drives being always on, with metadata stored with media.

[Waseh 4]

@@dcook - I get what your saying, but having artwork with the media also makes it possible for programs outside the Emby ecosystem to benefit.
I for one would appreciate a cache of the artwork so i could benefit from the ssd my emby installation resides on.
I might consider moving it all back to the ssd again, but i just got the idea that it was recommended to have it with the media anyway (i think ebr might have mentioned it at some point)

Edited May 16, 2016 by Waseh

[Happy2Play 8242]

It is a lot easier to do a clean installation with metadata with media.  The provider sites doen't get hammered having to download everything again.

[dcook 265]

It is a lot easier to do a clean installation with metadata with media.  The provider sites doen't get hammered having to download everything again.

 

I agree that if you are doing reinstallation or moving from old serve to new server it would be easy to keep the existing metadata that you have already instead of downloading it again.  

 

However I disagree that the metadata should be stored with the media library.  As discussed already there is much better performance if you have your metadata and cache on SSD Drive, and keep your media library on NAS or some other RAID storage.

 

A better solution would be to fix EMBY so that if you do a reinstall or move to new server and you have your metadata already setup in a separate folder, when you configure your EMBY on the new server it will use that metadata folder instead of downloading again.  

 

I should not have to keep metadata with the media library just to be able to move it from old server to new server

 

And as I mentioned already I strongly disagree that there is any benefit in having another 2nd copy of the same data, as in my case it would be over 25GB that is now wasted.

Edited May 16, 2016 by dcook

[Happy2Play 8242]

Is it just coverart that already caches every image in the cache folder already?  And what are all the already cache images used for then if it is looking for the media also?

[aspdend 174]

The point of the feature request was to retain all the metadata etc with the media files and to have a separate duplicate for rapid browsing of the interfaces without waking the remote drives that the media are stored on until I want to actually watch something. This will improve my use of Emby because everything will be quickly accessed but I still retain the metadata etc in the media locations.

[dcook 265]

If you uncheck "save artwork and metadata into media folders" you won't have the issue you are describing.

 

You are asking for additional feature to avoid a problem that can already be avoided by not storing your metadata in your library.

 

If this is going to be added, then this feature should only apply if you have the "save artwork and metadata into media folders" is selected and even still should be optional additional feature, as I am sure there are others like myself who don't want to waste (25GB in my case) for a 2nd copy of same data.

 

Also before spending time/resources on adding new features, Luke can we first get the existing issues fixed?

The collections dynamic slices for example? This has been broken for months

Edited May 17, 2016 by dcook

[aspdend 174]

I know exactly what you are saying, but I prefer my artwork and metadata to be in the media folders, but I want quick access - hence the Feature Request as Luke suggested in another thread.

 

I know it can be avoided, but that puts everything in the cache which I don't want - hence it's a feature I want to see - if there's only two of us that want it I can appreciate the fact it will almost never get implemented, but if I ask and everybody thinks its a great idea as well, then lucky me! For me 25Gb of duplicate space isn't an issue - my main storage array is 38Tb, not sure how big the artwork and metadata would be - but substantial

[kingom 5]

i have EXACTLY the same problem!

 

I use unraid and i like my drives spin down when not used the whole day.

Then an emby request comes along and you have to wait up to a minute to see something return because it takes so long spinning up all the disks in the NAS and accessing the metadata.

 

The point with "do not store metadata withhin files" doesn't come into play at my setup because when i add files to the NAS they already have all metadata within them - prescraped with TMM. So Emby hasn't to do anything metadata wise. So i have "store metadata withhin files" disabled but it doesn't work anyway.

For Setups where you have prestored metadata with a slow disk setup you would really need  the option to cache all metadata explizit with emby cache so you not got any performance issues when accessing the data.

 

Also it would be much easier on the NAS disks if you cache as much as possible on a cheap little cache ssd instead of expensive storage.

This also applies for systems where the disks are always up. Everytime not accessing the array will extend the lifetime of it.

 

 

So PLEASE add this!

Edited March 25, 2018 by kingom

[Luke 37007]

Yes that's a good idea for the future. Thanks.

[tdeblock 1]

I have the same issue. has the feature been developed?

[Luke 37007]

Not yet.

[Carlo 4330]

I want my meta-data stored with my media but I would prefer it cached on one of my SSD drives.

I originally thought this is the way Emby worked. Otherwise it's not really a cache now is it? 

 

I keep all drives spinning 24/7 and also use an OS cache to SSD drive so I get this functionality outside of Emby anyway but would prefer it built in.

[Luke 37007]

I originally thought this is the way Emby worked. Otherwise it's not really a cache now is it? 

 

Images are cached in the Emby program data, but not until after they are first requested. in other words, they are not preemptively cached in an attempt to reduce I/O to your media folders.

[kingom 5]

maybe any updates on this?

[Luke 37007]

2 hours ago, kingom said:

maybe any updates on this?

Hi there, what exactly are you asking for?

[kingom 5]

2 hours ago, Luke said:

Hi there, what exactly are you asking for?

You said:

Quote

Yes that's a good idea for the future. Thanks.

You said you maybe consider implementing this request.

Maybe you could  take it into the development pipeline?

Would much appreciate it. Every time a user searches something or goes into movies or tv shows overview, emby stands still for 1-2 minutes.

I and my users learned to live with it but is it really necessary? Also the disk MTBF value is one that goes up the fewer the disk is accessed.

Please consider this!

 

PS: Emby App for LG TV is absolutely terrible. When you have 4000+ movies, performance goes down. Browsing movie overview is at a klick per 3-4 seconds at max! Really terrible pls fix.