Completed Preemptively cache images

[Exor85 6]

+1

[mgworek 121]

I thought for some reason Emby used to do this. I am not sure why but Emby UI on web or ios has been slow for the past few months for me. When I first load it up it takes a while for the posters to show up, even ones that were there yesterday. Sometimes I have to hit refresh in the browser and then they are there. If I didn't hit refresh, I would have to wait 5 or so extra seconds for them to show up automatically. I know my drives go to sleep and I have to wait for them to wake up but it never used to be slow. 

 

UPDATE:

There is a chance I fixed my slow down. I did start noticing it when I upgraded my server. I gave my VM that is running Emby a total of 8 cores, up from the pervious 2 or 3 cores it had. I never touched the ram though. It was still using 6 gigs, I gave it 12 gigs just now and I believe it is running faster. I haven't used firefox in months. I loaded up the web UI and it was quickly loading posters for movies and I was scrolling on down without a hiccup besides a quick second sometimes for posters to catch up with me.

Edited August 5, 2021 by mgworek

[Luke 37020]

1 hour ago, mgworek said:

I thought for some reason Emby used to do this. I am not sure why but Emby UI on web or ios has been slow for the past few months for me. When I first load it up it takes a while for the posters to show up, even ones that were there yesterday. Sometimes I have to hit refresh in the browser and then they are there. If I didn't hit refresh, I would have to wait 5 or so extra seconds for them to show up automatically. I know my drives go to sleep and I have to wait for them to wake up but it never used to be slow. 

 

UPDATE:

There is a chance I fixed my slow down. I did start notice when I upgraded my server. I gave my VM that is running Emby a total of 8 cores, up from the pervious 2 or 3 cores it had. I never touched the ram though. It was still using 6 gigs, I gave it 12 gigs and I believe it is running faster. I haven't used firefox in months. I loaded up the web UI and it was quickly loading posters for movies and I was scrolling on down without a hiccup besides a quick seconds sometimes for posters to catch up with me.

Thanks for the update.

[jan_axhell 3]

+1

[TheRenegade 2]

On 05/08/2021 at 07:45, mgworek said:

e I fixed my slow down. I did start noticing it when I upgraded my server. I gave my VM that is running Emby a

It would be nice if there was even a way to move all the files from one location to another. I have quite a large media collection, and would love to move the artwork off my UnRaid array and onto the SSDs that host the Emby Docker, as there is some lag when Emby fires up due to it needing to spin up all the drives for the artwork. The Docker is fine speed wise, just no artwork. It would be a huge pain in the ass to delete all the artwork and then rescrape.... As sometimes the default artwork is not ideal...

PLEASE make a move button? 

[bonaminGR 19]

This is a really good idea. I'd like to have all my artwork cached on an SSD as well.  

Seems like the way to go. 

 

Is there any progress with this ? Will this be implemented ? 

[Luke 37020]

8 hours ago, bonaminGR said:

This is a really good idea. I'd like to have all my artwork cached on an SSD as well.  

Seems like the way to go. 

 

Is there any progress with this ? Will this be implemented ? 

Hi, I would expect we'll support this in future updates. Thanks.

[C.S. 60]

This would be the DIY solution: (for WIndows at least)

 

[sydlexius 240]

If you're running Unraid or some other linux-based OS, have a look at this script that can be run in as a cron job.  For the Unraid crowd, you'll want to use this in conjunction with the User Scripts plugin.  It will require some modification (you need to add your list of volumes in your Emby docker (I'm not sure if this works with native/bare-metal).  You can even add custom video extensions such as BIF to the list.

[Pixelpaule 4]

+1

[ginjaninja 533]

This could be useful for people who have an sdd for app and magnetic for media and put their hard disks to sleep. Browsing is somewhat and searching is heavily impacted by putting media drives to sleep.

A word of caution - its great that my emby folder is 11GB in size, i used to use a media server which promiscuously cached everything down to different sizes of images (presumably for performance) which used over 100GB for its metadata. Its getting less and less of an issue as SSD sizes increase but 100GB would still be problematic for some.

Anyways a scale of what is cached for different use cases and  drive sizes might be beneficial.

[rbjtech 4223]

On 07/09/2022 at 19:45, C.S. said:

This would be the DIY solution: (for WIndows at least)

 

Finally got around to doing this - and it's working perfectly.

In the drive pool 'File Placement' rules - just specify to place all non video data onto a fast nvme drive - ie  *.jpg / *.png / *.nfo / *.bif / *.srt etc

Drivepool does all the 'linking' - so you'll still 'see' all the above files on your media drives, but they are actually on the nvme drive - thus much faster to load.

Don't forget to backup the nvme drive though (if you don't back the 'pool' as an entity), as the 'real' files are now on there.  

Edited January 24, 2023 by rbjtech

[C.S. 60]

One thing I've noticed with Drivepool is the non-video files can still sometimes be written to the wrong disk, but then they get moved to the correct one whenever balancing occurs. So I set mine to balance immediately. But that might not be a good idea if you are using the SSD optimizer.

[rbjtech 4223]

1 minute ago, C.S. said:

One thing I've noticed with Drivepool is the non-video files can still sometimes be written to the wrong disk, but then they get moved to the correct one whenever balancing occurs. So I set mine to balance immediately. But that might not be a good idea if you are using the SSD optimizer.

Thanks, useful to know - as you say, the balancing schedule will move them but I'm actually using the nvme as a main drive write cache anyway - even the video files get dumped on there whenever I write to the pool - so technically, the video files get moved OFF the nvme, the other files should already be on there..

I'm using the combined SSD Optimzer with the other balancing plugins.

It's certainly more rapid then it was for non-cached emby searches etc - it's not 'night and day' but I can say for sure the images load faster.

[GreatFlashMan 75]

On 03/04/2018 at 06:30, AdrianW said:

It would be useful if the server (or perhaps clients) could cache most (if not all) content images (posters, backdrops, logos, etc) for movies, TV series and seasons. 

 

This would mainly be of benefit when a portion of someone's content is offline but the users would still like to view the content to know what exists.

 

I think caching on the server would be preferable, then all clients could make use of the same cached images.

 

if you think this would be a useful feature, please click the "Like This" button on this post.

+1

[nmkaufman 49]

+1 for caching 

I use the drivepool file placement rules recommended here, and while it does speed the server up, my media HDDs still spin up while browsing the library.

Edited March 28, 2023 by nmkaufman

[andrewds 93]

On 3/28/2023 at 3:38 PM, nmkaufman said:

+1 for caching 

I use the drivepool file placement rules recommended here, and while it does speed the server up, my media HDDs still spin up while browsing the library.

@nmkaufmandid you ever sort this out? I'm working on the same issue now per:

Currently investigating the possibility of adding another abstraction layer or filter somewhere that can present the file information to the OS/application but not actually spin up the disks until the actual data is requested.

 

 

[MediaEmby1968 19]

+1

I also find it interesting, especially for people who have old televisions (case of Samsung TV from 2016). That in emby they cannot see the background images of movies and series with that opacity that is applied to them and it is illegible when the images are light and not dark.

[nmkaufman 49]

On 1/3/2022 at 3:29 PM, TheRenegade said:

It would be nice if there was even a way to move all the files from one location to another.

Revisiting this thread after migrating to Unraid recently, to see if there had been any update.

Unraid is pretty vigilant about allowing disks to sleep (could never manage this with Windows,) but browsing/searching my Emby library causes most/all of my disks to spin up.

This also causes browsing my library to be extremely slow, while disks are still waking up.

"There appears to be an issue communicating with your server" error message level slow.

I'm close to abandoning the 'store metadata and images next to media files' option, altogether, but I know reversing that decision is next to impossible.

Would love to see either image/metadata caching or the ability to move these files to a custom location become a reality in the near future.

At the very least, it would be nice if Emby stored metadata in a user-readable library/title/files format, rather than the cryptic file/folder structure it uses, now.

This would make it less daunting to manually migrate / edit if necessary.

Edited December 13, 2023 by nmkaufman

[adminExitium 173]

35 minutes ago, nmkaufman said:

browsing my library to be extremely slow, while disks are still waking up

You may want to try using mergerfs with the metadata stored on a smaller SSD and the media on the larger HDD disks.

[nmkaufman 49]

1 hour ago, adminExitium said:

You may want to try using mergerfs with the metadata stored on a smaller SSD and the media on the larger HDD disks.

Unraid uses something similar, natively, but doesn't have a tool to handily collect all my existing files onto the SSD cache pool.

I can configure Unraid's 'mover' tool not to ever move metadata off the cache disks, but I would have to manually move all my existing metadata back onto it.

As a linux novice, I see a lot of potential for doing harm if I attempted this.

I know it's something like

cd /media folder/ &&  find ./ -name '*.nfo, sub, etc.' -exec cp -lprv --parents {} /cache drive/ ';' -delete

but I have not summoned the nerve to attempt it.

Edited December 13, 2023 by nmkaufman

[adminExitium 173]

I generally use rclone for such things on linux by doing a sync first and checking everything is fine before doing a move to remove the content from the source.

[nmkaufman 49]

3 hours ago, adminExitium said:

I generally use rclone 

It didn't end up being that scary. I just did a few dry runs without the -delete flag, to make sure I was satisfied with that it was moving.

Too early to know if it keeps my disks from spinning up, but Emby does feel snappier (especially from a cold start.)

For anyone else's future reference, I moved the metadata from each of my libraries (movies in this example) to my Downloads folder with: 

cd /mnt/user/Movies && find ./ -type f \( -iname \*.ass -o -iname \*.gif -o -iname \*.idx -o -iname \*.jfif -o -iname \*.jpeg -o -iname \*.jpg -o -iname \*.nfo -o -iname \*.png -o -iname \*.srt -o -iname \*.ssa -o -iname \*.sub -o -iname \*.svg -o -iname \*.txt -o -iname \*.vtt -o -iname \*.webp \)  -exec cp -a --parents {} /mnt/user/Downloads/Movies/ ';' -delete

And then when I was satisfied that all the metadata had successfully been moved back to the SSD, I just hard-linked it back to the media folders with

cd /mnt/user/Downloads/Movies && find ./ -type f  -exec cp -lprv --parents {} /mnt/user/Movies/ ';' -delete

And then, finally, used the 'Mover Tuner' plugin to tell mover to exclude all those same file-extensions, and leave them on the cache disk in the future.

[Q-Droid 634]

53 minutes ago, nmkaufman said:

It didn't end up being that scary. I just did a few dry runs without the -delete flag, to make sure I was satisfied with that it was moving.

Too early to know if it keeps my disks from spinning up, but Emby does feel snappier (especially from a cold start.)

For anyone else's future reference, I moved the metadata from each of my libraries (movies in this example) to my Downloads folder with: 

cd /mnt/user/Movies && find ./ -type f \( -iname \*.ass -o -iname \*.gif -o -iname \*.idx -o -iname \*.jfif -o -iname \*.jpeg -o -iname \*.jpg -o -iname \*.nfo -o -iname \*.png -o -iname \*.srt -o -iname \*.ssa -o -iname \*.sub -o -iname \*.svg -o -iname \*.txt -o -iname \*.vtt -o -iname \*.webp \)  -exec cp -a --parents {} /mnt/user/Downloads/Movies/ ';' -delete

And then when I was satisfied that all the metadata had successfully been moved back to the SSD, I just hard-linked it back to the media folders with

cd /mnt/user/Downloads/Movies && find ./ -type f  -exec cp -lprv --parents {} /mnt/user/Movies/ ';' -delete

And then, finally, used the 'Mover Tuner' plugin to tell mover to exclude all those same file-extensions, and leave them on the cache disk in the future.

I'm trying to understand what your goal is and whether you actually achieved it. Other Emby users might be tempted to do the same without understanding the actions and results.

1. You moved the metadata from the media library volumes/paths (/mnt/user/Movies - HDD) to a different location (/mnt/user/Downloads/Movies - SSD). This created a directory structure on the destination matching the media tree but only for the metadata files, right?

2. Your second command is the one that confuses me. You copied the "new" metadata files back to their original path, then deleted them from the source (effectively moved)? Yes, you included the -l  argument but hard links are not supported across filesystems. Did this move the metadata files back to their original paths?

I'm not familiar with how Unraid uses the cache drive. Does copying/moving files around automatically put them on the cache drive?

[nmkaufman 49]

49 minutes ago, Q-Droid said:

I'm trying to understand what your goal is and whether you actually achieved it. Other Emby users might be tempted to do the same without understanding the actions and results.

1. You moved the metadata from the media library volumes/paths (/mnt/user/Movies - HDD) to a different location (/mnt/user/Downloads/Movies - SSD). This created a directory structure on the destination matching the media tree but only for the metadata files, right?

2. Your second command is the one that confuses me. You copied the "new" metadata files back to their original path, then deleted them from the source (effectively moved)? Yes, you included the -l  argument but hard links are not supported across filesystems. Did this move the metadata files back to their original paths?

I'm not familiar with how Unraid uses the cache drive. Does copying/moving files around automatically put them on the cache drive?

Unraid is set up to (optionally) utilize a SSD cache/landing-zone. New files hit the SSD landing-zone, and are periodically moved to mechanical storage.

1.) In step one, I copied the metadata files (in their original folder/sub-folder hierarchy) to a temporary location (my downloads folder), allowing me to review their contents before migrating them back to my media folders. I copied (rather than moved) the files, so they would be re-created on the SSD.

2.) I migrated the metadata back to their original media folders (but now they're on the SSD). I copied, again (rather than moved) simply so I could re-use my same script. Linux cp and mv commands use slightly different syntax, so I didn't want to add any additional complexity. I used the -l (lower case "L") flag, so the files would simply be hard-linked, rather than actually copied again, so it was just as fast as a move.

Unraid does allow you to address the physical disks, directly, so you could potentially considate the metadata directly by moving the files from /mnt/disk1/, /mnt/disk2/ etc. to /mnt/cache/.. but this would have to be done for each mechnical disk. My copying method just creates new copies of all the files on the SSD, and deletes the originals.

Believe me, if this all sounds daunting it's because it is.

I'm pretty new to linux, and running scripts and batch files outside a VM or Docker container still gives me anxiety.

I'm happy to report that this was successful, however. I'm now able to search and browse my library, and not even one of my mechanical disks spins up until I play a file.

Gray circle means sleeping. They turn green when they wake up:

Edited December 14, 2023 by nmkaufman