Metadata folder getting huge

[nodiaque 62]

Hello everyone,

I have a new Emby server installation on unraid under docker. Everything work well. I've done the configuration a bit differently but I just notice that my /metadata/library folder is using 38GB of space. What is in there and how do I shrunk that?

 

Thank you!

[Luke 42079]

Hi, what subfolders under there are using the most space?

[nodiaque 62]

It's 256 folders named xx with subfolders (named like a windows update guid) in them. All root folder range from 5-200mb, and subfolders rang from 1 to 10mb. It's the number of files/folder that make the weight (125mb x 256 folders = 32GB)

edit:

According to windirstat:

36.8GB: BIF file

316.6MB: JPG file

13.1MB: PNG file

Edited January 14, 2022 by nodiaque

[GrimReaper 4740]

BIF files are 10-seconds video preview images for visual seeking you selected in your library (libraries) settings. If space is of concern, you can opt either to store images with media files OR use only Chapter thumbnails (5min intervals). AFAIK that'll apply only to new items.

[nodiaque 62]

Oh, didn't see that. Thanks!

Hmmm... what happen if I do a CTRL+A -> Delete in that folder? Or if I delete all BIG by hand, would I corrupt a DB somewhere?

[GrimReaper 4740]

7 minutes ago, nodiaque said:

what happen if I do a CTRL+A -> Delete in that folder?

You'd have no images (posters, thumbs...) throughout UI in any client app. I'd say that'd be unwise.

8 minutes ago, nodiaque said:

Or if I delete all BIG by hand

As imagine it, if you delete all BIFs, worse that can happen would be for DB to reference non-existing items, and you'll have no visual cues while seeking, running Thumbnail image extraction task should theoretically recreate them with new settings? Don't use BIFs (only Chapter markers) so never tested that particular scenario, @FrostByte or @rbjtech might jump in and give some insight.

[nodiaque 62]

I disabled the thumbnail, so I don't care if there's no more (I rarely skip anyway and I use right arrow to skip, thus image is really not needed). I just don't want to "break" something.

If I do delete all and refresh all metadata with overwrite image? Right now, image are a mix. Some are with the movies files and some are in the cache/metadata folder. On my old installation, I had all pictures/data with the files, on my new installation, I didn't check the box for this. Reason is my emby installation is a on raid 1 SSD pool thus having everything for the ui load from there make it very snappy, and prevent all drives spinning just to show pictures (on the long end). I'm currently on my way to delete all jpg, nfo and other non video file from my library so they are imported with the server installation.

Edited January 14, 2022 by nodiaque

[rbjtech 5284]

You can delete the .BIF files - but it will just re-create them again unless you re-configure it not to.

In the Library settings (for each library) turn on Advanced -

Then you will get the options below.

Set generate as 'Never' and it will not create them.

if your concern is disk space - then as @GrimReaper has said, if you choose 'Chapter' images instead - then the files will be much smaller (a few Kb each) as it will just save an image on each chapter point - or every 5 mins if you have no chapters.

Lastly - if you want to save the BIF's with your media (I'd recommend you do) - then you may want to keep the '10 second' images instead.

I hope that helps.

 

Edited January 14, 2022 by rbjtech

[nodiaque 62]

Ah, to be honest, I don't really care about the chapter/seek thumbnail since I'm not even using them.

But why keeping the BIF file with the media recommended? I guess for space management but aside of that?

Thanks a lot, I've just disabled it on all library, I'll run my bash command to delete all bif file in that folder.

edit: should I close my docker before deleting those file or I do it while it's online?

Edited January 14, 2022 by nodiaque

[rbjtech 5284]

7 minutes ago, nodiaque said:

But why keeping the BIF file with the media recommended? I guess for space management but aside of that?

Just to keep it all together and it makes it all more 'portable'.  If for example you built a 2nd emby instance, or wiped and re-installed your primary emby  - then it would just re-use the .bif files as long as they are named the same as the media file with a .bif extension. 

Anyway - if you don't want them - then just delete them - I'd suggest a library scan afterwards just to sync things up.

Edited January 14, 2022 by rbjtech

[nodiaque 62]

Ah yeah, that's one of the reason at first why I had everything in the movie folders. But then I saw that it "lag" when loading on the GUI and was seeing the drives spinning everytime someone just want to browse.

Thanks for the help, I'll get on deleting those oversized bif file.

[nodiaque 62]

I see I have 10GB in transcoding-temp. Looking at my dashboard, all I have is me playing a file in direct play with mpc-be on my computer (open the file directly through share). I deleted the transcoding files about an hours ago. The content are all .ts files but I don't have any iso / dvd file, everything is avi, mkv and such.

[Luke 42079]

23 hours ago, nodiaque said:

 

But why keeping the BIF file with the media recommended? I guess for space management but aside of that?

It makes it easier when moving to a new installation. Other than that it's personal preference.

[Happy2Play 9780]

On 1/14/2022 at 11:38 AM, nodiaque said:

I see I have 10GB in transcoding-temp. Looking at my dashboard, all I have is me playing a file in direct play with mpc-be on my computer (open the file directly through share). I deleted the transcoding files about an hours ago. The content are all .ts files but I don't have any iso / dvd file, everything is avi, mkv and such.

This is normal assuming items are being stopped properly as Emby will maintain the entire playback session that needs direct stream/remux/transcoded until properly stopped.

A Emby will also empty this folder of any rogue sessions that have been left behind also.

[nodiaque 62]

Reviving this thread for a little question.

Let's say I store thumbnail with the media file. If someone go to emby and select a series. My understanding is that emby will read the video file from the folder correct? It doesn't cache them somewhere?

My question is because I'm using unraid as my virtual host. In unraid, I have a share for my tv show on a hardrive and a pool cache which are SSDs. My HDD are normally spin down until someone start a playback. Since all of emby is on the ssd pool (except for the video themself), it's very fast and never spin-up the drive until there's an actual video playing.

But if I create thumbnail on the TV share, which is on a spindown hard drive and someone start emby client, this will cause a spin-up of the drive to show the information on the client, correct?

I hope I'm making sens. My goal is to lower drive spinning at a minimum, since the root folder of the series can be on a different drive then the movie itself, it's not efficient for me to have all drive always spinning just for emby browsing.

Thanks!

[Luke 42079]

1 hour ago, nodiaque said:

Reviving this thread for a little question.

Let's say I store thumbnail with the media file. If someone go to emby and select a series. My understanding is that emby will read the video file from the folder correct? It doesn't cache them somewhere?

My question is because I'm using unraid as my virtual host. In unraid, I have a share for my tv show on a hardrive and a pool cache which are SSDs. My HDD are normally spin down until someone start a playback. Since all of emby is on the ssd pool (except for the video themself), it's very fast and never spin-up the drive until there's an actual video playing.

But if I create thumbnail on the TV share, which is on a spindown hard drive and someone start emby client, this will cause a spin-up of the drive to show the information on the client, correct?

I hope I'm making sens. My goal is to lower drive spinning at a minimum, since the root folder of the series can be on a different drive then the movie itself, it's not efficient for me to have all drive always spinning just for emby browsing.

Thanks!

It could spin up the drive yes, if it's the first time the client is requesting the image. There are caches in both the server and clients to minimize this.

[nodiaque 62]

ok. Is the cache persistent or is lost upon server reboot?

[Luke 42079]

It's persistent but older files are cleaned out periodically.

[nodiaque 62]

ok great, thanks!

[lharris 3]

Reviving this thread once more. 

• Is the setting "Save video preview thumbnails into media folders" not set by default?  My emby installation is rather old, just curious what how new installs/library's behave.  
   
• If I were to enable this setting, would Emby move the BIF files at some point or would I have to facilitate this on my own?
   
• My current Emby deployment is backed by VMWare on a datastore with about 8x1TB in RAID6 that goes like hell... movie storage is 12x4TB disks that also go like hell, but doesn't have the IOPS.  Everything is interconnected with 10GB ethernet.    Would I be see a noticeable difference with BIF's residing on storage with lower IOPS or is the file loaded once in it's entirety?

[Luke 42079]

Hi, video preview thumbnails are off by default and changing the option will not move or delete existing ones.

[Luke 42079]

Regarding performance, what are the differences between the drives in the two setups?

[nodiaque 62]

I doubt you'll see a difference unless you are already bottlenecking the first array.

[lharris 3]

4 hours ago, Luke said:

Regarding performance, what are the differences between the drives in the two setups?

Whoops, I left out the critical detail:   The 8x1TB drives in RAID6 are all SSD, backed by an Perc h710 with a battery in a retired Dell r720xd, so the VM's are snappy.   

4k Q32 reads 225mb/sec 54k IOPS, writes 83.38mb/sec 20k IOPS

4k Q1 reads are 24mb/sec 5893 IOPS, writes are 2mb/sec and 3500 IOPS 

 

The 12x4TB are spinning disks and accessed via 10GB ethernet, backed by a Perc h700 with a battery in a retired Dell R510

4k Q32 reads 54mb/sec 13k IOPS, writes 4.49MB/sec 1095 IOPS

4k Q1 reads are 7mb/sec 1749 IOPS, writes are 3mb/sec and 703 IOPS 

I assume if the bif file is read in it's entirety when the movie is loaded, it's not a big deal vs repeatedly referenced each time a user seeks about.

 

I only care about the wife being happy... sooo if I have to allocate more storage from the local SSD datastore and grow the C:\ drive on the VM that runs emby... I'll do it to keep things snappy.   My metadata folder is about 28.9GB, so if having the files on spinning disk vs SSD has zero impact.... I'll just look at moving them or nuking them, settings that setting, and letting them regenerate.

Edited September 5, 2022 by lharris

[nodiaque 62]

Honestly, it's always depend on what's going on. My emby is on an unraid server. Unraid doesn't use raid, it's only parity protection without the raid. And I also have spinedown drive when they aren't in use.

 

The bif file are only for chapter preview and seeking, not for watching. 

In my setup, when I press play, I can have a delay of the drive isn't spinning. But after that,.no problem, can watch 4k without any problem. I've had many user at once watching stuff on the server local and remotely without any problem. Emby is on an ssd, but all the data are on spin drive.