Cadkey 47 Posted March 2, 2022 Share Posted March 2, 2022 (edited) 11 hours ago, cayars said: Have a look in /volume1/@appdata/EmbyServer/metadata and /volume1/@appdata/EmbyServer/cache after navigating the metadata run this command to get the total size of content du -s -h navigate to the cache folder and run the same du -s -h command I just had a look at a backup of my system and the bif sizes are far smaller than I was thinking they were. I had forgotten that softworkz did a fantastic optimization job on these and reduced the sizes a lot. Bifs on my system are about 400 GB so I'd feel pretty comfortable thinking every key folder could easily fit on a single TB NVMe. If you do want to find the specific size of bif files try this command from the metadata folder find -type f -iname *.bif -print0 | du -ch --files0-from=- | grep total$ Add the size of metadata and cache folder will form the base of storage needed. This is what I get, and it's not at all what I expected /volume1/@appdata/EmbyServer/metadata du -s -h 2.2G find -type f -iname *.bif -print0 | du -ch --files0-from=- | grep total$ 0 total No .bif file. How is it possible? Command echo "Total files \".bif\": $(( $(find . -type f -iname "*.bif" -printf '+%s') ))" also returns 0 /volume1/@appdata/EmbyServer/cache du -s -h 988M Add the size of metadata and cache folder will form the base of storage needed. Only 3G ? Edit: I assume that the .bif are thumbnail images of each video at regular intervals, like the chapters that Emby creates every 5min by default, if I'm not mistaken. I disabled the display of chapter images because I don't need them. I don't know what use it would be for me to create .bif files, which appear on the timeline when moving through a video, because I don't use that today. That's probably why I don't have any .bif. What I want to improve is the display speed of images, posters, banners, wallpapers. It's .jpg I think, and all the metadata that shows on the Tv screen. I think an NVME would be useful for that, even if the amount of data is lower than with .bif. This leaves room for temporary transcoding files and extends the life of the NVME. Edited March 2, 2022 by Cadkey Link to comment Share on other sites More sharing options...
Dusan78 18 Posted March 2, 2022 Share Posted March 2, 2022 3 hours ago, Cadkey said: This is what I get, and it's not at all what I expected /volume1/@appdata/EmbyServer/metadata du -s -h 2.2G find -type f -iname *.bif -print0 | du -ch --files0-from=- | grep total$ 0 total No .bif file. How is it possible? Command echo "Total files \".bif\": $(( $(find . -type f -iname "*.bif" -printf '+%s') ))" also returns 0 /volume1/@appdata/EmbyServer/cache du -s -h 988M Add the size of metadata and cache folder will form the base of storage needed. Only 3G ? Edit: I assume that the .bif are thumbnail images of each video at regular intervals, like the chapters that Emby creates every 5min by default, if I'm not mistaken. I disabled the display of chapter images because I don't need them. I don't know what use it would be for me to create .bif files, which appear on the timeline when moving through a video, because I don't use that today. That's probably why I don't have any .bif. What I want to improve is the display speed of images, posters, banners, wallpapers. It's .jpg I think, and all the metadata that shows on the Tv screen. I think an NVME would be useful for that, even if the amount of data is lower than with .bif. This leaves room for temporary transcoding files and extends the life of the NVME. I have a feeling that you are storing these files with media instead of metadata. Same with .bifs. Check under server settings -> library -> settings make sure that "save artwork to media folder" is not checked. same for "save video previews to media folder" If your posters, artwork,.bifs are inside media folders they will either have to be recreated or moved. This is a nightmare scenario. Posters was not a big a deal to recreate but 15000 .bifs would have taken a month to extract all over again (for me with 10sec option). So for that I had to write some code to look into database and move all the .bifs into correct folders. Overall you have a little more items than me. I have bif for every 10sec so my metadata will be much bigger. Everything else is small. Your 500Gb nvme is more than enough for sure. But, you'll have to wait for the script or a program to move files over, or you'll have to recreate them if they are stored with media. Cheers! Link to comment Share on other sites More sharing options...
Cadkey 47 Posted March 2, 2022 Share Posted March 2, 2022 3 minutes ago, Dusan78 said: I have a feeling that you are storing these files with media instead of metadata. Same with .bifs. Check under server settings -> library -> settings make sure that "save artwork to media folder" is not checked. same for "save video previews to media folder" If your posters, artwork,.bifs are inside media folders they will either have to be recreated or moved. This is a nightmare scenario. Posters was not a big a deal to recreate but 15000 .bifs would have taken a month to extract all over again (for me with 10sec option). So for that I had to write some code to look into database and move all the .bifs into correct folders. Overall you have a little more items than me. I have bif for every 10sec so my metadata will be much bigger. Everything else is small. Your 500Gb nvme is more than enough for sure. But, you'll have to wait for the script or a program to move files over, or you'll have to recreate them if they are stored with media. Cheers! Hello, I only have .mkv in my media folders, no .nfo and no .bif I think I have no .bif, because I have no chapter images, and I have disabled the creation of these thumbnails in my configuration. The only images are the posters, logos, banners and wallpapers of each media, films, series, episodes. They are .jpg. I don't have any .bif Link to comment Share on other sites More sharing options...
Dusan78 18 Posted March 2, 2022 Share Posted March 2, 2022 I see! (...said the blind man to the deaf dog) Well than you can probably stick a piece aluminum foil inside the m.2 slot and it should be enough space. hehehe 1 Link to comment Share on other sites More sharing options...
Bert 49 Posted March 2, 2022 Author Share Posted March 2, 2022 Do you want me to change the title of this discussion. It's in a way been Hijacked. I don't mind, I'm just saying should I change it? How far do we think we are from having documentation? Days, weeks, months? Link to comment Share on other sites More sharing options...
Cadkey 47 Posted March 2, 2022 Share Posted March 2, 2022 40 minutes ago, Dusan78 said: I see! (...said the blind man to the deaf dog) Well than you can probably stick a piece aluminum foil inside the m.2 slot and it should be enough space. hehehe I did not understand. Does this mean that an NVME is useless? Link to comment Share on other sites More sharing options...
Dusan78 18 Posted March 2, 2022 Share Posted March 2, 2022 37 minutes ago, Cadkey said: I did not understand. Does this mean that an NVME is useless? No no, it's very useful. I was just making a joke comment as to your space requirements. NVME is 110% worth it. 1 Link to comment Share on other sites More sharing options...
Carlo 4330 Posted March 2, 2022 Share Posted March 2, 2022 23 hours ago, Cadkey said: I have a 47To movies and series library. My two NVME 512Go are sufficient according to you? 1 NVMe is probably going to be fine for our purposes of meta-data, cache & transcoding. That's going to leave the other NVMe lonely. I'd suggest leaving the 2nd NVMe out for a few days to a week of use to best decide it's use moving forward. You could: Set it up as a mirror for redundancy - Really not needed as we'll be syncing the cache and meta-data folders back to the Emby Volume. Standard Read Cache - Add it back to the Synology file system as a read cache to your volumes. DVR Recording - Set it up for Emby to use specifically for DVR Recording. This provides Emby Server the best IOPs possible which will breath new life into DVR & Trick Play watching of sports games/matches by removing spinning rust. I probably won't have this included in the initial docs as it's a more advanced topic but you could also set both NVMes up in a striped configuration (RAID 0) that splits the IO between them and doubles the storage for even more IOPs. This will provide one new volume making the space of both available for use. This will likely only make sense if using the NVMe as a tempority DVR recording area otherwise the extra space will likely never be used. In that case the NVMe is probably best set backup as a read cache for Synology volumes. 1 hour ago, Bert said: Do you want me to change the title of this discussion. It's in a way been Hijacked. I don't mind, I'm just saying should I change it? How far do we think we are from having documentation? Days, weeks, months? 1 Link to comment Share on other sites More sharing options...
Carlo 4330 Posted March 2, 2022 Share Posted March 2, 2022 9 hours ago, Cadkey said: This is what I get, and it's not at all what I expected /volume1/@appdata/EmbyServer/metadata du -s -h 2.2G find -type f -iname *.bif -print0 | du -ch --files0-from=- | grep total$ 0 total No .bif file. How is it possible? Command echo "Total files \".bif\": $(( $(find . -type f -iname "*.bif" -printf '+%s') ))" also returns 0 /volume1/@appdata/EmbyServer/cache du -s -h 988M Add the size of metadata and cache folder will form the base of storage needed. Only 3G ? Edit: I assume that the .bif are thumbnail images of each video at regular intervals, like the chapters that Emby creates every 5min by default, if I'm not mistaken. I disabled the display of chapter images because I don't need them. I don't know what use it would be for me to create .bif files, which appear on the timeline when moving through a video, because I don't use that today. That's probably why I don't have any .bif. What I want to improve is the display speed of images, posters, banners, wallpapers. It's .jpg I think, and all the metadata that shows on the Tv screen. I think an NVME would be useful for that, even if the amount of data is lower than with .bif. This leaves room for temporary transcoding files and extends the life of the NVME. It's quite possible I didn't get the commands exactly correct as I couldn't check them. If you're really curious about the size of cache and meta-data folders, you could copy the cache and meta-data folders to a share already setup and have a look that way. Yes the bifs are used for chapter pics as well as time lapse images in many clients when you use trick play (ie RW, FF) or chapter jumps. They take up such little space these days and add so much to the UI I can't image not using them. You should try setting up a small test library with a few movies in it and add the thumbnail processing to re-evaluate it. You may change your mind! All processing can be done in off hours during nightly scheduled events. Link to comment Share on other sites More sharing options...
Carlo 4330 Posted March 2, 2022 Share Posted March 2, 2022 3 hours ago, Bert said: Do you want me to change the title of this discussion. It's in a way been Hijacked. I don't mind, I'm just saying should I change it? How far do we think we are from having documentation? Days, weeks, months? More like hours. With any luck 24-48. It's just finishing up the bulk copy so I want to run another pass for good measure. I'm going to check to see if I have 4 free SATA drives available and if so I'll just switch drives and start fresh, otherwise I'll reset the NAS and go from there. I have 60 large SAS drives available, but can't use them in the 920+. Link to comment Share on other sites More sharing options...
Cadkey 47 Posted March 2, 2022 Share Posted March 2, 2022 (edited) 2 hours ago, cayars said: Yes the bifs are used for chapter pics as well as time lapse images in many clients when you use trick play (ie RW, FF) or chapter jumps. They take up such little space these days and add so much to the UI I can't image not using them. You should try setting up a small test library with a few movies in it and add the thumbnail processing to re-evaluate it. You may change your mind! All processing can be done in off hours during nightly scheduled events. Maybe I'll try later. There I am mounting my 2x NVME 512GB in raid step by step... Edited March 2, 2022 by Cadkey Link to comment Share on other sites More sharing options...
Cadkey 47 Posted March 3, 2022 Share Posted March 3, 2022 (edited) On 1/22/2022 at 4:28 AM, Dusan78 said: My DS920+ is now unbelievably fast. Hello, What is your setup please? 1x NVME in cache as DSM allows and 1x NVME in volume2? Or 2x NVME mounted in RAID0 (or RAID1) volume2? Edited March 3, 2022 by Cadkey Link to comment Share on other sites More sharing options...
Dusan78 18 Posted March 3, 2022 Share Posted March 3, 2022 2 hours ago, Cadkey said: Hello, What is your setup please? 1x NVME in cache as DSM allows and 1x NVME in volume2? Or 2x NVME mounted in RAID0 (or RAID1) volume2? My set up is just 1x 500Gb NVME mounted as a volume2 with moved cache, metadata and transcode folders. I thought about experimenting with raid 1 but, it's a waste of money and nvme. I just run a mirror sync once a day instead so that I have a backup on my NAS disks. That's it. 1 Link to comment Share on other sites More sharing options...
Carlo 4330 Posted March 4, 2022 Share Posted March 4, 2022 Just wanted to let you know I've reviewed a few of the other existing sources of setting up NVMe(s) for storage use and I'm trying hard to make this more detailed and thorough. It includes instructions for a single NVMe use (in either slot) or use of two NVM(e) drives in a RAID 1 configuration. I've been completely through the first iteration of a single NVMe including updating to the latest DSM 7.1 release which is now in it's 3rd iteration/release that I know of. The RAID 1 setup of NVMe(s) does take quite a while so be prepared for that. As you can see it's 3.2% done with 915.4 minutes to go or about 15 hours. I wrote the documentation on the first pass with a single NVMe and double checking it during my second test run. It's 4am my time and nothing more to do for several hours until this process finishes. I'll have about a hour left of testing assuming everything goes smooth and instructions will be posted for both single and dual drive NVMe configurations plus the needed instructions to setup Emby Server to properly use the new volume. If you need to perform a backup (and you should) now's the time! 1 2 Link to comment Share on other sites More sharing options...
Carlo 4330 Posted March 4, 2022 Share Posted March 4, 2022 It's working fine for me. If possible I would like to have one "volunteer" walk through the instructions as a last check. If you like I can TeamView in with you and standby in case you have a problem. I mainly want to make sure the instructions for working with the NVMe, partitions, disks, volumes and pool is crystal clear. I would ask who wants to be the guinea pig but instead I'll ask who want to spend a short amount of time with me. LOL Link to comment Share on other sites More sharing options...
Bert 49 Posted March 4, 2022 Author Share Posted March 4, 2022 I could. How much time we talking? Now? Link to comment Share on other sites More sharing options...
FrostByte 5041 Posted March 4, 2022 Share Posted March 4, 2022 53 minutes ago, cayars said: instead I'll ask who want to spend a short amount of time with me. LOL I think you just scared everyone away with that 1 Link to comment Share on other sites More sharing options...
Cadkey 47 Posted March 4, 2022 Share Posted March 4, 2022 1 hour ago, cayars said: It's working fine for me. If possible I would like to have one "volunteer" walk through the instructions as a last check. I couldn't wait. I did the manipulation yesterday with 2x NVME 512GB in RAID0. Thanks again to @Dusan78 who told me the method to move the Emby folders to the NVME volume2 Link to comment Share on other sites More sharing options...
Carlo 4330 Posted March 4, 2022 Share Posted March 4, 2022 You could wait, you just didn't want to wait. LOL Hey @Cadkey can you do me a favor and show me what you get running this? df -h Link to comment Share on other sites More sharing options...
Cadkey 47 Posted March 5, 2022 Share Posted March 5, 2022 10 hours ago, cayars said: Hey @Cadkey can you do me a favor and show me what you get running this? df -h Link to comment Share on other sites More sharing options...
Carlo 4330 Posted March 5, 2022 Share Posted March 5, 2022 Thanks. I've been able to reproduce the crash I had a few times now. If md0 fills up all kinds of bad things can happen such as not being able to login or use sudo. No space available to install apps and usually no work space to remove apps. When it's already high the 7.1 update was filling it up and crashing my system. As you can see it's only 2.3G in size so it's not going to hold a lot. Link to comment Share on other sites More sharing options...
Carlo 4330 Posted March 5, 2022 Share Posted March 5, 2022 I tried to make this as easy to follow as possible and made it a bit flexible to handle 1 or 2 NVMe(s). You can choose a single disk or use RAID 0 or 1. You can also choose to use the ext4 file system or BTRFS. It's a lot of combinations to test but I did with the 7.1 DSM install afterward. I'd keep an eye on the size of /dev/md0 in the root directory using this command: df - h If your use column shows 75% or higher I would find out why and remove the files before doing the NVMe stuff. NVMe Synology Storage.pdf 2 3 Link to comment Share on other sites More sharing options...
kaj 240 Posted March 5, 2022 Share Posted March 5, 2022 @cayars Thank you for all your hard work and experimentation. Working perfectly, having followed the guide, on my 918+ with two 512gb NVMe sticks, using the raid 0 option. My only thoughts now are the best way to configure. I have, like others, about 1000 movies and 16000 varies epsiodes of Tv Shows. My total EmbyFast folder is less than 10gb, so maybe it's better to use the NMVe's in raid 1? 1 Link to comment Share on other sites More sharing options...
FrostByte 5041 Posted March 5, 2022 Share Posted March 5, 2022 Super cool, thanks Carlo. 1 1 Link to comment Share on other sites More sharing options...
usnscpo 24 Posted March 5, 2022 Share Posted March 5, 2022 Currently traveling abroad. Home tomorrow. Excited to get started on this. Will report back results in a couple days. You rock Carlo! 1 Link to comment Share on other sites More sharing options...
Recommended Posts
Create an account or sign in to comment
You need to be a member in order to leave a comment
Create an account
Sign up for a new account in our community. It's easy!
Register a new accountSign in
Already have an account? Sign in here.
Sign In Now