Jump to content

Improve Performance


Bert

Recommended Posts

Cadkey

I have this very simple script that saves the cache and metdata folders to the original Emby folder.
This allows to acquire a copy if the NVME disk fails, as well as to very quickly revert to 'standard' version of the Emby folders on volume1.
This SH file can be launched automatically in the task scheduler, every night.
To be adapted according to your configuration. This corresponds to the folders and configuration of the tutorial.

#!/usr/bin/env bash
cp -ru /volume3/EmbyFast/metadata /var/packages/EmbyServer/var
cp -ru /volume3/EmbyFast/cache /var/packages/EmbyServer/var

EmbyBackup.sh

 

As an example, my classic configuration:
#!/usr/bin/env bash
cp -ru /volume2/EmbyFast/metadata /volume1/@appdata/EmbyServer
cp -ru /volume2/EmbyFast/cache /volume1/@appdata/EmbyServer

Edited by Cadkey
  • Like 4
Link to comment
Share on other sites

10 hours ago, kaj said:

@cayars Thank you for all your hard work and experimentation.

Working perfectly, having followed the guide, on my 918+ with two 512gb NVMe sticks, using the raid 0 option. 

My only thoughts now are the best way to configure. I have, like others, about 1000 movies and 16000 varies epsiodes of Tv Shows. My total EmbyFast folder is less than 10gb, so maybe it's better to use the NMVe's in raid 1?

It's nice for the cache and meta-data folders which make screen scrolling pretty fast, but I think this helps the most performance wise for those recording and watching Live TV like sports.  It's an all around good performance booster.

If you do a lot of DVR recording you may want to leave both setup as that gives you a decent amount of storage before.  I didn't want to get into how you can process DVR stuff in the PDF.

Alternately, if you know you won't need it for this purpose you could copy everything off, unmount the drive.  Then setup one in Storage Manager as a read cache. Then follow the directions in the pdf again to setup just a single NVMe drive.
 

  • Like 2
Link to comment
Share on other sites

25 minutes ago, cayars said:

It's nice for the cache and meta-data folders which make screen scrolling pretty fast, but I think this helps the most performance wise for those recording and watching Live TV like sports.  It's an all around good performance booster.

If you do a lot of DVR recording you may want to leave both setup as that gives you a decent amount of storage before.  I didn't want to get into how you can process DVR stuff in the PDF.

Alternately, if you know you won't need it for this purpose you could copy everything off, unmount the drive.  Then setup one in Storage Manager as a read cache. Then follow the directions in the pdf again to setup just a single NVMe drive.
 

Thanks.

When you say recording and watching Live Tv, I don't record a lot, but do watch Live Tv Sports live.... So i would maybe lean towards one as per the single instructions here and one as a read cache, as you suggest. Does the fact that it's a single read cache, rather than the raid 1 R/W cache (as i had before) make a difference to performance?

Link to comment
Share on other sites

usnscpo

Quick question, or perhaps I missed it... should I upgrade to 7.1 before beginning this?  Just had a look at my /dev/md0 and its at 55%.

Edit:  Nevermind.  Just remembered 7.1 is a beta, and I'd rather wait for the public release of 7.1.  If I understand, correctly as long as I my /dev/md0 folder is say below 75%, when the public release rolls out, it won't break this configuration?

Edited by usnscpo
  • Like 1
Link to comment
Share on other sites

Good instructions.  As noted before I am very beginner and I was able to get this up and running.  Thank you!

  • Like 1
Link to comment
Share on other sites

usnscpo
22 hours ago, Cadkey said:

I have this very simple script that saves the cache and metdata folders to the original Emby folder.
This allows to acquire a copy if the NVME disk fails, as well as to very quickly revert to 'standard' version of the Emby folders on volume1.
This SH file can be launched automatically in the task scheduler, every night.
To be adapted according to your configuration. This corresponds to the folders and configuration of the tutorial.

#!/usr/bin/env bash
cp -ru /volume3/EmbyFast/metadata /var/packages/EmbyServer/var
cp -ru /volume3/EmbyFast/cache /var/packages/EmbyServer/var

EmbyBackup.sh 144 B · 3 downloads

 

As an example, my classic configuration:
#!/usr/bin/env bash
cp -ru /volume2/EmbyFast/metadata /volume1/@appdata/EmbyServer
cp -ru /volume2/EmbyFast/cache /volume1/@appdata/EmbyServer

Success!  Thank you so much Carlo.  I too am a noob and was able to get this running quite easily.

 

Now working on setting the copy script.  I'm a little confused?  Which example is correct for me?  Sending the copies to /var/packages/EmbyServer/var or to /volume2/@appdata/EmbyServer ? (My Emby is installed on volume2 and my NVMe is volume3)  I ask because I have both folders and they each contain a metadata and cache folder. 

Edited by usnscpo
Link to comment
Share on other sites

usnscpo

Copy that kaj, and thanks for your confirmation.  That's what I thought and is how its set.👍

Link to comment
Share on other sites

On 3/5/2022 at 6:13 PM, kaj said:

Thanks.

When you say recording and watching Live Tv, I don't record a lot, but do watch Live Tv Sports live.... So i would maybe lean towards one as per the single instructions here and one as a read cache, as you suggest. Does the fact that it's a single read cache, rather than the raid 1 R/W cache (as i had before) make a difference to performance?

To be honest I'm not sure how much read only cache is going to help outside of Emby.  Emby won't need it as it's working from NVMe for all practical purposes.  It can read movies/shows from disk much faster than it can't send them through the LAN so it won't help much that way.  It can read/write to disk faster than you can send it with a NIC or two, so unless you have another app similar to Emby with mucho data or other databases I'd find another use for it.  You can always try it as a read cache as it's easy to remove.

If you have happen to be using any "download" program that can be installed from the package store, it could be used for that. :)

4 hours ago, Bert said:

Good instructions.  As noted before I am very beginner and I was able to get this up and running.  Thank you!

Glad it's working!

7 hours ago, usnscpo said:

Quick question, or perhaps I missed it... should I upgrade to 7.1 before beginning this?  Just had a look at my /dev/md0 and its at 55%.

Edit:  Nevermind.  Just remembered 7.1 is a beta, and I'd rather wait for the public release of 7.1.  If I understand, correctly as long as I my /dev/md0 folder is say below 75%, when the public release rolls out, it won't break this configuration?

Nope, but on my first test I did the 7.1 upgrade afterward and all hell broke loose for me. It wasn't related to the RAM disk in any way but a SYNO file trying to update itself incorrectly on the 7.1 beta so it kept filling up md0 until my system would crash.

  • Like 1
Link to comment
Share on other sites

4 hours ago, usnscpo said:

Now working on setting the copy script.  I'm a little confused?  Which example is correct for me?  Sending the copies to /var/packages/EmbyServer/var or to /volume2/@appdata/EmbyServer ? (My Emby is installed on volume2 and my NVMe is volume3)  I ask because I have both folders and they each contain a metadata and cache folder. 

Volume 2 in the disk based version you started with and now will be the backup.
Volume 3 is the NVMe disk so you would just copy from 3 to 2.

Take a look at this post to see how to use the Task Scheduler to run jobs.  You are going to want to do almost exactly the same thing except the data put in "user-defined script" area will be different.  The script lines could be different for each person depending how they set it up.  You could try these:

cp -rv /volume3/EmbyFast/cache/* /volume1/@appdata/EmbyServer/cache
cp -rv /volume3/EmbyFast/metadata/* /volume1/@appdata/EmbyServer/metadata

I'd SSH into the NAS and do a "sudo -i" and test each line to make sure it works correctly.  If so those are the two lines for your backup task.

  • Thanks 1
Link to comment
Share on other sites

1 hour ago, cayars said:

To be honest I'm not sure how much read only cache is going to help outside of Emby.  Emby won't need it as it's working from NVMe for all practical purposes.  It can read movies/shows from disk much faster than it can't send them through the LAN so it won't help much that way.  It can read/write to disk faster than you can send it with a NIC or two, so unless you have another app similar to Emby with mucho data or other databases I'd find another use for it.  You can always try it as a read cache as it's easy to remove.

If you have happen to be using any "download" program that can be installed from the package store, it could be used for that. :)

Thanks. I think I'll leave as is for the time being, with the raid0 setup on the NVMe's and see how it goes.....

Don't think I've ever had downloads speeds fast enough to warrent extra spped from installing them there (all is in docker anyways....) 😜

Link to comment
Share on other sites

Cadkey

 

19 hours ago, kaj said:

@Cadkey If you create a Scheduled Task, you also have to run it as user root, or you may get permission errors

LOL You are explaining to me how I should use my script that I shared here 🤣

Link to comment
Share on other sites

Probably more to help others so they get it setup right.

Link to comment
Share on other sites

Dusan78

I have to make an argument against using the copy command.  Specially if your metadata and cache folders are large.  You could be needlessly copying hundreds of GBs daily when in reality you only need to update with new and changed files and remove deleted files.  That could be only few new added TV episodes daily.  It just doesn't make sense to copy the whole thing.  Perhaps you could be using an archive bit?

Just my 2 cents.

Link to comment
Share on other sites

2 hours ago, Cadkey said:

 

LOL You are explaining to me how I should use my script that I shared here 🤣

As @cayarssaid, just for those without your experience and wisdom, If you run the scritp from the default user (usually your user), as you know, it will fail with permission errors (I know because that's what happened to me the first time i ran it in Task Scheduler :) ). 

  • Haha 1
Link to comment
Share on other sites

1 hour ago, kaj said:

As @cayarssaid, just for those without your experience and wisdom, If you run the scritp from the default user (usually your user), as you know, it will fail with permission errors (I know because that's what happened to me the first time i ran it in Task Scheduler :) ). 

That gets nearly everyone the first time!

  • Haha 1
  • Agree 1
Link to comment
Share on other sites

Cadkey
7 hours ago, Dusan78 said:

I have to make an argument against using the copy command.  Specially if your metadata and cache folders are large.  You could be needlessly copying hundreds of GBs daily when in reality you only need to update with new and changed files and remove deleted files.  That could be only few new added TV episodes daily.  It just doesn't make sense to copy the whole thing.  Perhaps you could be using an archive bit?

Just my 2 cents.

you are certainly right, I only thought of my use.  I have less than 4GB in metadata and cache folders, no .bif files

Link to comment
Share on other sites

usnscpo

Thought I'd just note this for interested minds.  This morning my 920+ advised that DSM 7.1 was available for upgrade, so I let it update.  No issues with this NVMe configuration noted.  All seems well.

  • Like 1
Link to comment
Share on other sites

FrostByte
3 hours ago, usnscpo said:

Thought I'd just note this for interested minds.  This morning my 920+ advised that DSM 7.1 was available for upgrade, so I let it update.  No issues with this NVMe configuration noted.  All seems well.

Is 7.1 out of beta already?  I got notified of 7.0.1 update 3 yesterday

Link to comment
Share on other sites

So how's the speed difference having the cache and metadata directly on the NVMe vs just in the cache?

Link to comment
Share on other sites

usnscpo

I have roughly 500 mkv movies and 300 photos.  I can scroll them all at full speed and they populate the screen with zero lag.  Selecting a movie is instant.  Maybe 2 seconds tops for a movie to begin playing once I hit play.  So yea, for me, it has significantly increased browsing and accessing media.  I had a few 4k movies that would stutter a little and take a few seconds to buffer, and I haven't experienced any of that since the upgrade.  I don't do TV through Emby, so no comment there.  I'm about to open my Emby server up to others which will require transcoding of my 4k stuff, so we'll see how that goes.  This is why I wanted to do this upgrade, so when outside users watch a movie, their needs, won't impact me while I'm watching something.  Highly recommended "easy" upgrade that is worth the price of a 500Gb NVMe.  Again, most grateful to you Carlo, for putting so much R&D time and sharing your efforts!

Edit:  Might consider this thread, or at least a summation as a "sticky"

Edited by usnscpo
  • Like 2
Link to comment
Share on other sites

usnscpo
15 hours ago, FrostByte said:

A tutorial is probably more appropriate.  The thread is 7 pages.

I volunteer to create one from the material in this thread.  What format (doc, txt, docx) is preferred?  Where shall I send it/post it?

Edited by usnscpo
Link to comment
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now
×
×
  • Create New...