Jump to content

How to speed up Emby on Linux


Dan64

Recommended Posts

Assuming that there is memory to spare, placing Emby's data and/or cache to RAM offers significant advantages. Even though opting for the partial route is an improvement by itself, the latter can make Emby even more responsive compared to its stock configuration. Benefits include, among others:

- reduced drive read/writes;
- heightened responsive feel;
- many operations within Emby, such as quick search are nearly instantaneous.

 

To do so we can make use of a tmpfs.

 

Because data placed therein cannot survive a shutdown, a script responsible for syncing back to drive prior to system shutdown is necessary if persistence is desired. On the other hand, only relocating the cache is a quick, less inclusive solution that will slightly speed up user experience with Emby.

 

In this post is provided an example of script to move and sync the "data" folder (where is stored the emby db) to RAM.  Be sure that rsync is installed and save the "executable" script to /opt/emby-server/bin/emby-data_sync.sh, for example:

#!/bin/sh
# The script will first move emby's data to a new static location,
# make a sub-directory in /dev/shm, softlink to it and later populate it with the contents of data.
# The second time the script runs, it will then preserve the RAM profile by copying (sync) it back to disk.

PATH=/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin

static=data-static
link=data
volatile=/dev/shm/emby-data

IFS=
set -efu

cd /var/lib/emby

if [ ! -r $volatile ]; then
        mkdir -m0700 $volatile
fi

if [ "$(readlink $link)" != "$volatile" ]; then
        echo "--> Emby: Init RAM Disk"
        mv $link $static
        ln -s $volatile $link
fi

if [ -e $link/.unpacked ]; then
        echo "--> Emby: Sync Disk"
        rsync -av --delete --exclude .unpacked ./$link/ ./$static/
else
        echo "--> Emby: Sync RAM"
        rsync -av ./$static/ ./$link/
        touch $link/.unpacked
fi

It is best to add a cron job to guarantee the sync between RAM and disk, for example the following entry will run the sync process every 30min.

*/30 * * * * /opt/emby-server/bin/emby-data_sync.sh

Then it is necessary to change the service: /lib/systemd/system/emby-server.service to guarantee the initial and final sync, as follow:

[Unit]
Description=Emby Server is a personal media server with apps on just about every device.
After=network.target

[Service]
TimeoutSec=5min
EnvironmentFile=/etc/emby-server.conf
WorkingDirectory=/opt/emby-server
ExecStartPre=/opt/emby-server/bin/emby-data_sync.sh
ExecStart=/opt/emby-server/bin/emby-server
ExecStopPost=/opt/emby-server/bin/emby-data_sync.sh
RestartForceExitStatus=3
User=emby

[Install]
WantedBy=multi-user.target

After this change it is necessary reload the service with the command:

systemctl daemon-reload

Finally it is possible to activate the shared-memory for Emby with the command

systemctl restart emby-server

It is possible to verify the execution of the script with the command:

systemctl status emby-server

● emby-server.service - Emby Server is a personal media server with apps on just about every device.
   Loaded: loaded (/lib/systemd/system/emby-server.service; enabled; vendor preset: enabled)
   Active: active (running) since Tue 2019-01-01 09:46:44 CET; 7min ago
  Process: 8310 ExecStopPost=/opt/emby-server/bin/emby-data_sync.sh (code=exited, status=0/SUCCESS)
  Process: 8315 ExecStartPre=/opt/emby-server/bin/emby-data_sync.sh (code=exited, status=0/SUCCESS)
 Main PID: 8320 (EmbyServer)
    Tasks: 23 (limit: 4915)
   Memory: 676.5M
   CGroup: /system.slice/emby-server.service
           └─8320 /opt/emby-server/system/EmbyServer -programdata /var/lib/emby -ffmpeg /opt/emby-server/bin/ffmpeg

More details can be found with the command:

journalctl | grep emby-data_sync.sh

emby-data_sync.sh[23844]: sending incremental file list
emby-data_sync.sh[23844]: activitylog.db-wal
emby-data_sync.sh[23844]: authentication.db-wal
emby-data_sync.sh[23844]: library.db
emby-data_sync.sh[23844]: library.db-wal
emby-data_sync.sh[23844]: users.db-wal
emby-data_sync.sh[23844]: ScheduledTasks/6330ee8f-b4a9-57f3-3981-f89aa78b030f.js
emby-data_sync.sh[23844]: ScheduledTasks/81267bb8-bb49-51de-4b1b-446262cc7365.js
emby-data_sync.sh[23844]: ScheduledTasks/d459f39d-7abb-81f5-fcce-03048a7ecf4d.js
emby-data_sync.sh[23844]: SyncData/Emby.Kodi.SyncQueue.I.1.40.json
emby-data_sync.sh[23844]: sent 255,467,606 bytes  received 227 bytes  30,055,039.18 bytes/sec
emby-data_sync.sh[23844]: total size is 257,320,410  speedup is 1.01
emby-data_sync.sh[23852]: sending incremental file list
emby-data_sync.sh[23852]: sent 4,561 bytes  received 44 bytes  9,210.00 bytes/sec
emby-data_sync.sh[23852]: total size is 257,320,410  speedup is 55,878.48

The paths used in this post are the default ones for a Debian installation, in the case the distribution used is different could be necessary to change some path. But the main concept remain the same. On my installation I was able to speed up the "quick searches" by 100 times. This improvement is mainly due to the increase of speed obtained by moving the SQLite databases to RAM. In this article: https://www.directionsmag.com/article/3794   is reported an interesting comparison on memory access vs disk access speed and is stated that: "If you have a problem with slow database  access, placing the entire database in RAM can instantly provide a factor-of-100-thousand increase in access speed with no other programming or data structure changes required".

Edited by Dan64
  • Like 2
  • Thanks 2
Link to comment
Share on other sites

The problem is that the access to sqlite database is generally slow. Unless you decided to implement an in memory cache solution for access the database, moving it to RAM can speed up the the "searches" by more than 100 times. In any case I'm waiting for the new version, as the proposed solution is not able to speed up all the Emby's features.

Link to comment
Share on other sites

frabe8378910

Assuming that there is memory to spare, placing Emby's data and/or cache to RAM offers significant advantages. Even though opting for the partial route is an improvement by itself, the latter can make Emby even more responsive compared to its stock configuration. Benefits include, among others:

- reduced drive read/writes;

- heightened responsive feel;

- many operations within Emby, such as quick search are nearly instantaneous.

 

 

 

Thanks for your valueable idea. I'll always appreciate easy to moderate suggestions which could lead to (maybe significant) improvement.

My 8 years old 7x24 debian server running the latest emby Beta (with two SAT cards) has only 4GB RAM - so not to much to spend for caching. But taking your idea a little bit different, I might test this concenct with just adding one fast SSD card and throw cache and DB on that. Limitation for my server could be the speed of the older SATA interface and no free PCIE slot. Do someone already have experience with performance improvements with such an setup?

Thanks in advance.    

Edited by frabe8378910
Link to comment
Share on other sites

If your library is quite big, I can guarantee that the improvement is significant. Previously I had to wait between 20 to 200 sec to get an answer to my searches, now the answers are nearly instantaneous. Recently I decided to upgrade the hardware of my 7x24 debian server and since I was tired about the slow behavior of Emby with my big library; I decided to use, as server disk for OS and programs, a SSD. I was very impressed how much improved Emby (actually all the programs running on my server improved in speed, especially KDE). Unfortunately since all my 4 SATA ports were full, I had to use a USB to SATA bridge to add an internal SATA port using the onboard USB 3 adapter (my only PCIe port was already employed by a TV/turner). Doing so I discovered that the SSD disks when are connected to USB to SATA bridge in Linux are unreliable (It seems a combination of  bridge-firmware/OS issues not yet solved). So I had to remove the SSD and install the OS on a standard 2.5" 5400 rpm disk. As expected Emby and KDE started to run slow using the hard disk setup. I tried to figure out a way to have an SSD like speed using a standard hard disk, and I noted that the amount of space of folder "/var/lib/emby/data" was only 250 mb, small enough to stay in memory without problems and so I tested the approach described in this post, and I can guaranteed that the speed improvement is quite the same of the one previously observed using the SSD (with the only exception of images that, since are not in RAM, aren't loaded instantaneously). If you have low memory I suggest to use "zram" or "zswap" (better in my opinion) to improve the memory usage (data in RAM are compressed).  

Edited by Dan64
  • Like 1
Link to comment
Share on other sites

Tur0k

This is a really interesting approach to addressing latency with slower fixed disk systems.

 

 

Sent from my iPhone using Tapatalk

Link to comment
Share on other sites

  • 1 month later...
  • 1 year later...

I can't get this to work. I have 8 gigs of RAM is that not enough? When I do "systemctl restart emby-server" I get this error

"Job for emby-server.service failed because the control process exited with error code.
See "systemctl status emby-server.service" and "journalctl -xe" for details." I have no idea how to fix it

 

Link to comment
Share on other sites

7 hours ago, Mykids said:

I can't get this to work. I have 8 gigs of RAM is that not enough? When I do "systemctl restart emby-server" I get this error

"Job for emby-server.service failed because the control process exited with error code.
See "systemctl status emby-server.service" and "journalctl -xe" for details." I have no idea how to fix it

 

Hi, what exactly are you trying to do?

  • Like 1
Link to comment
Share on other sites

10 hours ago, Luke said:

Hi, what exactly are you trying to do?

It's okay. I rebooted my PC and it worked, Thanks! I was trying to follow this guide to speed up Emby. I have a large library, emby was taking 10 minutes to load up, search function didn't work, overall everything was slow. This guide fixed everything, Emby loads up instantly now, search has instant results and everything is perfect. Thanks

Link to comment
Share on other sites

1 hour ago, Mykids said:

It's okay. I rebooted my PC and it worked, Thanks! I was trying to follow this guide to speed up Emby. I have a large library, emby was taking 10 minutes to load up, search function didn't work, overall everything was slow. This guide fixed everything, Emby loads up instantly now, search has instant results and everything is perfect. Thanks

Thanks for the feedback.

Link to comment
Share on other sites

  • 1 year later...

I myself wouldn't run from RAM cache but instead use NVMe which can be done on NAS boxes as well.

Check out this thread for Synology specific instructions but essentially just Linux instructions.

 

Link to comment
Share on other sites

  • 3 weeks later...

How likely is this method to break through an Emby update? And how much more RAM usage are we looking at? It's a very interesting prospect and a nice way to avoid SSD wear. And is it possible to extend this to handle the temporary transcode files?

Link to comment
Share on other sites

Q-Droid

A $30 (USD) SSD can last years and give more than good enough performance for metadata and transcoding. Add the newer DB features to improve performance and the benefits of running a RAMdisk don't outweigh the risk or cost.

Link to comment
Share on other sites

Happy2Play
1 hour ago, Vitz said:

How likely is this method to break through an Emby update? And how much more RAM usage are we looking at? It's a very interesting prospect and a nice way to avoid SSD wear. And is it possible to extend this to handle the temporary transcode files?

What is too much wear?  Almost 9 year old ssd.  Lasted longer the most of my HDDs.

image.png.3d9beb973ebfe41af9526ebf40bd75a8.png

Link to comment
Share on other sites

I guess I'm just looking for something to experiment with. Emby is already running on an enterprise-grade SSD that probably won't wear out for another decade, but the web interface isn't quite as snappy as I'd like it to be (there's some overhead due to it running in a VM). It's not going to get any faster considering the growth rate of my library.

This probably isn't worth implementing but it might be fun to play around with.

Link to comment
Share on other sites

Q-Droid
1 hour ago, Vitz said:

I guess I'm just looking for something to experiment with. Emby is already running on an enterprise-grade SSD that probably won't wear out for another decade, but the web interface isn't quite as snappy as I'd like it to be (there's some overhead due to it running in a VM). It's not going to get any faster considering the growth rate of my library.

This probably isn't worth implementing but it might be fun to play around with.

Have you tried adjusting the DB tunables?

  <OptimizeDbOnShutdown>true</OptimizeDbOnShutdown>
  <DatabaseCacheSizeMB>96</DatabaseCacheSizeMB>
  <DatabaseAnalysisLimit>400</DatabaseAnalysisLimit>
 

Link to comment
Share on other sites

15 hours ago, Q-Droid said:

Have you tried adjusting the DB tunables?

  <OptimizeDbOnShutdown>true</OptimizeDbOnShutdown>
  <DatabaseCacheSizeMB>96</DatabaseCacheSizeMB>
  <DatabaseAnalysisLimit>400</DatabaseAnalysisLimit>
 

I'll go ahead and increase the cache size and see if that does anything. What does the analysis limit do?

Link to comment
Share on other sites

Q-Droid
43 minutes ago, Vitz said:

I'll go ahead and increase the cache size and see if that does anything. What does the analysis limit do?

It governs the sample size used when running analyze, number of rows examined per index per scan. The range of values is 100..1000, Emby default is 400(?) and 0 is not allowed by the app. Depending on how big your DB and how long it takes to analyze trying a higher value typically results in better stats but not always noticeable difference in perf. Like most things it depends...  Having current/good stats is a good thing more often than not.

Cache size is the most likely to yield noticeable improvements.

 

Link to comment
Share on other sites

Just now, Q-Droid said:

It governs the sample size used when running analyze, number of rows examined per index per scan. The range of values is 100..1000, Emby default is 400(?) and 0 is not allowed by the app. Depending on how big your DB and how long it takes to analyze trying a higher value typically results in better stats but not always noticeable difference in perf. Like most things it depends...  Having current/good stats is a good thing more often than not.

Cache size is the most likely to yield noticeable improvements.

 

Got it. Thanks for the info.

Link to comment
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now
×
×
  • Create New...