How to speed up Emby on Linux

[Dan64 30]

Assuming that there is memory to spare, placing Emby's data and/or cache to RAM offers significant advantages. Even though opting for the partial route is an improvement by itself, the latter can make Emby even more responsive compared to its stock configuration. Benefits include, among others:

- reduced drive read/writes;
- heightened responsive feel;
- many operations within Emby, such as quick search are nearly instantaneous.

 

To do so we can make use of a tmpfs.

 

Because data placed therein cannot survive a shutdown, a script responsible for syncing back to drive prior to system shutdown is necessary if persistence is desired. On the other hand, only relocating the cache is a quick, less inclusive solution that will slightly speed up user experience with Emby.

 

In this post is provided an example of script to move and sync the "data" folder (where is stored the emby db) to RAM.  Be sure that rsync is installed and save the "executable" script to /opt/emby-server/bin/emby-data_sync.sh, for example:

#!/bin/sh
# The script will first move emby's data to a new static location,
# make a sub-directory in /dev/shm, softlink to it and later populate it with the contents of data.
# The second time the script runs, it will then preserve the RAM profile by copying (sync) it back to disk.

PATH=/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin

static=data-static
link=data
volatile=/dev/shm/emby-data

IFS=
set -efu

cd /var/lib/emby

if [ ! -r $volatile ]; then
        mkdir -m0700 $volatile
fi

if [ "$(readlink $link)" != "$volatile" ]; then
        echo "--> Emby: Init RAM Disk"
        mv $link $static
        ln -s $volatile $link
fi

if [ -e $link/.unpacked ]; then
        echo "--> Emby: Sync Disk"
        rsync -av --delete --exclude .unpacked ./$link/ ./$static/
else
        echo "--> Emby: Sync RAM"
        rsync -av ./$static/ ./$link/
        touch $link/.unpacked
fi

It is best to add a cron job to guarantee the sync between RAM and disk, for example the following entry will run the sync process every 30min.

*/30 * * * * /opt/emby-server/bin/emby-data_sync.sh

Then it is necessary to change the service: /lib/systemd/system/emby-server.service to guarantee the initial and final sync, as follow:

[Unit]
Description=Emby Server is a personal media server with apps on just about every device.
After=network.target

[Service]
TimeoutSec=5min
EnvironmentFile=/etc/emby-server.conf
WorkingDirectory=/opt/emby-server
ExecStartPre=/opt/emby-server/bin/emby-data_sync.sh
ExecStart=/opt/emby-server/bin/emby-server
ExecStopPost=/opt/emby-server/bin/emby-data_sync.sh
RestartForceExitStatus=3
User=emby

[Install]
WantedBy=multi-user.target

After this change it is necessary reload the service with the command:

systemctl daemon-reload

Finally it is possible to activate the shared-memory for Emby with the command

systemctl restart emby-server

It is possible to verify the execution of the script with the command:

systemctl status emby-server

● emby-server.service - Emby Server is a personal media server with apps on just about every device.
   Loaded: loaded (/lib/systemd/system/emby-server.service; enabled; vendor preset: enabled)
   Active: active (running) since Tue 2019-01-01 09:46:44 CET; 7min ago
  Process: 8310 ExecStopPost=/opt/emby-server/bin/emby-data_sync.sh (code=exited, status=0/SUCCESS)
  Process: 8315 ExecStartPre=/opt/emby-server/bin/emby-data_sync.sh (code=exited, status=0/SUCCESS)
 Main PID: 8320 (EmbyServer)
    Tasks: 23 (limit: 4915)
   Memory: 676.5M
   CGroup: /system.slice/emby-server.service
           └─8320 /opt/emby-server/system/EmbyServer -programdata /var/lib/emby -ffmpeg /opt/emby-server/bin/ffmpeg

More details can be found with the command:

journalctl | grep emby-data_sync.sh

emby-data_sync.sh[23844]: sending incremental file list
emby-data_sync.sh[23844]: activitylog.db-wal
emby-data_sync.sh[23844]: authentication.db-wal
emby-data_sync.sh[23844]: library.db
emby-data_sync.sh[23844]: library.db-wal
emby-data_sync.sh[23844]: users.db-wal
emby-data_sync.sh[23844]: ScheduledTasks/6330ee8f-b4a9-57f3-3981-f89aa78b030f.js
emby-data_sync.sh[23844]: ScheduledTasks/81267bb8-bb49-51de-4b1b-446262cc7365.js
emby-data_sync.sh[23844]: ScheduledTasks/d459f39d-7abb-81f5-fcce-03048a7ecf4d.js
emby-data_sync.sh[23844]: SyncData/Emby.Kodi.SyncQueue.I.1.40.json
emby-data_sync.sh[23844]: sent 255,467,606 bytes  received 227 bytes  30,055,039.18 bytes/sec
emby-data_sync.sh[23844]: total size is 257,320,410  speedup is 1.01
emby-data_sync.sh[23852]: sending incremental file list
emby-data_sync.sh[23852]: sent 4,561 bytes  received 44 bytes  9,210.00 bytes/sec
emby-data_sync.sh[23852]: total size is 257,320,410  speedup is 55,878.48

The paths used in this post are the default ones for a Debian installation, in the case the distribution used is different could be necessary to change some path. But the main concept remain the same. On my installation I was able to speed up the "quick searches" by 100 times. This improvement is mainly due to the increase of speed obtained by moving the SQLite databases to RAM. In this article: https://www.directionsmag.com/article/3794   is reported an interesting comparison on memory access vs disk access speed and is stated that: "If you have a problem with slow database  access, placing the entire database in RAM can instantly provide a factor-of-100-thousand increase in access speed with no other programming or data structure changes required".

Edited January 1, 2019 by Dan64

[Luke 38830]

Hi there, what version of Emby Server?

[Dan64 30]

Hi there, what version of Emby Server?

I'm using Emby version 3.5.3.0 on Debian 10.0

[Luke 38830]

Ok, the upcoming 3.6 release will have considerably improved performance. thanks.

[Dan64 30]

The problem is that the access to sqlite database is generally slow. Unless you decided to implement an in memory cache solution for access the database, moving it to RAM can speed up the the "searches" by more than 100 times. In any case I'm waiting for the new version, as the proposed solution is not able to speed up all the Emby's features.

[Luke 38830]

That will be improved with the next release. Thanks.

[frabe8378910 3]

Assuming that there is memory to spare, placing Emby's data and/or cache to RAM offers significant advantages. Even though opting for the partial route is an improvement by itself, the latter can make Emby even more responsive compared to its stock configuration. Benefits include, among others:

- reduced drive read/writes;

- heightened responsive feel;

- many operations within Emby, such as quick search are nearly instantaneous.

 

 

 

Thanks for your valueable idea. I'll always appreciate easy to moderate suggestions which could lead to (maybe significant) improvement.

My 8 years old 7x24 debian server running the latest emby Beta (with two SAT cards) has only 4GB RAM - so not to much to spend for caching. But taking your idea a little bit different, I might test this concenct with just adding one fast SSD card and throw cache and DB on that. Limitation for my server could be the speed of the older SATA interface and no free PCIE slot. Do someone already have experience with performance improvements with such an setup?

Thanks in advance.    

Edited January 3, 2019 by frabe8378910

[Dan64 30]

If your library is quite big, I can guarantee that the improvement is significant. Previously I had to wait between 20 to 200 sec to get an answer to my searches, now the answers are nearly instantaneous. Recently I decided to upgrade the hardware of my 7x24 debian server and since I was tired about the slow behavior of Emby with my big library; I decided to use, as server disk for OS and programs, a SSD. I was very impressed how much improved Emby (actually all the programs running on my server improved in speed, especially KDE). Unfortunately since all my 4 SATA ports were full, I had to use a USB to SATA bridge to add an internal SATA port using the onboard USB 3 adapter (my only PCIe port was already employed by a TV/turner). Doing so I discovered that the SSD disks when are connected to USB to SATA bridge in Linux are unreliable (It seems a combination of  bridge-firmware/OS issues not yet solved). So I had to remove the SSD and install the OS on a standard 2.5" 5400 rpm disk. As expected Emby and KDE started to run slow using the hard disk setup. I tried to figure out a way to have an SSD like speed using a standard hard disk, and I noted that the amount of space of folder "/var/lib/emby/data" was only 250 mb, small enough to stay in memory without problems and so I tested the approach described in this post, and I can guaranteed that the speed improvement is quite the same of the one previously observed using the SSD (with the only exception of images that, since are not in RAM, aren't loaded instantaneously). If you have low memory I suggest to use "zram" or "zswap" (better in my opinion) to improve the memory usage (data in RAM are compressed).  

Edited January 4, 2019 by Dan64

[Tur0k 144]

This is a really interesting approach to addressing latency with slower fixed disk systems.

 

 

Sent from my iPhone using Tapatalk

[/Ky 3]

May I ask, why is rsync required? Is it to run the script or just to cron the job every 30min??

[Mykids 0]

I can't get this to work. I have 8 gigs of RAM is that not enough? When I do "systemctl restart emby-server" I get this error

"Job for emby-server.service failed because the control process exited with error code.
See "systemctl status emby-server.service" and "journalctl -xe" for details." I have no idea how to fix it

 

[Luke 38830]

7 hours ago, Mykids said:

I can't get this to work. I have 8 gigs of RAM is that not enough? When I do "systemctl restart emby-server" I get this error

"Job for emby-server.service failed because the control process exited with error code.
See "systemctl status emby-server.service" and "journalctl -xe" for details." I have no idea how to fix it

 

Hi, what exactly are you trying to do?

[Mykids 0]

10 hours ago, Luke said:

Hi, what exactly are you trying to do?

It's okay. I rebooted my PC and it worked, Thanks! I was trying to follow this guide to speed up Emby. I have a large library, emby was taking 10 minutes to load up, search function didn't work, overall everything was slow. This guide fixed everything, Emby loads up instantly now, search has instant results and everything is perfect. Thanks

[Luke 38830]

1 hour ago, Mykids said:

It's okay. I rebooted my PC and it worked, Thanks! I was trying to follow this guide to speed up Emby. I have a large library, emby was taking 10 minutes to load up, search function didn't work, overall everything was slow. This guide fixed everything, Emby loads up instantly now, search has instant results and everything is perfect. Thanks

Thanks for the feedback.

[Carlo 4479]

I myself wouldn't run from RAM cache but instead use NVMe which can be done on NAS boxes as well.

Check out this thread for Synology specific instructions but essentially just Linux instructions.

 

[Vitz 1]

How likely is this method to break through an Emby update? And how much more RAM usage are we looking at? It's a very interesting prospect and a nice way to avoid SSD wear. And is it possible to extend this to handle the temporary transcode files?

[Q-Droid 830]

A $30 (USD) SSD can last years and give more than good enough performance for metadata and transcoding. Add the newer DB features to improve performance and the benefits of running a RAMdisk don't outweigh the risk or cost.

[Happy2Play 9140]

1 hour ago, Vitz said:

How likely is this method to break through an Emby update? And how much more RAM usage are we looking at? It's a very interesting prospect and a nice way to avoid SSD wear. And is it possible to extend this to handle the temporary transcode files?

What is too much wear?  Almost 9 year old ssd.  Lasted longer the most of my HDDs.

[Vitz 1]

I guess I'm just looking for something to experiment with. Emby is already running on an enterprise-grade SSD that probably won't wear out for another decade, but the web interface isn't quite as snappy as I'd like it to be (there's some overhead due to it running in a VM). It's not going to get any faster considering the growth rate of my library.

This probably isn't worth implementing but it might be fun to play around with.

[Q-Droid 830]

1 hour ago, Vitz said:

I guess I'm just looking for something to experiment with. Emby is already running on an enterprise-grade SSD that probably won't wear out for another decade, but the web interface isn't quite as snappy as I'd like it to be (there's some overhead due to it running in a VM). It's not going to get any faster considering the growth rate of my library.

This probably isn't worth implementing but it might be fun to play around with.

Have you tried adjusting the DB tunables?

  <OptimizeDbOnShutdown>true</OptimizeDbOnShutdown>
  <DatabaseCacheSizeMB>96</DatabaseCacheSizeMB>
  <DatabaseAnalysisLimit>400</DatabaseAnalysisLimit>
 

[Vitz 1]

15 hours ago, Q-Droid said:

Have you tried adjusting the DB tunables?

  <OptimizeDbOnShutdown>true</OptimizeDbOnShutdown>
  <DatabaseCacheSizeMB>96</DatabaseCacheSizeMB>
  <DatabaseAnalysisLimit>400</DatabaseAnalysisLimit>
 

I'll go ahead and increase the cache size and see if that does anything. What does the analysis limit do?

[Q-Droid 830]

43 minutes ago, Vitz said:

I'll go ahead and increase the cache size and see if that does anything. What does the analysis limit do?

It governs the sample size used when running analyze, number of rows examined per index per scan. The range of values is 100..1000, Emby default is 400(?) and 0 is not allowed by the app. Depending on how big your DB and how long it takes to analyze trying a higher value typically results in better stats but not always noticeable difference in perf. Like most things it depends...  Having current/good stats is a good thing more often than not.

Cache size is the most likely to yield noticeable improvements.

 

[Vitz 1]

Just now, Q-Droid said:

It governs the sample size used when running analyze, number of rows examined per index per scan. The range of values is 100..1000, Emby default is 400(?) and 0 is not allowed by the app. Depending on how big your DB and how long it takes to analyze trying a higher value typically results in better stats but not always noticeable difference in perf. Like most things it depends...  Having current/good stats is a good thing more often than not.

Cache size is the most likely to yield noticeable improvements.

 

Got it. Thanks for the info.