Jump to content

EMBY TRANSCODING AND LOGS TO RAMDRIVE TUTORIAL - FULL GUIDE


Recommended Posts

forenaite
Posted

Hi guys. I hope everyone is well.

This is for Linux and is designed to save the lifespan of your SSDs, without compromising Emby's usability.

Files older than 15 minutes will be removed from the ramdrive every 5, IF the ramdrive is 90% full.

My media server:

I'm running Ubuntu 22.04 LTS on a budget PC with these hardware:

  • Intel(R) Core(TM) i7-6700 CPU @ 3.40GHz
  • 64GB RAM 2133MHZ
  • NVIDIA 2060 GPU for hardware transcoding

Nothing fancy, but it does the job very well. 😄


Full Tutorial:

As not everyone has experience with Linux, I've tried to be as detailed as possible.

1) First, check emby's user and group ID:

# grep ^emby /etc/{passwd,group}
/etc/passwd:emby:x:997:983:Emby Server:/var/lib/emby:/usr/sbin/nologin
/etc/group:emby:x:983:


2) Mount Emby's transcoding-temp and logs directories into ramdrives:

# grep emby /etc/fstab
emby-logs       /var/lib/emby/logs             tmpfs rw,nodev,nosuid,noexec,nodiratime,size=1024M,uid=997,gid=983 0 0
emby-trans      /var/lib/emby/transcoding-temp tmpfs rw,nodev,nosuid,noexec,nodiratime,size=24G,nr_inodes=1m,uid=997,gid=983 0 0
  • I'm using 24GB RAM for transcoding temp, and 1GB RAM for logs.
    • Beware of user and group IDs! They must match or Emby won't be able to use the ramdrive.

Guidelines:

  • I recommend using at least 4GB of RAM for transcoding-temp ramdrive, for every active transcoding.
  • If the video is of high quality (large UHD files / REMUX), the transcoding ramdrive will need more space (e.g. 24 GB).
  • Adjust the size of the ramdrive according to the number of simultaneous users transcoding, taking into account the usual size of your media files.

2.1) Do a reload to inform systemd about the fstab changes:

# systemctl daemon-reload

2.2) Mount your ramdrives:

# mount -a
# df -h -t tmpfs
Filesystem      Size  Used Avail Use% Mounted on
(...)
tmpfs           256M  4.0K  256M   1% /tmp
emby-logs       1.0G  5.5M 1019M   1% /var/lib/emby/logs
emby-trans       24G     0   24G   0% /var/lib/emby/transcoding-temp


3) RAM Drive auto-cleanup
Let's use a systemd shell script to perform regular ramdrive cleanup.

3.1) Cleanup script:

# mkdir -p /var/lib/emby/scripts
# chown emby: /var/lib/emby/scripts

Create a new script with the following contents:

# vi /var/lib/emby/scripts/monitor_emby_temp.sh

#!/bin/bash
# Define directory to monitor, maximum file age (minutes), and disk space threshold
TARGET_DIR="/var/lib/emby/transcoding-temp"
# Don't delete the transcoding files too quickly.
# They need to exist for at least 10 minutes to avoid playback problems.
MAX_FILE_AGE=15 # Minutes
# Remove files older than the above time if the ramdrive is 90% full.
DISK_SPACE_THRESHOLD=90
# Check ramdrive consumption every 5 minutes
CHECK_TIME=5m
# Main loop. Don't change anything down here.
while true; do
# Get current disk usage percentage
DISK_USAGE=$(df -h "$TARGET_DIR" | awk 'NR==2 {print $5}' | sed 's/%//')
# Check if disk usage is above threshold
if [[ "$DISK_USAGE" -ge "$DISK_SPACE_THRESHOLD" ]]; then
  # Find and delete files older than MAX_FILE_AGE minutes, excluding hidden files
  find "$TARGET_DIR" -type f -mmin +$((MAX_FILE_AGE / 60)) -name "*.ts" -not -path '*/.*' -delete
  # Log information about cleanup with disk usage
  echo "$(date) - Cleaned up old files older than $MAX_FILE_AGE mins in $TARGET_DIR (Disk usage: $DISK_USAGE%)" >> "$TARGET_DIR"/emby_temp_cleanup.log 2>&1
fi
sleep "$CHECK_TIME"
done


Change the owner of the script:

# chown emby: /var/lib/emby/scripts/monitor_emby_temp.sh

The script needs execution permission:

# chmod a+x /var/lib/emby/scripts/monitor_emby_temp.sh


4) SYSTEMD UNIT
To run this script at boot, create a unit file for it.

# vi /etc/systemd/system/clear-emby-temp.service

[Unit]
Description=Cleanup script for /var/lib/emby/transcoding-temp
StartLimitIntervalSec=30
StartLimitBurst=5

[Service]
Type=simple
User=emby
ExecStart=/var/lib/emby/scripts/monitor_emby_temp.sh
ExecStop=/usr/bin/killall monitor_emby_temp.sh
Restart=on-failure

[Install]
WantedBy=multi-user.target

4.1) Do a reload to inform systemd about the changes:

# systemctl daemon-reload


4.2) Enable and start the new unit cleanup system:

# systemctl enable clear-emby-temp.service
# systemctl start clear-emby-temp.service
# systemctl status clear-emby-temp.service

● clear-emby-temp.service - Cleanup script for /var/lib/emby/transcoding-temp
     Loaded: loaded (/etc/systemd/system/clear-emby-temp.service; enabled; preset: enabled)
     Active: active (running) since Wed 2024-06-12 01:52:51 -03; 39min ago
   Main PID: 1451 (monitor_emby_te)
      Tasks: 2 (limit: 77032)
     Memory: 4.1M (peak: 5.7M)
        CPU: 366ms
     CGroup: /system.slice/clear-emby-temp.service
             ├─ 1451 /bin/bash /var/lib/emby/scripts/monitor_emby_temp.sh
             └─11971 sleep 5m

Jun 12 01:52:51 ubuntu systemd[1]: Started clear-emby-temp.service - Cleanup script for /var/lib/emby/transcoding-temp.

4.3) Checking the emby user process:

# ps -fu emby
UID          PID    PPID  C STIME TTY          TIME CMD
emby        1451       1  0 01:52 ?        00:00:00 /bin/bash /var/lib/emby/scripts/monitor_emby_temp.sh
emby        1719       1  2 01:52 ?        00:01:12 /opt/emby-server/system/EmbyServer -programdata /var/lib/emby -ffdetect /opt/emby-server/bin/ffdetect -ffmpeg /opt/emby-server/bin/ffm
emby        9115    1719 65 02:17 ?        00:10:44 /opt/emby-server/bin/ffmpeg -loglevel +timing -y -print_graphs_file /var/lib/emby/logs/ffmpeg-transcode-8a2c2030-bd03-4151-b6ab-56a3e9
emby       13244    1451  0 02:32 ?        00:00:00 sleep 5m


Reboot your system and check if the ramdrives and cleanup unit are up and running.

# reboot


5) Show time.

Play a video that needs transcoding, and monitor the use of ramdrive to check that everything is working. You can use a command like this for tracking:

# watch -d 'free -h ; echo ; df -h -t tmpfs|grep emby ; echo ; ls -la /var/lib/emby/transcoding-temp/* | head -30'

For my environment, this technique has made a noticeable difference to transcoding startup response times, as well as saving my SSD's lifespan. 🙂

I hope this tutorial has been useful to some people who have been waiting for a working ramdrive solution for Emby for some time.


Bonus:

If you want to use a ramdrive for the /tmp directory (recommended), do the following.

# sudo cp /usr/share/systemd/tmp.mount /etc/systemd/system/tmp.mount
# sudo systemctl enable tmp.mount
# sudo systemctl start tmp.mount
  • This will automatically use half of your RAM to ramdrive the /tmp directory. To adjust the size, simply edit the file /etc/systemd/system/tmp.mount and replace the parameter “size=50%%” by something like “size=256M”.
  • Like 1
Posted

Appreciate the guide but there is absolutely no reason to do this.

 

  • Haha 1
forenaite
Posted

Fair enough. Feel free not to use it.

Neminem
Posted (edited)

Try that with the Le-mans race this weekend.

And then try to skip back 2 hours 🤣😂

Without recording it 😊

Edited by jaycedk
forenaite
Posted

Works for movies. I have no live TV on Emby for testing. Can you check it out for us?

  • 1 year later...
bandit8623
Posted
On 6/12/2024 at 9:08 AM, Lessaj said:

Appreciate the guide but there is absolutely no reason to do this.

 

This person hates ramdisks. i think they jealous they dont have that much ram..  or doesnt know anything about ssd's and how writes are bad.

  • Disagree 1
Posted
10 hours ago, bandit8623 said:

This person hates ramdisks. i think they jealous they dont have that much ram..  or doesnt know anything about ssd's and how writes are bad.

The myth that SSD's and NVME's today are small fragile things that needs babysitting all the time is greatly  exaggerated.

You would have to constantly write at blazing speeds to kill a NVME within any time that people would deem fast.
I used to write chia plots to SSD's and NVMEs that are still running strong today.

Samsung SSD Write Endurance Test – How Long Will Samsung 990 EVO, 990 PRO, and 870 EVO Last? – Advanced Data Recovery for NAS, SSD & RAID Systems
They tested the Samsung 990 Pro to 28 petabytes and still working. :)

The endurance written in the technical specs is very conservative.

  • Agree 1
GrimReaper
Posted
8 minutes ago, yocker said:

The myth that SSD's and NVME's today are small fragile things that needs babysitting all the time is greatly  exaggerated.

You would have to constantly write at blazing speeds to kill a NVME within any time that people would deem fast.
I used to write chia plots to SSD's and NVMEs that are still running strong today.

Samsung SSD Write Endurance Test – How Long Will Samsung 990 EVO, 990 PRO, and 870 EVO Last? – Advanced Data Recovery for NAS, SSD & RAID Systems
They tested the Samsung 990 Pro to 28 petabytes and still working. :)

The endurance written in the technical specs is very conservative.

Besides, prices have come down so much that TBW/DWPD shouldn't even be a factor anymore. 

  • Like 1
bandit8623
Posted
40 minutes ago, yocker said:

The myth that SSD's and NVME's today are small fragile things that needs babysitting all the time is greatly  exaggerated.

You would have to constantly write at blazing speeds to kill a NVME within any time that people would deem fast.
I used to write chia plots to SSD's and NVMEs that are still running strong today.

Samsung SSD Write Endurance Test – How Long Will Samsung 990 EVO, 990 PRO, and 870 EVO Last? – Advanced Data Recovery for NAS, SSD & RAID Systems
They tested the Samsung 990 Pro to 28 petabytes and still working. :)

The endurance written in the technical specs is very conservative.

My drives are all used with 35 % health left.  So yes I do manage the writes.  Everyones situation is different.  But your stats are correct. End of day ram disks work great.

bandit8623
Posted
35 minutes ago, GrimReaper said:

Besides, prices have come down so much that TBW/DWPD shouldn't even be a factor anymore. 

100tb for brand new ssds cheap eh?

GrimReaper
Posted
5 minutes ago, bandit8623 said:

100tb for brand new ssds cheap eh?

Aside from that that's completely unrelated to topic at hand, RAMDisk/SSD transcoding for which you'd use a cheapo few tens of USD SSD (if you're really keen on using that in the first place) which will ikely last several times over its worth  - why would you need a 100TB of SSDs? 

bandit8623
Posted
20 minutes ago, GrimReaper said:

Aside from that that's completely unrelated to topic at hand, RAMDisk/SSD transcoding for which you'd use a cheapo few tens of USD SSD (if you're really keen on using that in the first place) which will ikely last several times over its worth  - why would you need a 100TB of SSDs? 

I'm Not going to add an extra drive to a system that aleady has 256gb of ram.. why not just use the ram disk...it's basically free.  I didn't buy that ram for a ram disk.. it's already there. The whole issue stemmed from the guy saying ram disk has zero benefits.. being free and not putting writes on my current ssds is another.

bandit8623
Posted (edited)
26 minutes ago, GrimReaper said:

Aside from that that's completely unrelated to topic at hand, RAMDisk/SSD transcoding for which you'd use a cheapo few tens of USD SSD (if you're really keen on using that in the first place) which will ikely last several times over its worth  - why would you need a 100TB of SSDs? 

Would I go out and buy ram to make a ramdisk? No.. I would probably get 1 drive if I had space...  But if you already have the ram capacity a ram disk is great.

Edited by bandit8623
GrimReaper
Posted (edited)
11 hours ago, bandit8623 said:

This person hates ramdisks. i think they jealous they dont have that much ram..  or doesnt know anything about ssd's and how writes are bad.

 

15 minutes ago, bandit8623 said:

The whole issue stemmed from the guy saying ram disk has zero benefits..

 

12 minutes ago, bandit8623 said:

But if you already have the ram capacity a ram disk is great

If it works for you - great. But the "guy" is one of the most prominent Emby Devs, particularly in video manipulation/transcoding department, number of things you use daily are his handywork, and if he finds (and provides reasonable explanation) RAMDisk usage for this particular purpose potentially/likely detrimental - I tend to believe that. Anyway, if you're happy - I'm happy too, having a smooth running Emby server is always a pleasant news, it's typically (and understandably, these being technical support forums) issue upon issue day after day. 

Edited by GrimReaper
bandit8623
Posted
3 minutes ago, GrimReaper said:

 

 

If it works for you - great. But the "guy" is one of the most prominent Emby Devs, particularly in video manipulation/transcoding department, number of things you use daily are his handywork, and if he finds (and provides reasonable explanation) RAMDisk usage for this particular purpose potentially/likely detrimental - I tend to believe that. Anyway, if you're happy - I'm happy too, having a smooth running Emby server is always a pleasant news, it's typically (and understandably, these being technical support forums) issue upon issue day after day. 

no not the emby dev....  Lessaj  is the one im talking about has hijacked 2 threads now at least spewing nonsense about how ram drives are bad.  

 

GrimReaper
Posted
3 minutes ago, bandit8623 said:

no not the emby dev....  Lessaj  is the one im talking about has hijacked 2 threads now at least spewing nonsense about how ram drives are bad.  

Maybe because he tends to agree with @softworkzas well, as per linked topic...? Have you followed that one you quoted and saw the Devs argumentation for it? 

  • Agree 1
bandit8623
Posted (edited)
32 minutes ago, GrimReaper said:

Maybe because he tends to agree with @softworkzas well, as per linked topic...? Have you followed that one you quoted and saw the Devs argumentation for it? 

yep.  and his remarks are also not all true.  depends on how the ram disk is implemented as well.  also depends on your hardware capabilities.  

a 4 core pc with slow ram is going to be a bad candidate for ramdisk.   i totaly get most people here are probly not using enterprise server rated gear,  but some are.

 

 

Edited by bandit8623
Posted
3 hours ago, bandit8623 said:

My drives are all used with 35 % health left.  So yes I do manage the writes.  Everyones situation is different.  But your stats are correct. End of day ram disks work great.

I personally don't see any benefit and mostly downsides from using a ramdisk for transcoding but i do agree, it's personal preference and use case plus using ramdisk will work perfectly fine without problems for most.

What works best for a person is usually the best solution for that person. :)

 

  • Like 1
Posted

I've been using it for over a year. It works very well for my use case (which, of course, may be different for other people). I use ramdrives for other things, and the performance gain is relevant for cases where it makes sense. For others, as some know, it may be useless. I find this “do” or “don't do” flame war curious 😁. Personally, I try to learn when someone posts something more technical and interesting.

  • Like 1
bandit8623
Posted
2 minutes ago, forenaite said:

I've been using it for over a year. It works very well for my use case (which, of course, may be different for other people). I use ramdrives for other things, and the performance gain is relevant for cases where it makes sense. For others, as some know, it may be useless. I find this “do” or “don't do” flame war curious 😁. Personally, I try to learn when someone posts something more technical and interesting.

i would call it more of a debate/disagreement/learning.  not really a flame war  :)

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now
×
×
  • Create New...