Transcode to ram

[steve1977 67]

Is there an easy way to use ram for transcoding. I am running Emby server in an Unraid docker. Thanks!

[NikeSwe 65]

This is how I did it.

 

Mapped /mnt/tmp in container to /tmp/emby on host (if folder not there, docker will create it when starting the Emby container, do not create it manually. Because in ram, the folder will not be there when rebooting unraid but it will be recreated when the container starts). Do not use the /tmp-folder in the container as it will store the transcoded stuff inside the docker image.
 

In Emby, server settings, transcoding, set path for temp transcoding storage there to /mnt/tmp

 

Everything not in /mnt on host = ram (when running unraid).

 

I think you could do it this way. I just did it minutes ago. I think I was reading that the max ram storage is half of the installed ram.

 

The mapping in Unraid:

 

On the host, you can browse to the /tmp/emby folder using SSH or console, Emby will create "/tmp/emby/transcoding-temp" folder there when first needed.

 

What happens if ram runs out or storage full? Don't know yet.

 

I use SSD as cache but I want to limit writes to it. This will not give me any performance boost but it will make my ssd last longer. How Emby handles the restricted storage space, I don't know. Yet.

Edited October 23, 2018 by NikeSwe

[Happy2Play 8282]

What happens if ram runs out or storage full? Don't know yet.

 

I use SSD as cache but I want to limit writes to it. This will not give me any performance boost but it will make my ssd last longer. How Emby handles the restricted storage space, I don't know. Yet.

The transcoded media will stop when you run out of storage space.  Emby currently does not do block transcoding.  There is another topic on this.  I believe it is in the Emby - Plex topic also.

[NikeSwe 65]

The transcoded media will stop when you run out of storage space.  Emby currently does not do block transcoding.  There is another topic on this.  I believe it is in the Emby - Plex topic also.

I think I saw it when I wanted to try this. Just saw one thread on this over at the unraid forum regarding Plex, almost the same result as how I did it. Seems like Emby and Plex handle transcoding differently but as I understand it, it may be considered for Emby too?

Edited October 23, 2018 by NikeSwe

[NikeSwe 65]

I have 5 years warranty but "only" 150 TBW on my cache ssd. Samsung 860 Evo (250GB). It will probably not be a problem but I want to limit the amount of writes, if I can.

Edit: I could transcode to the array but that will eventually slow it down...
Im new to Unraid (and Docker). Just two weeks in to it. Didn't know it existed before that, but..  WOW.

Edited October 23, 2018 by NikeSwe

[Happy2Play 8282]

@@NikeSwe

 

This topic covers the questions also.

 

https://emby.media/community/index.php?/topic/62109-emby-transcoder-fills-up-32gb-ramdisk-then-stops-playing

[NikeSwe 65]

Thanks.

 

I'm going back to transcode on ssd for the time being. My server has 16GB ram. Rootfs is at 7,7 (about half the ram). Filling up quickly on live tv.

[pir8radio 1292]

Thanks.

 

I'm going back to transcode on ssd for the time being. My server has 16GB ram. Rootfs is at 7,7 (about half the ram). Filling up quickly on live tv.

 

Yea not worth it...   A cheap SSD would be fine.   I seldom see over 60 or 70 Mbps on my transcode SSD, and that drive is capable of:

         Sequential Read (T= 1) :   523.235 MB/s

         Sequential Write (T= 1) :   197.775 MB/s

And I have quite a few streams going on at times, DVR, IPTV in & out movies being played...  Keep in mind drive is tested at Mega Bytes/s, graph is Mega Bits/s so there is plenty of room for many streams to be transcoded.

 

[NikeSwe 65]

For me, it has nothing to do with speed. Just want to save some writes to the ssd.

[pir8radio 1292]

For me, it has nothing to do with speed. Just want to save some writes to the ssd.

 

Gotcha'  Yea thats why i said "cheap ssd" lol, I have not lost my first one yet, but i suspect its coming... 

[Happy2Play 8282]

I have 5 years warranty but "only" 150 TBW on my cache ssd. Samsung 860 Evo (250GB). It will probably not be a problem but I want to limit the amount of writes, if I can.

 

Edit: I could transcode to the array but that will eventually slow it down...

Im new to Unraid (and Docker). Just two weeks in to it. Didn't know it existed before that, but..  WOW.

 

It comes down to how much you transcode but here is my 180GB ssd in my stable server after 5 and half years.

 

[NikeSwe 65]

It comes down to how much you transcode but here is my 180GB ssd in my stable server after 5 and half years.

 

and all other stuff you can do with the cache ssd. Not only used for transcoding.

 

My Intel SSD in my main workstation has 100% drive health and 100% estimated life remaining after 2 years power on time, but still. Limit the LBAs written is not a bad thing when talking SSDs.

My server, 16 GB ram with about 2-3 used. I have 10GB+ that I could use for temporary storage if Emby handled the transcoding differently. With 0 wear on the ssd.

Edited October 24, 2018 by NikeSwe

[Q-Droid 644]

I have 5 years warranty but "only" 150 TBW on my cache ssd. Samsung 860 Evo (250GB). It will probably not be a problem but I want to limit the amount of writes, if I can.

 

Edit: I could transcode to the array but that will eventually slow it down...

Im new to Unraid (and Docker). Just two weeks in to it. Didn't know it existed before that, but..  WOW.

 

What makes you think transcoding to the array will slow it down? As someone else mentioned I/O is not the bottleneck when transcoding which is measured in double digit mbit/s while HDDs and SSDs handle triple digit MB/s. 

[NikeSwe 65]

Takes time to spin up sleeping disks and why would I do that with temporary data? What is the problem here. We have ram available and want to use it. Not for speed in my case but to save some wear on the ssd for highly temporary data.

Edit: And the array are quite slow with the parity disk and all. (I have no parity disk, yet. hehe).

I do other stuff on my server too. It is not Emby only. Just to be clear.

Edited October 24, 2018 by NikeSwe

[sluggo45 47]

With Unraid, mapping a path, "EMBYTMP", whatever, to /tmp in the Docker container settings then configuring Emby to use "EMBYTMP" (or whatever you called it) works fine, but you really need to be ram-rich to do this.

 

I have 128GB in my Unraid server so in theory, why not, but for folks with 16 or 32GB you're probably going to run in to problems at some point. And unless you are beating the crap out of it with multiple encodes all day every day it's really not going to do much harm to an ssd to use it for temp files like this.

 

What folks do miss is the shares you have Emby use for temp folders should be configured with COW (Commit on Write) disabled. If you are, say, using two ssds as a cache drive array and it uses btrfs (which is the default for Unraid in recent releases). You may not notice a performance difference but COW with btrfs that scenario will result in more writes and for temporary files is also pointless.

 

Newer versions of Unraid have a setting, Enable Copy-on-write, for shares. If it's on an SSD based btrfs array make sure the share you use for temp files, etc. has that disabled. This goes for other transitory data like for example SABNZB downloads (that are then renamed and moved to your array) and so on.

[NikeSwe 65]

Still not talking about speed.... I don't want to wake up my array because of temporary transcoding storage that easily could be done in ram only.

 

I have one 250 GB Samsung 860 Evo as cache. Btrfs yes.
If Emby handled transcoding as Plex does, it would work fine but as it is now, transcoding live tv to a file that grows and grows and grows is not ram friendly.

 

edit: Btw, can't change the COW-settings on the standard shares. Guess I need to recreate the shares to be able to change that and as of now, I need to learn more about COW and Btrfs.

 

Edit: My ONE and ONLY goal with this is to limit the LBAs written on my SSD.
Nothing less, nothing more. Don't know what the thread starter has in mind.

The SSDs warranty is 5 years OR 150 TBW. I would like to have the warranty for the full 5 years.

Can't see any good arguments for not placing transcoding in ram if you have the space and lots of not used ram. Nothing wrong or strange in saving your SSD... I limit the writes to the ssd for ALL stuff I use it for. Not only for transcoding while still benefit from the speed it gives me (connected using sata6 and so on). We say "många bäckar små" in Swedish. Google translate say "every little helps"
I probably don't need to fiddle with this. The drive will last. But... like.. if I can limit it without performance hit, I will. Emby could split the files in smaller chunks and remove the older ones. If we need to skip, it could start transcoding from the place we skip to (as it already does). No big problem and Luke is talking about looking at this for a future release in the thread that Happy2Play linked earlier: https://emby.media/community/index.php?/topic/62109-emby-transcoder-fills-up-32gb-ramdisk-then-stops-playing

Edited October 24, 2018 by NikeSwe

[sluggo45 47]

You are overthinking this problem. Babying a 250GB Evo 860 like this? Look, you aren't going to wear that thing out before you chuck it, ok? Not using it for this, anyway. Believe it, or don't, but your ssd is going to last the same amount of time either way.

 

Commit-on-write is also really only useful if you have two or more devices in an array. And for temp files that change a lot it isn't useful at all.

 

Create a new share for temp storage (you can use this for more than Emby, and on the same drive) and disable COW. Map that to a folder Emby can use for temp recordings. Done. Don't mess around with your default folders, there's no need, it's not like there is a penalty for creating a new share in Unraid.

[NikeSwe 65]

You are overthinking this problem. Babying a 250GB Evo 860 like this? Look, you aren't going to wear that thing out before you chuck it, ok? Not using it for this, anyway. Believe it, or don't, but your ssd is going to last the same amount of time either way.

 

Commit-on-write is also really only useful if you have two or more devices in an array. And for temp files that change a lot it isn't useful at all.

 

Create a new share for temp storage (you can use this for more than Emby, and on the same drive) and disable COW. Map that to a folder Emby can use for temp recordings. Done. Don't mess around with your default folders, there's no need, it's not like there is a penalty for creating a new share in Unraid.

Yes, I agree. Overthinking but why not? I just don't want to reach 150TBW before 5 years has passed. IF we can limit the wear by writing to RAM. WHY not??

Could switch to Plex... or run Plex side by side to test (I LOVE Docker). 

 

Edit:

My Samsung HDDs from my old server works great. Manufactured 2007.09, 2010.01 and 2010.03. Was running 24/7 for at least 6-7 years and works great even today. Only used SSD for like 2 years. The one in my workstation and the new one in the new unraid server.

Edited October 24, 2018 by NikeSwe

[NikeSwe 65]

I started to spin down my drives after going to Unraid. Reading online, its a topic that could generate page after page about if it is better to leave them running all the time vs spinning them up when needed. No clear answer, most anecdotal stuff. More "do what feels best for you", and I do, spin them down and use the SSD for more stuff. For me, it has nothing to do with saving electricity. The SSD will probably outlast the life of the server even if used for transcoding too. Just being a bit... overprotective with my new baby. It will calm down. Soon...... lol.

Edited October 24, 2018 by NikeSwe

[NikeSwe 65]

My main Windows workstation with Intel SSD.
 

Runtime almost exactly 2 years.

Only 26 TBW (no transcoding, hehe)

 

Looking good.

Edited October 24, 2018 by NikeSwe

[NikeSwe 65]

I have 3 drives (2 spinners, 0 parity until afford new big disk, 1 ssd as cache) in my server as of now. The first one is not full and have several TBs free. Disk 2 is not used and is spun down most of the time. Now, I transcode on disk 2, save finished data to share on disk 1. Would still like to transcode in ram but this is OK.
 

Appdata (cache, metadata, data etc) for Emby on cache SSD. XFS file system, had some troubles with databases locking up in containers, other people with similiar problems changed from btrfs to xfs and the problem disappeared, let's see.... The docker image is still btrfs but appdata and video files stored outside the image anyways..

This is because of live tv. Recording. Transcoding and saving to two files at the same time. Want to spread the burden on the individual disks. Or... i mean... it will do the same thing if watching something that needs transcoding but in that case, we have reading from disk 1, transcoding (writing) to disk 2 and then reading from disk 2. Whatever. This works for now.

Like I said. Emby is not the only thing running on the server.  Doing some heavy developing work and other stuff.

Leaving this here to give others some ideas for how you CAN do it. I'm not the thread starter.

Edited October 25, 2018 by NikeSwe

[shummo 0]

HI. My problem, that if I add the /tmp/emby (host) to my volumes, when container started, the folder will be with permission 755 (root:root) created on host. So my USER(PUID1000) in docker cant write it...

Is there any solution to forve docker to create the non exist folder with permission 775 before mount it to the container? 

      environment:
        - PUID=1000 (non root user)
        - PGID=1000
        - GIDLIST=44,105
        - UMASK=002
      volumes:
       - /tmp/emby:/mnt/tmp
Thanks for help

[Luke 37068]

6 hours ago, shummo said:

HI. My problem, that if I add the /tmp/emby (host) to my volumes, when container started, the folder will be with permission 755 (root:root) created on host. So my USER(PUID1000) in docker cant write it...

Is there any solution to forve docker to create the non exist folder with permission 775 before mount it to the container? 

      environment:
        - PUID=1000 (non root user)
        - PGID=1000
        - GIDLIST=44,105
        - UMASK=002
      volumes:
       - /tmp/emby:/mnt/tmp
Thanks for help

Hi, are you trying to transcode to RAM?