Make sure I understand size requirements for the transcoding directory

[markkundinger 3]

It seems to me that as of right now, if I want to be on the safe size, I need to dedicate a hefty amount of space to my transcoding-temp directory for emby:

 

I should have enough space available to store the full size of the source file for whatever show(s) are being watched at that moment, plus a small buffer.

 

In other words, if there are three 4K remuxes playing at the same time, 80GB each,  and they're all transcoding at once, then I need at least 240GB of transcoding space available.

 

No idea what the requirement would be for live tv, I don't have that feature.

 

I say this from observation when my Apple TVth gen is a client.  It seems to be able to direct stream a Blu-ray or DVD rip except it doesn't like the mkv format.  The resulting transcode file seems to roughly match the file size of the original movie. And then it's deleted when I stop watching the show.

 

Also, it seems that if I run out of disk space, the whole movie (and sometimes the server) just stops with an out of disk space condition, there's no auto-pruning while the show is playing.

 

Does this mean that transcoding to RAM is a bad idea unless I have tremendous amounts of RAM?

 

So in other words, if I'm running in a virtual machine, I can't use the default folder with my piddly small storage dedicated to that VM.  I need to specify something with more serious space capability.

 

Did I get that right?

Edited November 8, 2019 by markkundinger

[Happy2Play 9140]

Currently yes that is correct.

[Luke 38803]

As happy2play said this is correct, but we plan to improve this in the future to be more adaptive to low storage situations.

[markkundinger 3]

That would be good. I'm a Linux noob so when the whole system stopped working when out of space it took me a while to figure out what was going on.

 

In the plus side it let me enforce that my kids could only watch for 15 minutes before bedtime

[Luke 38803]

Thanks for the feedback.

[caffeineshock 14]

is this still true?

right now im transcoding into a ram disk that is 20gb in size and this is sufficient right now. but in the future more ppl will join my server and im afraid i will need to upgrade my ram or switch to a ssd for the transcoding if this is still the case. but i need to know it before i run into an issue due to the size requerements of the transcoding

Edited July 8, 2022 by caffeineshock

[rbjtech 4806]

4 minutes ago, caffeineshock said:

is this still true?

right now im transcoding into a ram disk that is 20gb in size and this is sufficient right now. but in the future more ppl will join my server and im afraid i will need to upgrade my ram or switch to a ssd for the transcoding if this is still the case. but i need to know it before i run into an issue due to the size requerements of the transcoding

A RAM disk will not give you any (noticeable) performance improvements over a SATA SSD yet alone an NVME SSD.    

Size is impossible to say - with 1-2 streams transcoding - 20Gb is probably ok, for 10+ streams transcoding - it's probably not.

NVME's are cheap for the performance/storage you are getting - a lot cheaper than the equivalent DDR4/5 RAM anyway - and ultimately is more than sufficient for the emby transcoding requirement.

[caffeineshock 14]

2 minutes ago, rbjtech said:

A RAM disk will not give you any (noticeable) performance improvements over a SATA SSD yet alone an NVME SSD.    

Size is impossible to say - with 1-2 streams transcoding - 20Gb is probably ok, for 10+ streams transcoding - it's probably not.

NVME's are cheap for the performance/storage you are getting - a lot cheaper than the equivalent DDR4/5 RAM anyway - and ultimately is more than sufficient for the emby transcoding requirement.

i know that ramdisk wont give me this. but ram forgets the data after a power loss so i dont need to encrypt this   dont question my decisions please 

so the transoce requirement is still the same? allright. time for more ssd then

Edited July 8, 2022 by caffeineshock

[rbjtech 4806]

3 minutes ago, caffeineshock said:

i know that ramdisk wont give me this. but ram forgets the data after a power loss so i dont need to encrypt this   dont question my decisions please 

so the transoce requirement is still the same? allright. time for more ssd then

Understand that answers given here are to help the community - not just you.

[caffeineshock 14]

5 minutes ago, rbjtech said:

Understand that answers given here are to help the community - not just you.

theres never been a question about ram transcode speeds or similar. only about size... teaching me about "ram does not give you transcode speed" is going full offtopic sorry
please understand this

so you do not know if its still the same as several years ago? how much space will i need if for example we take lord of the rings UHD HDR that is ~120GB in size and i want to stream it in, lets say, 10mbit fhd

will i need 10mbit*(runtime in seconds)? 
or will it autodelete the files and only keep the transcoded media that is needed on the client for buffering?

Edited July 8, 2022 by caffeineshock
typo

[Luke 38803]

If the server detects low disk space then it will clean up transcoding segments on the fly as you go in order to conserve space.

[caffeineshock 14]

8 hours ago, Luke said:

If the server detects low disk space then it will clean up transcoding segments on the fly as you go in order to conserve space.

cool

that means, for example, if i watch movie with the size of 100 gb and i transcode it while playing it so that the resulting size would be 10gb (just an example) 5gb space would be sufficient because emby would transcode the first 5gb and then clean up the older segments "first in first out" so that the transcoded movie will never be completely in the temp. transcoding directory?

is this right?

[Luke 38803]

3 hours ago, caffeineshock said:

cool

that means, for example, if i watch movie with the size of 100 gb and i transcode it while playing it so that the resulting size would be 10gb (just an example) 5gb space would be sufficient because emby would transcode the first 5gb and then clean up the older segments "first in first out" so that the transcoded movie will never be completely in the temp. transcoding directory?

is this right?

Correct, although if there is enough free space, then the whole thing will be there.

[caffeineshock 14]

13 hours ago, Luke said:

Correct, although if there is enough free space, then the whole thing will be there.

yeah sure ofc. if there is enough space
nice. so in this case i can stay on my ram-disk for the transcoding, since only a small part of the file will be stored there. and after its buffered or played it gets removed

thanks ❤

[MrMackey 31]

Greetings

i was planning to transcode to ram to avoid having to write to the ssd.
So I created a ramdisk with limited size.
In this test I took 1GB to make it faster but I also tried it with 10GB with the same result.
I understood that emby automatically creates space again and deletes the .ts files when the memory is full.
But unfortunately this is not the case with me and the stream stops after a short time when the limit is reached. The .ts files are only deleted when the stream is closed.

Does anyone know what I am doing wrong?

ffmpeg-transcode-f197df58-3776-46df-a79c-b582f9b4deb6_1.txt

Edited September 21, 2022 by MrMackey

[Luke 38803]

12 hours ago, MrMackey said:

Greetings

i was planning to transcode to ram to avoid having to write to the ssd.
So I created a ramdisk with limited size.
In this test I took 1GB to make it faster but I also tried it with 10GB with the same result.
I understood that emby automatically creates space again and deletes the .ts files when the memory is full.
But unfortunately this is not the case with me and the stream stops after a short time when the limit is reached. The .ts files are only deleted when the stream is closed.

Does anyone know what I am doing wrong?

ffmpeg-transcode-f197df58-3776-46df-a79c-b582f9b4deb6_1.txt 67.38 kB · 0 downloads

Hi, please attach the main emby server log as well. Thanks.

[MrMackey 31]

@Luke

Hey
Sure, I had already deleted the logs so I did the test again with a 2gb ramdisk.

To better understand what I did:
I have in Emby Docker in Extra Parameters settings: --mount type=tmpfs,destination=/tmp,tmpfs-size=2000000000

And in Emby in Transcode settings:
/tmp

embyserver.txtffmpeg-transcode-7a38281a-8905-490a-a751-5de7cf3b6668_1.txt

[MrMackey 31]

Hey @Luke

Were you able to determine what might be the problem?

[Happy2Play 9140]

A guess but seems to be a mismatch in paths here.

2022-09-22 04:34:31.516 Info Main: Transcoding temporary files path: /config/transcoding-temp

Eventhough "/tmp/transcoding-temp/F8B2F0/F8B2F0_%d.ts" path is being used.

[Luke 38803]

@MrMackey can you please temporarily enable debug logging under the logs section in the server, then repeat the test and attach the debug server log? Thanks !

[MrMackey 31]

Hey @Luke, Of course thanks for looking at it again!

embyserver.txt ffmpeg-transcode-27d79382-6014-40f0-bbc5-85b94dbf8081_1.txt ffmpeg-transcode-8ff4de56-bffb-473f-8693-50c85e7f5784_1.txt ffmpeg-transcode-016040ac-3f67-4621-b819-7fe81991651e_1.txt  ffmpeg-transcode-04de06eb-9b9a-4191-9eb1-a55ebdd8e9ab_1.txt

 

[Happy2Play 9140]

@Luke I am still confused as to why there is the mismatch unless it is just something with this platform as these should be same path.

2022-10-10 00:59:29.043 Info Main: Transcoding temporary files path: /config/transcoding-temp

/tmp/transcoding-temp/A9D49B/A9D49B.m3u8

[Luke 38803]

44 minutes ago, Happy2Play said:

@Luke I am still confused as to why there is the mismatch unless it is just something with this platform as these should be same path.

2022-10-10 00:59:29.043 Info Main: Transcoding temporary files path: /config/transcoding-temp

/tmp/transcoding-temp/A9D49B/A9D49B.m3u8

It's just a logging mistake that needs to be looked at. the logger is printing the default path.

[MrMackey 31]

Greetings!

@Luke Could you see what might be the problem this time?

[drait 2]

I have a Radian RMS-200 (basically an 8 GB NVMe on a PCIe card) in my Emby server that I mount and use as my transcode directory. If I tune my Live TV (via HDHR 4K) to one of the 4K stations and let it play via Emby Theater, I can watch the RMS mount point fill up to 100% and the stream will eventually freeze. Stopping the Live TV stream will empty the transcoding directory, at which point I can restart playing the channel, and it continues to work. 

 

embyserver.txt