Transcoding-temp full

[Gabi92 0]

28 minutes ago, Lessaj said:

Yes it should be fast enough, especially with throttling enabled. If you really did need to transcode that file 8x at the same time you're probably going to run into a GPU bottleneck first, and you'd need over 1tb of space for that many streams if played beginning to end. I doubt that would happen but it's possible. TVs shouldn't usually need a transcode, but they may not support PGS subs and that has to be burned in with a transcode. That's the main incompatibility that I've seen with TVs.

This is a graph of the space usage of my transcoding-temp folder over the past couple weeks or so, since it's a ZFS dataset I can look historically at how much space has been used. The continuous space usage for a few days might have been a folder that didn't get cleaned up properly. Aside from one large spike, which I could probably figure out if I look through playback reporting for what coincides with that date, most of the time only a few GB are needed

Thank you for the info and for your time, i hope that not everyone will do so but who knows.

I´ll setup a new "disk" for the CT so i can have around 120GiB only for transcoding and i´ll be checking how is it working.
Btw, nice monitoring setup, are you just using zabbix or do you have something extra?

[Lessaj 467]

Just Zabbix.

[softworkz 5066]

With regards to transcoding-temp, there is generally no need to be worried about "wearing out" SSDs. The way how Emby is using transcoding-temp is actually a pattern that naturally prevents wearing out SSDs:

• It requires relatively large amounts of storage
• If that's not available, transcoding fails
• Deletion is happening only after playback stop

Wearing out would only be a risk if the disk would be filled close to capacity and the same small remaining space would be frequently written and deleted - so quickly that the SSD is unable to relocate fast enough for equalizing out across memory cells.

 

But as Emby doesn't act like that, there's really nothing to worry about in this regard.

In general, the best measure you can take to avoid wearing out those disks is to:

• Make sure to keep a reasonable amount of free space available at any time (e.g. 5%)
• Have a good mix of static (rarely changing) and volatile content (frequently changing)

For example: A disk with 10% free space, 70% static and 20% volatile content will have 5 times the lifespan of a disk which is solely used for volatile data.
(his is because SSDs are internally relocating memory to equalize out read-write counts)

So whenever you are concerned about worn-out SSDs, keep that in mind, this is the most powerful leverage you have at hand for controlling this.

 

[Happy2Play 9780]

@softworkzis there a reason Emby is not doing only the fly segment deletions do to disc space?  Or is Emby just not aware or disc size here? 

As I know there are topics showing segment deletion do to disc space.

[softworkz 5066]

There have been posts about transcoding-temp not being cleaned up after playback had ended. Those were legitimate and I think they hae been addressed already.

What I don't consider reasonable is asking to minimize disk usage while transcoding is in-progress. Following considerations:

Emby is all about managing, storing and accessing one's own media library. Those libraries typically comprise dozens, hundreds or even thousands of video items. Even a full transcode of a video typically requires only a fraction of a single video item. So how can it be that somebody has disk space available for dozens or hundreds of videos but not half the size of a single video available for transcoding-temp? Something like 5 GB is just nothing these days and even when you have all your storage somewhere else on the network or running on special devices like Nvidia Shield or RPI, you can always plug in some USB storage for transcoding-temp and you're done.
None of the (few) topics where this has come up in the past was about problems encountered by average users (except those about Emby not deleting segments from finished playback), they were all rather special  and/or driven by special (and often wrong) ideas.

What stands against just-in-time deletion on the other side are substantial benefits and technical issues:

• Transcoding requires a significant amount of system resources and energy, while the storage of the results does not
  These results are very specific to the playback session, so there's no point in keeping the results beyyond the lifetime of a session, but while a session is active, there's always the chance that a segment is needed for playback more than once, so when the (expensive) work has been done already, there's no point in throwing it away, taking the risk of having to do the same work once or multiple times again
• Skipping Delay
  When a segment is not available in the client cache, it requests it from the server, if the server has it, it can serve it to the client an the client can continue playback almost instantly at the desired position.
  But if the server doesn't have it, it needs to start transcoding once again. The startup alone can take from a half to several seconds (e.g. if hwa fails and it must fall back to sw transcoding). Then it needs to transcode the segment first, until it can be delivered to the client. If trranscoding is slow, it adds 2-3s plus download time - makes 4s in the worst case. But most clients wait until they have at least two segments. In total, it can easily take 10s when skipping to a position for which the server doesn't have any segments, while it would be just 2s in this example for downloading existing segments.
• Stream/Segment Consistency
  Currently, generated segments aren't forward-compatible. This means that when the server has segments 100-500 and the user is skipping back to 90, all future segments (100-500) need to be abandoneded, because the newly generated segment 99 might not semlessly match with the pre-existing segment 100 (video and audio glitches or playback hanging in the worst case)
  Frequently deleting old segments would make it even more unlikely that existing segments can be re-used for playback when a user is skipping around
• Emby didn't track generated segments
  (in the past), so it wasn't possible to safely determine which segments to delete
• TVnext
  Will require even larger amounts of storage for transcoding-temp, so there's not much point to invest in segment deletion right now. But it also introduces new mechanisms for segment tracking and segment consistency (see above) which might allow to do smooth bandwidth switching during playback (later on) and also some management of segment deletion. Yet, I'm not a fan of keeping just a small window (like keeping just the past 10 segments), but when running live TV non-stop over hours with transcoding, there needs to be a cleanup at some point of course.

 

Edited May 14, 2024 by softworkz

[Q-Droid 989]

Something's been bugging me about this, from the OP's ffmpeg log.

Considering the resulting stream bitrate:

Quote

23:19:23.042 Output #0, segment, to '/var/lib/emby/transcoding-temp/322800/322800_%d.ts':
23:19:23.043   Metadata:
23:19:23.044     encoder         : Lavf59.27.100
23:19:23.045   Stream #0:0: Video: h264 (Main), cuda(tv, bt709, progressive), 1920x1080 [SAR 1:1 DAR 16:9], q=2-31, 11691 kb/s, 23.98 fps, 90k tbn
 

And first segment duration:

Quote

SegmentComplete=video:0 Index=145 Start=0.000000 End=435.393278 Duration=435.393278 offset_pts=0 start_pts=0 Frames=8 filename=322800_145.ts
Error writing trailer of /var/lib/emby/transcoding-temp/322800/322800_%d.ts: No space left on device
 

Wouldn't this single first segment exceed the space originally allocated for the RAM drive? If this is correct that would be a very large segment. Is it a bug or a logging error?

 

[TMCsw 248]

22 hours ago, Happy2Play said:

Sure as devs have already recommended against it.  

You obviously didn't read the rest of my reply, before posting that. (really no offence meant)(yes I gave a knee-jerk response)

Looks to me like I was on the right path here...

[softworkz 5066]

This is just a logging issue. You can always see this for the first segment when not starting from 0.

[Gabi92 0]

Thank you all for the responses, so far i´ve been testing and now it´s working fine, i´ve added a disk with 110 GiB dedicated for transcode.

In the other hand, i´ve also added more ram 4GiB in total now. I´ve been checking and with the bigger movie when is being directly playing, ram gets almost full then it switch to transcode and ram get back to usual usage. Is this ok?, i mean, direct play should be always the best option but it´s using a lot of ram.

Regards

[Luke 42077]

11 hours ago, Gabi92 said:

Thank you all for the responses, so far i´ve been testing and now it´s working fine, i´ve added a disk with 110 GiB dedicated for transcode.

In the other hand, i´ve also added more ram 4GiB in total now. I´ve been checking and with the bigger movie when is being directly playing, ram gets almost full then it switch to transcode and ram get back to usual usage. Is this ok?, i mean, direct play should be always the best option but it´s using a lot of ram.

Regards

Is there a ramdisk involved?

[isamudaison 12]

If you're dead-set on using a RAM disk, I've had success with the following:

- Allocate 40GB

- A cron job that runs every hour (here is my specific cron job):

0 * * * * bash /home/isamudaison/clean-emby-transcode >> /home/isamudaison/log/clean-emby.log 2>&1

This executes the clean-emby-transcode script and outputs to a log file specified

Here is the `clean-emby-transcode` script:

#!/bin/bash
docker exec emby sh -c 'find /transcode/transcoding-temp/ -mmin +360 -delete -print'

This deletes any file modified over 360 min in the past (assuming a docker container that uses the specified path).

This falls apart if someone leaves a live tv stream transcoding forever (event shows like super bowl/oscars/whatever), I suspect 80GB would be an ideal size for a RAMdisk.

[sa2000 674]

On 16/05/2024 at 08:21, Gabi92 said:

i´m facing some issues related to the transcode but not related to the space :).

Please close this off as resolved and start a new topic with your new problem

 

Thanks