emby transcoding folder fills disk from time to time

[dmshimself 1]

Every so often emby fails on my debian 11 system because the folder where all the transcoding files are kept fills up the disk.  In the past I've just deleted the files and restart emby, but I'd like to try and get this fixed if possible.  I've added a cron job to delete all the files as I don't actually use transcoding because I used Apple TV as the only client which doesn't need them.  The  scheduled jobs is turned off and so I'm not sure why transcodings are being done at all.

I am running the latest version of debian and emby and I've added an extra disk just to hold transcodings.

I've included the log files from the time the last error was reported and it seems to be due to one file being recorded.

config.zip

[dmshimself 1]

It's just happened again ...

[Luke 37116]

Hi @dmshimself I don't see anything in your zip file that looks like emby server log files. Can you please attach the emby server log from when this happened?

[dmshimself 1]

Hi Luke - In the zip file you will see two logs, one called embyserver-63805363199.txt and the other called embyserver-63805432689.txt both of which I downloaded from the logs area of the web app.  The other files were 'useful' looking log files from the time of the error.  I've just unzipped that file myself and a screenshot of what I can see is enclosed.  Is this also what you can see in the same zip?

 

 

[Happy2Play 8296]

@dmshimself Did you post the wrong zip as I am not seeing anything Emby related in that CONFIG.zip.

[dmshimself 1]

My thick fingers .... I'll upload the right one this time!  (Hopefully)

emby.zip

[Happy2Play 8296]

@Lukeis this the livetv buffer?

	Line 1208: 2022-11-29 13:27:01.485 Info SharedHttpPipelineSource: Beginning SharedHttpPipelineSource stream to /var/lib/emby/transcoding-temp/9392f85fda6347499fcf493d66f83945.ts
	Line 1573: 2022-11-29 15:33:07.431 Info SharedHttpPipelineSource: Beginning SharedHttpPipelineSource stream to /var/lib/emby/transcoding-temp/78e19ceda1aa4792ac03031283a26a02.ts
	Line 4762: 2022-11-29 19:02:28.077 Info SharedHttpPipelineSource: Beginning SharedHttpPipelineSource stream to /var/lib/emby/transcoding-temp/7f5ea72132414c498b713d07b2a58e31.ts
	Line 6728: 2022-11-29 19:52:00.325 Info SharedHttpPipelineSource: Beginning SharedHttpPipelineSource stream to /var/lib/emby/transcoding-temp/4dd61d6370b1454fb4dd59b7f9fa5621.ts

Only 2 streams are deleted via log I looked at.

    Line 1317: 2022-11-29 14:50:00.088 Info SharedHttpPipelineSource: Deleting temp files /var/lib/emby/transcoding-temp/9392f85fda6347499fcf493d66f83945.ts
    Line 6690: 2022-11-29 19:51:57.772 Info SharedHttpPipelineSource: Deleting temp files /var/lib/emby/transcoding-temp/7f5ea72132414c498b713d07b2a58e31.ts

Looks like 78e19ceda1aa4792ac03031283a26a02 has been streaming for 4 hours.

[dmshimself 1]

just as an observation, its perfectly possible 1 of the 2 people in our household left something recording or live playing for ages by accident, but the only device we use to view TV is an apple TV and in the past it has always used direct play (and in fact is doing so now as we speak).  I used to occasionally played a recorded program on my browser and so I guess transcoding would have kicked in for that, but in the end, to try and cover up this problem, I turned off all transcoding jobs on the server.  Maybe there is one still lurking!

[dmshimself 1]

So is there anyway to turn off transcoding completely to try and cover this problem up?  I'd rather get an error at the time of something trying to play the video than getting the disk filling up with transcodes that I don't want or need.  

[Happy2Play 8296]

24 minutes ago, dmshimself said:

So is there anyway to turn off transcoding completely to try and cover this problem up?  I'd rather get an error at the time of something trying to play the video than getting the disk filling up with transcodes that I don't want or need.  

That is exactly what the per user options do.  If disabled and transcoding is required item fails to play.

At the same time Live TV usually require container changes and use transcode-temp folder.  So you could eliminate playback means completely.

[dmshimself 1]

That is such a useful answer and I never would have thought of looking  in user options for that.  Time to create a new user!  Many thanks

[Luke 37116]

Keep in mind the server will clean up the transcoding temp files when the user has finished playing.

[dmshimself 1]

Well that is the (I think) the root cause of the problem.  'Something' causes the transcoding folder to fill up no matter how big I make it and that causes emby to stop responding.  That seems to mean the folder never gets cleaned up.  I did put in a cron job to delete these every day, but again that doesn't help as the disk fills up before the cron job kicks in.  I'm hoping that running with a user that does allow any form of transcoding at all will help...

I started with setting up a manual transcoding folder with a drive that was 20G, then 50G and so on..

[Luke 37116]

Hi, are you still running into this? Have you updated to Emby Server 4.7.11?

[dmshimself 1]

You must be reading my mind.  Yes it happened last night in the very early morning (my time), but I have now made the disk big enough so that my monitoring gives me 3-4 hours notice before the disk completely fills.  When I looked this morning the upgrade to .11 had already happened.  I rebooted and the disk space cleared.  At that time, no-one was watching anything.  I did notice the task which updates thumbnails was the only likely regular task which might have been involved and I've now restricted that job to a maximum of 1 hour running.

Given that I now have a bit of time when this happens, is there anything you would like me to collect other than the server logs?

The user running the Apple TV app has no ability to run any form of transcoding and so my thinking goes to some regular system level jobs that might run from time to time.

My own suspicion is that ffmpeg is getting a .TS file recording from somewhere our TV recordings which it cannot cope with.

 

[dmshimself 1]

It happened again this morning and I've grabbed all the log files plus a couple of screen shots to show a recording that started at much the same time as the disk started to fill and a list of the files themselves.  I then used the web front end to restart the server and once that was done, all the files disappeared from the tmp transcoding area.  Perhaps for some types of recordings a transcoding is done during the live recording?

Anyway log files enclosed....

emby1.zip

[UCM_1 10]

This, or something similar to it, has been a problem for literally years -- I've posted about it myself in the past.

Something goes haywire after a while and will eventually fill an entire drive after some time.

The problem did seem to go away for a while, but it certainly seems to be back with a vengeance with the lastest version or two.

I've given up looking for the cause, and just reset it when I need to.....

[dmshimself 1]

cheers - I can see that when a live recording is running, emby uses ffprobe for something and my feeling is that the processing of the video data sometimes goes wrong, the temp data isn't removed and the next recoding can start the same process.  It would be handy if this extra manipulation of the data could be turned off.  My data is coming from a vbox satellite receiver and looks to be a standard .TS file.  I do understand that some over the air data is not as well structured as it could be and gets the occasional glitch.  I also understand that I might be wrong!

[Luke 37116]

Are you still having an issue with this?

[UCM_1 10]

On 1/18/2023 at 7:19 PM, Luke said:

Are you still having an issue with this?

Yes, this problem still happens.  In fact, it just happened an hour ago.

Inside the temp transcoding directory there was a .ts file that was about 112GB in size, and apparently was created/started about 21 hours earlier.

Someone may have started watching TV around that creation time, but certainly did not watch it continuously since then (and the control panel never showed anyone watching today either).  Plus, we do have the "are you watching" prompt system enabled, so it should have quit on its own at some point even if someone had left something running.

My best guess is that somehow Emby left a TV process running in the background?

 

On a side note.... Nobody needs 21 hours of rewind capability....  Maybe limit how much is allowed in a file?  (In the off-chance someone would want 21 hours of rewind ability, maybe make the amount configurable)

[UCM_1 10]

On 1/18/2023 at 7:19 PM, Luke said:

Are you still having an issue with this?

I had not been keeping a close eye on what was going on with the drive filling up every now and then, resolved to just have to reset the system every now and then...  But, since you are asking about it again, I figured I'd try and look closer.

At this exact moment, my instance of Emby is doing absolutely nothing, yet the HD space is about 10GB lower than it should be.

And even though Emby isn't doing anything, inside the transcoding-temp directory there is:

drwxr-xr-x.  2 emby emby      69632 Jan 26 09:43 14CA75
-rw-r--r--.  1 emby emby 5273325568 Jan 31 01:47 5095d09327154be08162eadf5e027c09.ts
-rw-r--r--.  1 emby emby 4011491328 Jan 28 18:10 5d471804a4ba4143a3958e08d52c023f.ts
drwxr-xr-x.  3 emby emby      69632 Jan 30 10:47 7C3206
-rw-r--r--.  1 emby emby          0 Jan 30 20:58 da985ac163ae44e8bb31321eecf9752a.ts
drwxr-xr-x.  2 emby emby      28672 Jan 30 15:59 ECC50A

 

And inside each of the three directory are hundreds of small .ts files and a .m3u8 file.   And one of those directories also contains a Fonts directory, although that is empty.

 

Past observed HD space behavior, that space will never be reclaimed until Emby is restarted.  And space will continue to either slowly disappear over time until it runs out.  OR that space will disappear fairly quickly, as it did in my previous post where one .ts file grows to fill almost all the available space by itself.

 

Edited January 31, 2023 by UCM_1

[Luke 37116]

6 hours ago, UCM_1 said:

I had not been keeping a close eye on what was going on with the drive filling up every now and then, resolved to just have to reset the system every now and then...  But, since you are asking about it again, I figured I'd try and look closer.

At this exact moment, my instance of Emby is doing absolutely nothing, yet the HD space is about 10GB lower than it should be.

And even though Emby isn't doing anything, inside the transcoding-temp directory there is:

drwxr-xr-x.  2 emby emby      69632 Jan 26 09:43 14CA75
-rw-r--r--.  1 emby emby 5273325568 Jan 31 01:47 5095d09327154be08162eadf5e027c09.ts
-rw-r--r--.  1 emby emby 4011491328 Jan 28 18:10 5d471804a4ba4143a3958e08d52c023f.ts
drwxr-xr-x.  3 emby emby      69632 Jan 30 10:47 7C3206
-rw-r--r--.  1 emby emby          0 Jan 30 20:58 da985ac163ae44e8bb31321eecf9752a.ts
drwxr-xr-x.  2 emby emby      28672 Jan 30 15:59 ECC50A

 

And inside each of the three directory are hundreds of small .ts files and a .m3u8 file.   And one of those directories also contains a Fonts directory, although that is empty.

 

Past observed HD space behavior, that space will never be reclaimed until Emby is restarted.  And space will continue to either slowly disappear over time until it runs out.  OR that space will disappear fairly quickly, as it did in my previous post where one .ts file grows to fill almost all the available space by itself.

 

Hi there, please attach the emby server log from when this happened. Thanks.

[DamnedUndies 17]

Reviving this thread as I believe I'm seeing a similar issue.

There was no user activity at the time that Emby reported a disk full error in the transcoding_temp directory.

Log attached.

After rebooting, >175GB of space was freed up by Emby.

embyserver-63836396832.txt

[DamnedUndies 17]

Digging a little deeper, I see the following error with the same stream in an earlier log file (attached):

2023-11-23 17:55:55.895 Error Server: Error processing request
	*** Error Report ***
	Version: 4.8.0.60
	Command line: /opt/emby-server/system/EmbyServer.dll -programdata /var/lib/emby -ffdetect /opt/emby-server/bin/ffdetect -ffmpeg /opt/emby-server/bin/ffmpeg -ffprobe /opt/emby-server/bin/ffprobe -restartexitcode 3 -updatepackage emby-server-deb_{version}_amd64.deb
	Operating system: Linux version 5.15.0-88-generic (buildd@lcy02-amd64-058) (gcc (Ubuntu 11.4.0-1ubuntu1~22.04) 11.4.0, GNU ld (GNU Binutils for Ubuntu) 2.38) #98-Ubuntu
	Framework: .NET 6.0.20
	OS/Process: x64/x64
	Runtime: opt/emby-server/system/System.Private.CoreLib.dll
	Processor count: 4
	Data path: /var/lib/emby
	Application path: /opt/emby-server/system
	Microsoft.AspNetCore.Server.Kestrel.Core.BadHttpRequestException: Microsoft.AspNetCore.Server.Kestrel.Core.BadHttpRequestException: Unexpected end of request content.
	   at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Http.Http1ContentLengthMessageBody.ReadAsyncInternal(CancellationToken cancellationToken)
	   at System.Runtime.CompilerServices.PoolingAsyncValueTaskMethodBuilder`1.StateMachineBox`1.System.Threading.Tasks.Sources.IValueTaskSource<TResult>.GetResult(Int16 token)
	   at System.IO.Pipelines.PipeReader.CopyToAsyncCore[TStream](TStream destination, Func`4 writeAsync, CancellationToken cancellationToken)
	   at ServiceStack.StreamExtensions.CopyToNewMemoryStreamAsync(Stream stream) in /home/runner/work/ServiceStack/ServiceStack/ServiceStack.Text/src/ServiceStack.Text/StreamExtensions.cs:line 687
	   at ServiceStack.Text.NetCoreMemory.DeserializeAsync(Stream stream, Type type, DeserializeStringSpanDelegate deserializer) in /home/runner/work/ServiceStack/ServiceStack/ServiceStack.Text/src/ServiceStack.Text/NetCoreMemory.cs:line 175
	   at Emby.Server.Implementations.Services.ServiceHandler.CreateRequest(HttpListenerHost host, IRequest httpReq, RestPath restPath)
	   at Emby.Server.Implementations.Services.ServiceHandler.ProcessRequestAsync(HttpListenerHost httpHost, IServerApplicationHost appHost, IRequest httpReq, IResponse httpRes, IStreamHelper streamHelper, RestPath restPath, String responseContentType, CancellationToken cancellationToken)
	   at Emby.Server.Implementations.HttpServer.HttpListenerHost.RequestHandler(IRequest httpReq, ReadOnlyMemory`1 urlString, ReadOnlyMemory`1 localPath, CancellationToken cancellationToken)
	Source: Microsoft.AspNetCore.Server.Kestrel.Core
	TargetSite: Void Throw(Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Http.RequestRejectionReason)

 

embyserver-63836380800.txt

[DamnedUndies 17]

Hi @Luke- please let me know if there's any additional info that may help root-cause this.  Thank you.