Massive transcoding temp files

[andersoc27 2]

Hi Guys

I seem to have a massive transcoding temp file build up - is this normal? Screenshot below.

I presume this is what causes the "rotate log file" error in the dashboard.  Aside from losing space on my drives is there any other impact from this?

I'm going to delete now and win back ~21 Gigs of space but that seems bit over the top in the first place right? something I should have set up differently 

 

[ebr 14446]

Is someone streaming live TV?

[andersoc27 2]

You mean in general or at the time i noticed the file size? Streaming and recording live tv is my primary use of emby.

[andersoc27 2]

Incidentally, not sure if connected but playback seems faster since i deleted those files... could be in my head.

Edited May 9, 2019 by andersoc27

[ebr 14446]

At the time.  A 10 hour Live TV session could create that size file.

[gtmnyc 12]

what .. 10 hrs of streaming will give a 21 gig of transcoding?

[Spaceboy 2434]

what .. 10 hrs of streaming will give a 21 gig of transcoding?

easily. 4Gb an hour is not unreasonable based on my usage

[denz 385]

Can a limit be set how many minutes you can go back

[ebr 14446]

Can a limit be set how many minutes you can go back

 

Not at this time but managing this is on our list of things we'd like to do.

[andersoc27 2]

Thanks for all the feedback here ebr (and all in the community for jumping into the discussion )

Maybe a shorter term opportunity that would in general help with identifying these temp files and deleting would be to hyperlink any of the error paths in the dashboard......unless it's just me they are not copyable (is that a word?) so it makes it little bit of a pain to navigate and find and delete the big temp files.........if they could hyper linked from dashboard (presuming deleting from the dashboard would mean some bigger investment in dev work) would just put deleting it one click away........

[hshah 19]

I was showing a friend yesterday how I could stream on my phone.  After the 5 second demo, I just went back and closed the Android Emby app.  Noticed today that there is a ~30GB file in my transcoding temp folder, which spans 12 hours and 37 minutes.

 

Points to note:

- I am the only Emby user who has access to Live TV as I am just testing things at the moment

- I didn't pause the stream or try to record anything

- the stream was loaded in VLC for Android (via the Emby app) rather than being played within Emby

- both VLC and Emby were definitely closed after the quick demo (proper closed rather than left in the background)

- my phone battery had died a few hours later, so that would have killed anything in case I accidentally left it paused or something

 

I have just gone and repeated the entire process, and the ~30GB file is still there, and no other new files are present.  Don't fully understand why there is no file this time round, but there was after the same steps last night.

 

 

/Edit: Sorry, just noticed that EmbyServer.exe has the file currently locked and is actually appending to it following my test.  My server dashboard shows no active streams or anything actually going on, but since the file is still growing, there clearly is something happening.

 

/Edit2: I have gone and removed any entries for my Android from the Devices page (via the Server Dashboard) and the file is still growing and locked to EmbyServer.exe

 

/Edit3: File ended up reaching around 32GB and was automatically deleted when I restarted the Emby Server.

Edited May 10, 2019 by hshah

[andersoc27 2]

@@ebr - is there a cap/limit on the tmp file size and auto delete after emby server restarts? 

[andersoc27 2]

OK - it needs a server restart and then deletes itself automatically. Is this behavior working as designed? is there anyway to schedule server restarts from the dashboard as this would mean deleting the file if we could schedule say middle of the night ever 24hrs......

[hshah 19]

OK - it needs a server restart and then deletes itself automatically. Is this behavior working as designed? is there anyway to schedule server restarts from the dashboard as this would mean deleting the file if we could schedule say middle of the night ever 24hrs......

 

The file being deleted when the server is related may be as intended, but in my case the file even being there in the first place is most definitely not as intended.  There were no signs of any active streams, my phone had been restarted multiple times, I have removed all occurrences of my phone from the Devices screen etc.

 

@@ebr, is the Dashboard designed to show all activity?  I am trying to understand why it shows no active streams / recordings, but is silently dumping to disk and creating huge files.  IPTV providers tend to have connection limits as well, so one would have been taken up by this and could have other significant ramifications.

[ebr 14446]

If you are positive there were no recordings happening at the time then, no, that would not be intended.  The server tries to watch for any devices that "go away" unexpectedly and terminate them but that is not always that easy to determine.  I imagine that the use of the external player may complicate this as well.

 

If you can reproduce the scenario and provide the server log, we can take a look.

 

Thanks.

[Dan_Austin 62]

Fixing hung and invisible streaming sessions is a worthwhile endeavor, but perhaps just

implementing a rotating transcoding buffer (configurable or based on EGP program

start/stop times) would be more productive.

 

I had transcoding-temp fill up an 80GB volume quite a few times.  I frequently just leave

TV on for background noise while I work, and often without changing channels.  I got tired

of restarting Emby, or changing channels just to manage the temp files, so I bought a

pair of 512GB SSD drives to upgrade the volume, and getting religious about exiting from

Live TV on my Roku every single time.

 

I've still had two or three instances of a completely full volume, 450+GB of temp files.  If

Emby would rotate transcoding temp files every 30 minutes, or when a program ends, per

the EPG, then the smaller volume would have been enough.

[andersoc27 2]

[hshah 19]

[jad3675 26]

I've seen a similar issue with the Samsung TV app - I had a user watching a non-hdhomerun Live TV stream @10pm. Emby kept writing to the .ts file well until the morning (the dashboard showed they disconnected at 1:04am). I have a cron job that cleans up the transcoding-temp folder every 15 minutes - it removes TS files older than 10 hours and anything larger than 20GB. It grabbed the phantom .ts file at 6 this morning based on size.

 

I know, useless without logs, but don't have access to them at the moment.

 

John

[ebr 14446]

Does this still happen if, instead of "force closing" both VLC and Emby, you exit them normally?

[hshah 19]

Does this still happen if, instead of "force closing" both VLC and Emby, you exit them normally?

 

VLC doesn't appear to have an option to "exit" when playing the stream.  Pressing the "back" returns to Emby, where I can see the stream active but there is no option to stop it via the Android app like there is when the Dashboard is accessed via a browser.  Pressing the "back" key a few more times closes the Emby app, but I can still see the stream active on the Dashboard (via Chrome on my PC).  

 

I think one of the key points to note now is the next action, which happens to be the same as step number 10) from this morning's tests.  I have pressed the "stop" button, and can now see no evidence of activity on the Dashboard.  So at this point anything and everything to do my with my phone becomes irrelevant, because I have pressed "stop" on the Dashboard (via a browser on the same machine as the Emby Server) and naturally expect it to completely stop.

 

Thoughts:

• Is the Android app somehow not recognising the state of the stream to that device? I never have the option to stop the stream via the app when the stream was played on that device, but if I start a stream on my PC, the usual control buttons are available in the Android app.
• Does not occur when the streaming takes place within the Emby app (ie External Video Players is disabled)
• The code behind the "stop" button doesn't seem to be as thorough as it should be because it should have done as it says on the tin.
• Should the Dashboard show any/all activity occurring on the Emby Server?  It is obviously recording in the background but not providing any visibility of said action. 

(Note: I do have some Custom CSS  so incase it was inadvertently hiding something important, I have already tried viewing the Dashboard without it in place)

Edited May 14, 2019 by hshah

[Dan_Austin 62]

I get that external clients/players present a challenge, but I only use the native Emby app on

Roku, and once in a very rare while Emby Theater.  After figuring out the lack of buffer management,

I always cleanly exit out of the channel/guide/app.  In that order of course.

 

I still have had three instances of 100+ hour transcoding temp files.  Of course I have recordings during

those periods.  But none were 100+ hours long.  I can also guarantee that I did not go 100+ hours without

turning TV on and changing channels at least once.

 

I know buffer management has been identified as a future improvement.  If it is not a priority improvement,

how about making the official apps provide state back to the server?  A simple I'm still here streaming on

session-id <whatever>.  The server can then decide to drop sessions if the client misses a reasonable duration

or number of status updates.  And since such a feature will likely infuriate some percentage of the user base,

make it a server side option from day one.

[ebr 14446]

I know buffer management has been identified as a future improvement.  If it is not a priority improvement,

how about making the official apps provide state back to the server? 

 

We already do this.  The problem comes in when the app is exited in such a way that it isn't possible for that state to be transmitted.  One example of this is pressing the "Home" button on the Roku remote while playing live TV.  When you do that, the Roku system simply kills our app with no notification or warning.  A crash of the app can have the same effect but is less likely.  My guess is your 100 hour examples were from someone pressing home on the Roku.

[hshah 19]

We already do this.  The problem comes in when the app is exited in such a way that it isn't possible for that state to be transmitted.  One example of this is pressing the "Home" button on the Roku remote while playing live TV.  When you do that, the Roku system simply kills our app with no notification or warning.  A crash of the app can have the same effect but is less likely.  My guess is your 100 hour examples were from someone pressing home on the Roku.

 

I am assuming that since current activities (along with their states) are displayed on the dashboard, regular communications must be taking place between the server and client(s)?

 

Interestingly, I decided to see if I could replicate the behaviour with another "client" and the experience was very different:

 

1) Watch a stream via Internet Explorer, on my laptop, whilst not on the LAN

2) Could see it on the Dashboard and files were being created in the transcoding directory.

 - this time it was creating multiple smaller files rather than just maintaining the one

 - the difference being that it was transcoding as opposed to the direct play for my phone

3) I killed the IE process from Task Manager, mimicking a "force close" on the Android

4) The activity disappeared from the Dashboard

5) The temporary files were deleted shortly after

 

Now I don't know whether the difference is simply down to the transcoding vs direct or if there is something else going on.  I will try and play a transcoded stream on my phone later on tonight and feedback.

 

The point around how the server was aware of the stream being terminated still stands.  With respect to the experience I described when streaming on my phone, I had actually pressed the "stop" button on the Dashboard, on the machine the server was running.  There must be a bug in the code somewhere because the message to terminate cannot have gotten lost on the same machine

[Dan_Austin 62]

 My guess is your 100 hour examples were from someone pressing home on the Roku.

 

Sorry, but not only unlikely, impossible.  Single guy living alone.  No one has access to

emby outside of my home, and my remotes do not have a 'Home' button, or specifically

the one I use.

 

I know it will get fixed eventually, so I have accepted that the DVR may miss programs

due to transcoding-temp filling up.  And that I may occasionally not be able to logon, which

is another signal of a rogue file filling the filesystem that Emby uses.  It's only TV after all

and not the end of the world. 

 

Oh, and even if I did manage to dig out the Roku remote and ignore everything I know

about shutting down cleanly to avoid this issue, a rotating buffer would have prevented

unpleasant behavior...