Live TV Pause Buffer fills HD and crashes server

[dr_faulkner 9]

The issue is not where the pause buffer location is configured - its when the pause buffer is used and flushed. It appears entirely unnecessary to have indefinite pause buffering - particularly when it will eventually fill your C-Drive and crash your server.

 

By default the WinTV app does not use the pause buffer (correction, it does and also has a max buffer setting defaulting to 30 minutes)- its a UI user activated feature in the app. Once the pause function is used, WinTV appears to accumulate pause buffer files until the app is closed. WinTV also purges the pause buffer files when the app is restarted.

 

However, emby doesn't use the WinTv UI app, and therefore apparently bypasses the pause buffer purging the app performs. Further, Emby always turns on pause buffer (why? Is it using the buffer for streaming? if not, why can't Emby do its own buffering?), but has no replacement for the WinTv's pause buffering purge - and therein lies the problem.

 

I am going to try disabling the pause buffer in WinTV (setting lenth to 0 mins)

Edited February 2, 2019 by dr_faulkner

[dr_faulkner 9]

PS - Don't know if this is actually doable, but the pause buffer could be purged by emby server when users stop/start watching tv,

[PenkethBoy 2068]

Lot of assumptions in you post and some are from my testing incorrect

 

setting the buffer to 0 will likely cause problems

 

in wintv move the pause buffer to a larger disk and your problems will go away

[dr_faulkner 9]

After some further observations I think I know what's going on here.

 

Setting the pause buffer length only changes the buffer file sizes. It has nothing to do with when WinTv purges those buffer files. The buffer files continually grow whilst watching a buffered channel.

 

Gracefully exiting a tv channel in Emby Theatre does cause the buffer for that channel session to be purged - but closing the Emby Theatre app does not - and Emby Server continues to happily play that TV channel forever (and the pause buffer files for that channel session grow in number until they consume all disk space).

 

Emby Server knows when Emby Theatre is closed - so the obvious solution appears to be for Emby Server to switch off any channel that that instance of Emby Theatre was watching rather than play it forever.

 

 

As an aside, Emby requires the pause buffer for some reason. Setting the buffer length to zero stopped live tv from working in Emby. Also, I set the buffer to 1 minute and noticed a playback glitch in Emby Theatre each time a new buffer file was created (I had noticed this glitch occurring before on occasions). So its probably a good idea to set a large buffer size to mitigate this issue.

[dr_faulkner 9]

Yes, I know there's a workaround for this Emby problem - I've unfortunately had to rely on it for some time now - and this is an Emby problem as Emby Server continues to play a tv channel (and thereby grow the associated pause buffer) indefinitely if the client is closed whilst watching a tv channel). In lieu of any evidence to the contrary I believe this observation to be entirely correct.

 

This workaround required me to install another local drive to avoid unnecessary writing to my main storage array - and still leaves the Emby server unnecessarily writing gigabytes of tv buffer files for channels no-one is watching.

 

And if you happen to be a hapless customer who is not aware of the problem or the workaround your server will simply crash sooner or later when it runs out of disk space - and that leaves a very sour taste in the customer's mouth.

 

If Emby Server simply turned off any channel being watched when the client shuts down then this issue would all but vanish. How difficult could that be to implement? Add in a file purge on service startup or scheduled task for good measure and the issue would be entirely eliminated and you won't be crashing your customers machines anymore.

 

The existence of a workaround hack is no excuse for letting this kind of known serious issue go on for years...

[FordGT90Concept 97]

Gracefully exiting a tv channel in Emby Theatre does cause the buffer for that channel session to be purged - but closing the Emby Theatre app does not - and Emby Server continues to happily play that TV channel forever (and the pause buffer files for that channel session grow in number until they consume all disk space).

 

Emby Server knows when Emby Theatre is closed - so the obvious solution appears to be for Emby Server to switch off any channel that that instance of Emby Theatre was watching rather than play it forever.

That sounds like a serious oversight that needs to be fixed. Matches the symptom too that made me necro this thread. Edited February 3, 2019 by FordGT90Concept

[dr_faulkner 9]

Couldn't agree more - its basically a time-delay bomb that should be fixed as a high priority IMHO. I am an IT Architect by profession and I'd expect to be in trouble for a design oversight that was serious enough to bring down the server(s) - but I'd expect to be fired if I knew of the issue and did nothing about it.

 

Please fix this

 

PS: Of course the other workaround is to always remember to exit live tv before closing the client - but if you forget to do so even once - BOOM!

[Luke 42077]

We'll get to this, thanks guys.

[dr_faulkner 9]

Thanks Luke. Squeaky wheel gets the oil hopefully :-)

 

The Live TV has been the main problematic part of Emby for me. Even with the workaround described above, live tv can be hit and miss sometimes (it will just refuse to work via Emby but WinTV works fine). If these were ironed out I'd be extremely happy with Embys reliability. The only other issues I have concern suggested cosmetic UI interaction improvements.

 

I'm wondering if the problems are related given it appears Emby is using the pause buffer files for streaming?

[FordGT90Concept 97]

The Live TV has been the main problematic part of Emby for me. Even with the workaround described above, live tv can be hit and miss sometimes (it will just refuse to work via Emby but WinTV works fine). If these were ironed out I'd be extremely happy with Embys reliability. The only other issues I have concern suggested cosmetic UI interaction improvements.

Ditto.

 

HDHR EXTENDs have problems crashing (application watchdog).

HDHR (##.#) and QuadHD (1###) don't share channel numbers.

Can't assign priorities for tuner usage (QuadHD should be preferred for live, HDHR for recordings).

HDHRs have caused Emby to fill C: twice.

Live TV audio desync on SHIELD TVs when using HDHR EXTEND heavy profile.

Xbox One S keeps stopping streams sourced from tuners (seems like it can't handle any hiccup in the data whatsoever). Xbox One S used to be the best but now it is the worst for recording playback/live TV.

QuadHD has filled C: too causing Emby Server to crash.

Lots of recordings are only half there (30 minutes instead of 60) even when there's 10 tuners to choose from/fall back on.

Lots of recordings are missed probably because of problems mentioned above.

 

Doesn't matter what I try with live TV/recordings, there's always problems. Everything else works well enough. Live TV is the most complex (and expensive...probably blew $800 on tuner hardware) thing Emby does and it needs the most work.

Edited February 3, 2019 by FordGT90Concept

[Spaceboy 2573]

Just to say this isn’t universal. I use emby predominantly (>75%) for live tv and it’s great

[FordGT90Concept 97]

40+ miles from broadcast antennas captured by two rooftop, 6' directional antennas 20+ feet off of ground level

ChannelPlus 3 input, 8 output amplified distributor

3 x HDHomeRun EXTEND

1 x WinTV QuadHD in server

all 4 connected to an Ubiquiti 24-port gigabit managed switch

2 x SHIELD TV 2017 edition

1 x Xbox One S

1 x RX 590 powered computer

1 x Intel Atom powered miniPC

all connected via CAT6 shielded cable

 

I think that first line is the crux of the problem. Signal quality is not perfect all of the time and it can never be. 20 MPH winds are typical, gusts in excess of 60 mph happens. Nevermind the occasional blizzard and torrential downpours. There are inevitable artifacts in the signal and the Xbox One S especially seems to struggle with it.

 

Perhaps you live closer to the broadcast antennas so that translates to fewer problems for you. Doesn't mean there isn't problems and major areas to improve on though.

Edited February 4, 2019 by FordGT90Concept

[dr_faulkner 9]

I am glad LiveTV is reliable for some, but this is little comfort to those of us experiencing persistent issues.

 

The other main problem I experience is that Emby seemingly randomly refuses to play particular WinTv channels (typically 1080 HD channels) - when the WinTV app itself is playing them flawlessly! - and often this leads to a complete LiveTV meltdown where no channels will play and Emby Server and client must be restarted.

 

This is a frequent occurrence for me - and an extremely annoying one. LiveTV MUST work reliably for Emby to be a proper universal media aggregator

 

There is obviously some problem occurring with Emby picking up the stream. Based on the observations I made regarding the infinite pause buffering issue, it appears that Emby is streaming from the pause buffer (perhaps from the files themselves?)

 

The code doing this is clearly neither resilient nor robust - it simply disappears up it own wazoo if something goes wrong (infinite spinning circle that won't go away). I would strongly suggest creating some worst-case test cases so when something goes wrong with reading the LiveTv stream at least Emby is robust enough to recover gracefully (i.e. no restarts required)

 

PS: I have been religiously ensuring that I exit from LiveTV before closing down Emby - this does indeed appear to stop Emby growing the channel pause buffer indefinitely - and I haven't experienced a LiveTV hang yet either (fingers crossed).

[mastrmind11 722]

Doesn't runaway playback prevention work on LiveTV?  It used to IIRC.  It certainly works with other media.

[dr_faulkner 9]

Apparently not...

[Spaceboy 2573]

No I don’t think so

[FordGT90Concept 97]

It happened again on 4.1.0.17.  Transcoding-temp had a 167 GB file in addition to three others that filled C: to the point that I couldn't even copy a 700 KB file to the machine's desktop.

 

I have no idea when it happened so I'm attaching all of the logs that were created today (it had to have been today).

 

Because the Hauppauge source isn't working in this beta, like all of the previous times, this had to be triggered by HDHomeRun.

 

As of right now, all HDHomeRun tuners are not in use.

 

 

Edit: I realize that this thread is dedicated to Hauppauge but I can't help but think that the cause of the HDHomeRun issue is similar.  Server keeps streaming a feed that isn't there.

logs.zip

Edited March 30, 2019 by FordGT90Concept

[dr_faulkner 9]

I have religiously exited Live TV before closing down Emby and the buffer issue has not reoccurred for me - so it looks almost certain to be the cause

[murky024 3]

I just ran into this for the first time... Honestly, I am surprised I didn't get bit earlier by it. My pause buffer storage had over 800GB of unused videos. Restarting the Emby server did nothing... I had to go in and change the location for pause buffering and that cleared up the files as over 82 GB of them were locked and would not delete.

 

I really hope you can come up with a more graceful solution than having to exit the live TV as the Roku app always has issues and will exit live TV less gracefully than the Android one. Honestly, the Roku app has more problems than any other app I have used. I am not sure if its just my TV, or if other people have that same experience as well.

[FordGT90Concept 97]

I've had it happen to me again about a week ago. I was the only one home at the time so I think it had to have been caused, somehow, through the Edge browser accessing Emby Server.

[Luke 42077]

We're planning on adding limits to this based on disk space in the near future. Thanks for the feedback.

[kevinf63 3]

@@Luke I can confirm this is a consistent problem for me as well. Ideally, a size limit for the pause and transcoding buffer if possible. Its regularly trashing my NVMe SSD at 99% usage, then a brisk Emby restart allows it to clear it down.

 

• Emby Server - v4.3.1.0 Docker
• Samsung UE40KU6470 TV - Emby Theater v1.0.69 client
• Samsung UE40MU6400 TV - Emby Theater v1.0.62 client
• Firefox Browser - v71.0 through Web client
• HD HomeRun Quattro - DVB-T2 quad tuner.
• Telestar Digibit R1 - 4 DVB-S/S2 SAT>IP tuner box that outputs M3U playlist.

Edited January 1, 2020 by kevinf63

[dr_faulkner 9]

As I previously reported (and can confirm), the root cause of this is shutting the client down whilst in live tv, in which case Emby server happily keeps recording/buffering the tv channel even tho no-one is watching it.

 

If you exit live tv emby server stops buffering. If emby also stopped buffering when a client disconnects this would fundamentally fix the problem

 

Implementing a buffer size limit will still not stop emby server senselessly using up your SSD write cycles by  buffering tv no-one is watching

[kevinf63 3]

As I previously reported (and can confirm), the root cause of this is shutting the client down whilst in live tv, in which case Emby server happily keeps recording/buffering the tv channel even tho no-one is watching it.

 

If you exit live tv emby server stops buffering. If emby also stopped buffering when a client disconnects this would fundamentally fix the problem

 

Implementing a buffer size limit will still not stop emby server senselessly using up your SSD write cycles by  buffering tv no-one is watching

So if that is the case, realistically the way to solve that on the client side is that if the TV is signaled to turn off through the remote, a yes/no dialog box would interrupt/halt the shutdown with a "are you sure wish to close this Live TV connection?". Alternatively you'd put some sort of try/catch/finally clause around open (Live TV) connections that when the Emby app is force closed or TV is shutdown it would clean this up before the full shutdown. Both of these are entirely dependent on whether Samsung's API/framework for developing the apps allows for this level of interruption to the TV OS.

 

Surely Emby Server polls the clients with some kind "are you alive" messaging/handshake?

 

Also apologies for not reading through the full thread for your answer above @@dr_faulkner!

Edited January 1, 2020 by kevinf63

[Happy2Play 9780]

 

Don't all apps have the "Are You Still Watching?" setting?

 

Enable 'Are You Still Watching?' prompt

Prevent playback from continuing indefinitely by periodically prompting for user input.