Remote playback buffering

[RaisonDetre72 5]

Ah, everyone's favorite "ghost in the machine" that seems to randomly pop up: playback pausing / buffering problems.  Recently I've been receiving reports from multiple remote users that playback is skipping, though I'm not sure (a) where to look in logs that can help me better understand what I need to troubleshoot and (b) what more I can do.  Hoping you guys can help me out (fingers crossed!).

- Hardware:  Ryzen 3700X (8 cores for Emby); 8GB RAM; metadata / cache / transcoding folders are located on SSDs

- Network:  ISP is 300/25Mbps; server has been running behind TorGuard since start of 2020, which vastly helped improve video quality and stuttering issues (fck u ISP throttling!)

User is running (I believe the latest, 4.0.4) Roku Emby.  I've got yesterday's (8/31) log, as well as today's (9/1) for reference.  Example files are 'I May Destroy You' episodes 7, 8, 9, 10, 11.  8/31: User watched Eps. 7 and 8 with no playback issues, then encounters stuttering at Ep. 9.  9/1: Retried playing 9, as well as 10, and 11, but all have pausing / buffering issues.  I've performed file integrity checks twice on these files, both times passed.  On my side, I'm monitoring hardware and network use, and am not noticing any abnormal CPU utilization, and besides the 4Mb network upload blips, the line is essentially idle.

Admins: just let me know whom I should DM the logs to, as well as what other info I can provide for troubleshooting.  Thanks in advance, all!

[Happy2Play 9781]

How many remote users are connected at the same time?  

Do you apply user or global "Internet streaming bitrate limit"?

What quality is the client set to? 

But is sounds like either the server is not transcoding fast enough or to high a bitrate for the connection.

 

[RaisonDetre72 5]

How many remote users are connected at the same time?

- Just the one user, no one else is on at the time.  I've had up to 3 users, including this one, watching simultaneously, some transcoding others direct playback, with no reported issues.

Do you apply user or global "Internet streaming bitrate limit"?

- I don't restrict by user; global internet playback quality is set to: Audio - Auto; Video - 4K-120Mbps.

What quality is the client set to? 

- Client is set to 1080p-4Mbps.  Probably unrelated though I've noticed with non-1080p files, like 720p, the player will fall back to Auto, which usually looks really blocky / low-res unless I have them manually turn up the quality, where it still plays without issue.

But is sounds like either the server is not transcoding fast enough or to high a bitrate for the connection.

- That's what I thought, but even for the files in question, they're not doing any transcoding, AFAIK.  On server side, it shows 'Direct Streaming'.  I'll DM you the logs.

[Happy2Play 9781]

Fyi there are several topic on Auto and it sending data at to low a bitrate.  I personally have the user set the client to max quality and control the quality by the stream limit I apply to the user.

[Carlo 4561]

Definitely try what Happy2Play mentioned above!

[RaisonDetre72 5]

First, my bad re: your question, @Happy2Play on universal bandwidth setting, I was looking at my own client's restriction (D'oh!).  The server's universal bandwidth setting is currently blank, so unlimited I'm guessing.

 

I'll troubleshoot with them tomorrow and get an update.  For testing:

- Confirm Quality settings on client-side when playing one of the problematic episodes (I'll use Ep. 9)

- Confirm Quality settings on client-side when playing one of the non-problematic episodes (I'll use Ep. 8 )

- Try playing back Ep. 9 with no Quality changes, confirm it still stutters.  If it doesn't, I'd assume its maybe something w/ ISP?  

- Change server-side user bandwidth limitation to 4Mbps and re-test Ep. 9.

 

Just to confirm: Did the logs indicate any latency issues?  I do see some HTTP 200 responses that are between 4000-7000ms+ but because they're so infrequent I'm not sure if that is impacting playback, or if its unrelated.  Also curious if the errors occurring have anything to do with it?  Example at 2020-09-01 16:57:21.452.

Edited September 2, 2020 by RaisonDetre72

[Luke 42079]

Did you find out the user's quality setting?

[rbjtech 5284]

"User watched Eps. 7 and 8 with no playback issues, then encounters stuttering at Ep. 9.  9/1: Retried playing 9, as well as 10, and 11, but all have pausing / buffering issues. "

Going back to basics - what's different about Episode 9 ? 

Have you tried remuxing those episodes ?

I had a similar issue the other day that was driving me nuts - played fine on a web browser, fine on the FireTV stick - but would stutter (unwatchable) on an Android tablet.    I figured it must be the video file as everything else played fine - so I remuxed it from mp4 into mkv - and the problem was resolved - now plays perfectly.  

Don't assume your source is good - the player may be compensating/correcting the bad/poor encode.

Edited September 2, 2020 by rbjtech

[speechles 2055]

What does the stats for nerds on the Roku app show? What is the transcoding reason?

A direct stream means an incompatible container. FFmpeg is being used to place this into an m3u8 manifest with TS slices. This is the same HLS as transcoding. It does involve ffmpeg to handle this translating into a new container and segementing slices of video/audio. This is the responsibility of ffmpeg.

The Roku stats for nerds will have a transcoding percentage on both progress and buffer. You can literally watch the Roku transcoding buffer on the stats for nerds. It asks the server for updates every 10 seconds. Those updates will either show the transcoding buffer growing or shrinking. If it is continuously eroding your transcoding buffer this means the transcoding progress may meet the buffer. This means you catch up to real time and have no buffer. It must go to that "Retrieving" screen on your Roku for a bit because of this. You are too close to the edge of progress versus buffer. You can stall progress for a bit. Pause and let the buffer build. Maybe this is all it needs. It might still erode the buffer which means ffmpeg does not have enough power to handle this. Because ffmpeg is the work horse doing everything. The Roku is just the messenger. It can show you. But it cannot correct the situation. You must find why it is transcoding. Perhaps it isn't the bitrate constraint. Maybe it is trying to do subtitles. These won't presently show as transcode reasons when it is a subtitle issue.

The important part is the Roku stats for nerds is a tool. Please use this. It can help you self diagnose.

If the transcode reason shows "Direct play error" and you did not use the "Playback Correction" button then the header of your file is a problem. You need to run it through MKClean. The Roku trusts the media header "absolutely" when running the format detection of the video player.

Edited September 2, 2020 by speechles

[RaisonDetre72 5]

@Luke:  User's quality was set to 1080p - 10Mbps

@rbjtech:  All episodes (7 - 11) are MKVs.

@speechles: Attaching the screenshots of nerd stats from Eps. 8, 9, 10.  Ep. 10 is where I set bandwidth restriction to 5Mbps for the user on the server-side.  I don't think its a transcoding issue, as the server is fully finished transcoding usually before playback even reaches 30 seconds.  I was not aware of MKClean and will definitely check it out, as well.  Thanks for the tip.   

 

Playback did not stutter this time on Ep. 9, which was one of the skipping ones.  Interestingly, though, I did notice significantly higher overall upload bandwidth usage within the Emby VM, that generally exceeds my 25Mbps pipe.  During my earlier troubleshooting, I'd only been observing my host's bandwidth usage, which was minimal (<4-5Mbps blips).  For instance, playing Ep. 8, once its done transcoding, its still running at upload speeds between 12-40Mbps.  Once I turned on the user bandwidth restriction to 5Mbps, it dropped off, though is still consistently running between 8-20Mbps.  Is there that much overhead that I should expect during remote playback?

[Luke 42079]

Try having them lower it further and see if that helps.

[RaisonDetre72 5]

Yea, I've set the bandwidth restriction server-side to 5Mbps, and will report back if there's further skipping.  Though I am curious if there's really that much overhead I should expect, where even playing back at 5Mbps on client-side is creating 8-20Mbps upload traffic server-side.

[Luke 42079]

Hi, based on the log you sent privately it doesn't look like you've done that. The quality setting appears to be set to 20 mbps. Can you try lowering it in the Roku app and then see if that helps? Thanks.

[RaisonDetre72 5]

I'll have them confirm they've done that, though the server-side 'Internet streaming bitrate' is already set to 3Mbps.  Does the app's settings override this?

[Luke 42079]

3 minutes ago, RaisonDetre72 said:

I'll have them confirm they've done that, though the server-side 'Internet streaming bitrate' is already set to 3Mbps.  Does the app's settings override this?

No, assuming it is a remote connection and the server detects it as such, then it's fine to just set the internet streaming bitrate.

The app setting doesn't override, but rather, the lower of the two will be used.

[RaisonDetre72 5]

That's curious, then, if you're saying the log is showing its playing back at a higher bitrate than that.  I know I was watching something via local LAN during this issue, perhaps those are the higher bitrate values you're seeing?  I've reduced my bitrate since then, though am unsure if that's what was causing it.

The remote playback show being watched was 'Des'; my local LAN was watching 'Scary Movie 2' (terrible movie).

[Luke 42079]

Yes is it possible you just provided the wrong log?

[RaisonDetre72 5]

Gosh, sorry I missed your last reply, @Luke, my bad.  Though fear not, I've returned with more chasing the buffer ghost!

I have a single user on the server right now and they report that playback works okay to start, and then at random points with have fits of buffering / playing / buffering, and go back to being fine again for a few minutes, followed by more buffering during playback.  User's playback is throttled to 3Mbps, and I've confirmed the max my upload ever spikes is ~6Mbps (I have 25Mbps upload on my ISP).  Literally nothing else is running on the network right now so there shouldn't be any bandwidth bottlenecks.  I've included the general Emby server log, though let me know if there's another one I should include.

The times at which these stutters were reported are 8:52PM and 9:02PM, lasting for a minute or two before returning to normal playback.  User is 'Maam and Bob', file is 'Your Honor' S01E02.  CRC checked file (is clean).

If this is some ISP- / latency-related issue, what else can I be doing to try to rectify this so it stops happening?

Also strange, attaching the remux log file, despite having watched the entire hour-long episode, the log seems like it stopped just a couple minutes into playback.  Looks like it did the same thing for Episode 1 when they watched it.  Probably unrelated, but figured I'd mention.

I've uploaded the log files to our message thread on this topic already.  Thanks again for all your help!

[Carlo 4561]

Hi, I have not looked at logs but this sounds like network/Internet issues.

Is your Emby Server machine connected to the network via Ethernet cable?
How about on the user's side?

The devices they are using are connected how?

The reason I say this is if the clients are WIFI it could be some device or another user on their network using bandwidth or some electrical interference.

Real possibility of Internet issues as well.

[rbjtech 5284]

12 hours ago, RaisonDetre72 said:

If this is some ISP- / latency-related issue, what else can I be doing to try to rectify this so it stops happening?

I would force the bandwidth to 1Mbit and see if you then get a steady streaming experience.  If this works, then slowly increase until it 'breaks' and drop down 500Kb/sec to try and keep it steady.

'Speedtest' results from ISP's are meaningless as they just test to the local (usually accelerated..) dedicated servers - but connecting real point to point connections can sometimes be very poor across different ISP's.

of course, It may well be that the client simply has poor wifi - so if they can test by directly plugged into their router - then all the better and you'll be able to pinpoint the issue or at least eliminate this as the problem.

 

[RaisonDetre72 5]

Thanks for the replies.  I'll go ahead and try dropping them to 1Mbps to see if that helps.

Server-side:  Plugged directly into the router.

User-side:  The Roku they are playing off of is maybe 5' from the router.  They have a stable 100Mbps connection that I've tested multiple times while onsite.

This may be a silly question, not knowing how the client-side buffer on Emby works, but if I drop them to 1Mbps, how does that potentially resolve this if it's a latency issue?  Is there a set buffer size on the client where at a lower bitrate, it has more space to pre-load the next frames?

Also, if anyone has any suggestions on a utility to use to actually test a connection's "health" (speed / latency) between two points, I'd be greatly appreciative.

[Luke 42079]

Let us know if dropping it helps. Thanks.