2 Rokus Buffer, 1 Doesn't

[Jdiesel 1114]

Definitely seems funny and not bandwidth related. If you have Netflix on you Roku you can use the apps bandwidth test to dispel any issues due to bandwidth, before any says anything yes I am aware that Netflix has preferential routing but using the bandwidth test is still a good way to troubleshoot local and wan connections.

 

Yes you are correct direct stream should only be a container swap or audio transcode both of which should have minimal impact on server resources or increase in bandwidth required.

[snodrog742 31]

Ok that's different, I thought we were just talking about buffering here.

No, this seems to have gone away with changing of transcoding settings or just coincidentally.  Either way, not complaining since buffering is gone.  I do understand this kind of switched the issue half-way through but figured easier to keep condensed instead of brand new topic.

 

Definitely seems funny and not bandwidth related. If you have Netflix on you Roku you can use the apps bandwidth test to dispel any issues due to bandwidth, before any says anything yes I am aware that Netflix has preferential routing but using the bandwidth test is still a good way to troubleshoot local and wan connections.

 

Yes you are correct direct stream should only be a container swap or audio transcode both of which should have minimal impact on server resources or increase in bandwidth required.

 

Good idea.  I don't personally have Netflix but will see if I can borrow one for a test since Roku has killed every speedtest app.

 

We also use CloudFlare for Emby so kind of on par with Netflix in routing in some sense.  Still a good idea!

Edited July 4, 2018 by snodrog742

[Luke 37066]

Based on your description of scenes being skipped it sounds timestamp related. We're stream copying the original video stream and it probably has some bad timestamps in there. When you drop the bitrate to force a full transcode, all of this gets corrected because it's essentially a brand new video. @@Waldonnis what do you think?

[snodrog742 31]

Based on your description of scenes being skipped it sounds timestamp related. We're stream copying the original video stream and it probably has some bad timestamps in there. When you drop the bitrate to force a full transcode, all of this gets corrected because it's essentially a brand new video. @@Waldonnis what do you think?

 

That sounds pretty accurate.  Looking at ffmpeg logs there were some where it opened the file to write almost back to back.  I'm not sure if these were the timestamps of when it skipped and plan to try to test that theory later this week.

[ebr 14913]

If it is timestamp related, why doesn't it happen consistently on both devices?

[Waldonnis 148]

Based on your description of scenes being skipped it sounds timestamp related. We're stream copying the original video stream and it probably has some bad timestamps in there. When you drop the bitrate to force a full transcode, all of this gets corrected because it's essentially a brand new video. @@Waldonnis what do you think?

 

 

If it is timestamp related, why doesn't it happen consistently on both devices?

 

That's what I'm wondering as well.  If it's the same file and the two devices are being served differently (HLS remux vs just serving the file in the original container), wacky timestamps could cause some problems on one and not the other, but there are other potential causes.  The jumping and buffering are what puzzle me more than anything and would definitely want to know if it's only happening with an HLS remux (or audio-only transcode, same difference) and if it's only happening with certain files.  If it's only certain files, I may end up wanting to take a closer look at one (especially the GOP structure and timestamps).

 

I'll look at the logs to see if anything jumps out at me first, though.

[LindsayCole 0]

We can also provide access to a server for testing and such if needed

[snodrog742 31]

We can also provide access to a server for testing and such if needed

 

Absolutely was going to say the same thing, just got distracted with work. Bleh.

 

Happy to test further as well but am usually limited to Friday-Sundays due to work.  Thanks!

[Waldonnis 148]

That sounds pretty accurate.  Looking at ffmpeg logs there were some where it opened the file to write almost back to back.  I'm not sure if these were the timestamps of when it skipped and plan to try to test that theory later this week.

 

I'm seeing the same thing in the "Last Man Standing" ffmpeg log for the remux, which isn't normal.  Looking over some of the other logs, serving the file itself without remuxing (Log-2 in the OP) is reported to be fine, and transcoding the video (the Logan log) also was reported to be fine barring a small hiccup that one could attribute to latency or environmental/network factors.  This is starting to look more like a problem with a video remux/segmentation situation, so we may be narrowing it down a bit.  Are there any videos where the problem doesn't pop up even when it's remuxing the video?

 

There could be a few reasons for the pixelation.  Since ffmpeg is being told that starting a segment on a non-keyframe is okay, any interruption in the stream could mean the player doesn't have a keyframe to reference for properly decoding subsequent B- or P-frames (both types are kinda like "diffs from other frames" rather than full frames themselves, so without an I frame, they can't be decoded entirely on their own).  This ordinarily shouldn't be a problem with HLS, but some players don't handle it too well, especially when seeking.  This would definitely lead to some pixelation if it tried decoding anyway or maybe even some jumping/freezing if larger segments were ignored due to decoding problems.  I've never encountered such extreme symptoms personally so I have no clue if the Roku playback is that fragile/intolerant of malformed streams or segments not starting with keyframes (I've always just segmented on GOP boundaries, but can see why the concept of break_non_keyframes would be attractive for a remux).

 

Discontinuous or funky timestamps, of course, could also cause problems, and definitely could lead to weird segmentation, so I'm not ruling that out yet either.

 

Any chance I can get a copy of one of the misbehaving files?  I'd like to check the timestamps and try segmenting it locally to see if those "back-to-back" segment write messages are accurate (meaning we could have single-frame segments, which is bad and something I've seen pixelation and less drastic jumpy behaviour from before) or if the logged messages were just delayed during the log write.

[snodrog742 31]

Any chance I can get a copy of one of the misbehaving files?  I'd like to check the timestamps and try segmenting it locally to see if those "back-to-back" segment write messages are accurate (meaning we could have single-frame segments, which is bad and something I've seen pixelation and less drastic jumpy behaviour from before) or if the logged messages were just delayed during the log write.

 

Yes.  How would you like to get it?

[speechles 1917]

You can also, use the transcoding-temp folder to see what is going on. Reproduce the issue on the roku and pause right after it happens. In the servers transcoding-temp folder are the m3u8 and ts slices the roku was fed. Therein lies the answer.

 

NOTE: make sure to just pause on the roku and not to stop, or press back as these will tell the server playback stopped and it will dispose of the temp files.

Edited July 6, 2018 by speechles

[snodrog742 31]

You can also, use the transcoding-temp folder to see what is going on. Reproduce the issue on the roku and pause right after it happens. In the servers transcoding-temp folder are the m3u8 and ts slices the roku was fed. Therein lies the answer.

 

NOTE: make sure to just pause on the roku and not to stop, or press back as these will tell the server playback stopped and it will dispose of the temp files.

 

I can try this I'm sure tomorrow. Good idea. Thanks!

[Waldonnis 148]

Yes.  How would you like to get it?

 

Sorry for the delay...had to re-encode 10 files because someone (not me!) screwed up when they authored a disc.

 

Hmm, not sure how to arrange that.  If you have space on a Dropbox or Google Drive account, that's probably good enough and I can let you know when I've nabbed it so you can free up the space.

[snodrog742 31]

Sorry for the delay...had to re-encode 10 files because someone (not me!) screwed up when they authored a disc.

 

Hmm, not sure how to arrange that.  If you have space on a Dropbox or Google Drive account, that's probably good enough and I can let you know when I've nabbed it so you can free up the space.

 

Sent file by PM.

[snodrog742 31]

Tried a completely different show last night, was Transcoding this time and was unwatchable due to skipping scenes on Roku.  Playing today on Apple TV (Direct Stream which is usually what Roku was doing) and not a bit of issue.

 

I didn't get a chance to grab the m3u8 files as @@speechles suggested but this seems to just be Roku issue as same file is fine on other devices.

[Waldonnis 148]

Hmm, one odd thing off the bat that's probably unrelated to this file or problem: segment time isn't in-line with the forced keyframe timing.  -segment_time is set to 3 seconds, but keyframes are being forced every two seconds.  As a result, segment durations of the output end up being irregular.  As a result, the segment_time is ignored unless the split time happens to align with a keyframe.  Example from the generated playlist using the command from one of the logs:

#EXTINF:3.628633,
foo0.ts
#EXTINF:2.377378,
foo1.ts
#EXTINF:3.086422,
foo2.ts
#EXTINF:2.919589,
foo3.ts
#EXTINF:3.169844,
foo4.ts
#EXTINF:2.836178,
foo5.ts
#EXTINF:4.004000,
foo6.ts
#EXTINF:2.002000,
foo7.ts

If force_key_frames were set to "expr:if(isnan(prev_forced_t),eq(t,t),gte(t,prev_forced_t+3))" instead (+3 instead of +2), you get more predictable segment durations since the keyframe generation will line up with -segment_time 3.  There are still some oddball durations in the new playlist, but all are +/- 0.5sec or so (which could probably be further fixed by disabling scenecut and/or modifying min-keyint):

#EXTINF:3.003011,
foo0.ts
#EXTINF:3.003011,
foo1.ts
#EXTINF:3.003000,
foo2.ts
#EXTINF:3.003000,
foo3.ts
#EXTINF:3.003011,
foo4.ts
#EXTINF:3.003011,
foo5.ts
#EXTINF:3.003000,
foo6.ts
#EXTINF:3.003000,
foo7.ts

I'm still looking into this, but so far, I'm seeing no timestamp anomalies in the file itself.  I also haven't done a remux scenario yet (still transcoding-only), but the above caught my eye immediately when I saw the segments first start to pop out.  It's not likely to be a huge problem, but something that should be fixed either way so that it's consistent. Edited July 10, 2018 by Waldonnis

[Luke 37066]

@@Waldonnis, when the segment length is 3, i will change that expression to 3. Thanks.

[snodrog742 31]

I'm going to pretend I have some idea of what was said... lol.  Appreciate the attention and happy to test further whenever needed.

[Waldonnis 148]

I'm going to pretend I have some idea of what was said... lol.  Appreciate the attention and happy to test further whenever needed.

 

Hehehe, it's just details about how the video is cut into small sections for streaming.

 

I think I spotted what's going on here, but still want to confirm it and I'm not sure how to fix it (yet).  If I'm right, it should only occur when segmenting video via remuxing rather than transcoding, so that's a plus.  Might have to bust out the hex editor and stream analyser for this one, so it might take some time (stupid analyser is segfaulting on me right now, so I have to recompile it).

[Jdiesel 1114]

In layman's terms what is the benefit of a segment length of 3s over a segment length of say 5s? I assume a segment of 3s will start quicker but would a longer length improve performance on high latency networks? Just curious

Edited July 10, 2018 by Jdiesel

[snodrog742 31]

Hehehe, it's just details about how the video is cut into small sections for streaming.

 

I think I spotted what's going on here, but still want to confirm it and I'm not sure how to fix it (yet).  If I'm right, it should only occur when segmenting video via remuxing rather than transcoding, so that's a plus.  Might have to bust out the hex editor and stream analyser for this one, so it might take some time (stupid analyser is segfaulting on me right now, so I have to recompile it).

 

Does it mean anything if I say a movie was transcoding last night and was doing the same thing?

[Waldonnis 148]

In layman's terms what is the benefit of a segment length of 3s over a segment length of say 5s? I assume a segment of 3s will start quicker but would a longer length improve performance on high latency networks? Just curious

 

There's a lot of disagreement over optimal segment size, and a lot of research done by CDNs about it with conclusions that also conflict.  For CDNs, shorter vs. longer affects how quickly their adaptive streaming will...adapt (how quickly it can swap to a lower/higher bitrate variant), for one.  That doesn't really apply to Emby since it's not using adaptive streaming/variants, but making the segments too short or too long can affect throughput and buffering potential, as well as cause the encode itself to be less efficient or impact quality (partially due to keyframe frequency). And then there's the devices themselves, some of which are more tolerant of longer segments than others.

 

Does it mean anything if I say a movie was transcoding last night and was doing the same thing?

 

It depends.  If the video was transcoding, then yep, I'd be interested in that.  If it was just transcoding the audio and copying the video, then it's the same situation that I'm investigating now.

[snodrog742 31]

It depends.  If the video was transcoding, then yep, I'd be interested in that.  If it was just transcoding the audio and copying the video, then it's the same situation that I'm investigating now.

 

I will look which was transcoding tomorrow but pretty sure it said both audio & video.

[Waldonnis 148]

Well, I'm not quite sure of what's going on here.  There are some things in the bitstream of the LMS file that are a tad unusual and may affect decoding in some circumstances, but I'm just not positive that's happening in this case (still within spec, so it's not like the file's broken).  What's stranger is that I tried to play this back locally on one of my Rokus making sure Emby remuxed the video while transcoding the audio and didn't notice any oddities during playback.  This was a low-latency local connection (playback device is in the same room as the server and the router), though, so it probably does not reflect the conditions of the OP's situation.

 

I'll keep looking, though.  Is there a specific spot that it skips repeatedly and reliably so I don't have to watch the entire video a bunch of times?  It'll also help me narrow down where to look in the segments and source. 

[snodrog742 31]

Well, I'm not quite sure of what's going on here.  There are some things in the bitstream of the LMS file that are a tad unusual and may affect decoding in some circumstances, but I'm just not positive that's happening in this case (still within spec, so it's not like the file's broken).  What's stranger is that I tried to play this back locally on one of my Rokus making sure Emby remuxed the video while transcoding the audio and didn't notice any oddities during playback.  This was a low-latency local connection (playback device is in the same room as the server and the router), though, so it probably does not reflect the conditions of the OP's situation.

 

I'll keep looking, though.  Is there a specific spot that it skips repeatedly and reliably so I don't have to watch the entire video a bunch of times?  It'll also help me narrow down where to look in the segments and source. 

 

I totally understand.  I can't accurately say it does it at a specific spot because 1. I rarely re-ewatch something especially multiple times to see it skip.  2. Once it does it one time, I wait and go to the dreaded Plex.

 

I did watch Ready Player One last night and it did it like a minute in to the movie so might try playing it again and see if it does it in the same spot.  This was transcoding audio only.