Shield Experience 9 Release

[FrostByte 5049]

52 minutes ago, ebr said:

Actually there have been problems reported by some since the Nvidia update. Singling out the file access being the culprit in this situation is a complete guess at best. 

 The problem is people are easily fooled by what they think they are seeing. Take the recent conclusion by a user that file access was quicker for seeking when both tests were actually via http.

Bottom line, there are a lot of changed variables here. 

Well, the only way to prove it I suppose is to get DFA working and then see.  If that isn't going to happen then everything anyone says is just a guess.

[rbjtech 4260]

I'll get to the SMB vs HTTP testing later on - but this is a packet capture I saved on Android 11 when DFA was not working.

It repeats the SMB Create Request 3 times - below is the 3rd time and after the last Close Request, Emby falls back to HTTP.

This is the packet capture from today - with Android 9 and DFA Working - exactly the same hardware with the same file.

This is from the very first create Request File - we get an immediate (102.993 secs > 103.004 secs) response with data.  tbh, I'm not sure what it is doing here - possibly probing the file to get the playback data, refresh rate etc ?  the 2nd screenshot is the actual stream @ 104.45 secs

This is where the stream starts.

To note this includes a Refresh Rate switch on my TV.

So in summary

initial request - 102.993

playback starts - 104.447

DFA = 1.454 seconds

Let me turn off Refresh rate switching to see if that makes any difference or adds a delay ...

request - 63.859

playback starts - 65.281

DFA = 1.422 seconds

So it appears Refresh rate switching does not make any difference to the background task (both are 1.4 seconds)

Next I'll do the same with HTTP - I expect it in this scenario to be pretty identical, it may even be faster - the emby server doing the HTTP conversion is on a i7 12700K with Direct Attached Storage ..  the real test will be on much lower powered hardware via NAS (and indirect SMB connection).   That will take a little while to setup.    

Edited February 4, 2022 by rbjtech

[rbjtech 4260]

so the HTTP is a little bit more difficult to decipher what's going on ..

This appears to be the stream GET (51.714) - 1284691 is the correct ItemId..

and this is the Session POST in the log -

the first POST doesn't make a lot of sense, as that is before the GET ?

the second POST is probably the actual playback (being written to the log - SessionManager) ? (52.939)

That is 1.2 Seconds from the GET - which seems 'about right' - but I'm not sure if I'm reading the pcap correctly .. hmmm

....

Edited February 4, 2022 by rbjtech

[caalin 67]

On 2/3/2022 at 6:22 PM, ebr said:

Actually there have been problems reported by some since the Nvidia update. Singling out the file access being the culprit in this situation is a complete guess at best. 

 The problem is people are easily fooled by what they think they are seeing. Take the recent conclusion by a user that file access was quicker for seeking when both tests were actually via http.

Bottom line, there are a lot of changed variables here. 

He may have placeboed himself, but his conclusion was correct. In my setup (emby server on the shield, NAS mounted to the shield) direct access is and always was faster for skipping than http and it's even more noticeable for high bitrate files. Not only that, but something must have happened a couple of months ago and now files with high bit rates (>30mb) hang when using http

[Spaceboy 2493]

4 hours ago, caalin said:

 now files with high bit rates (>30mb) hang when using http

i have never experienced this

[rbjtech 4260]

5 hours ago, caalin said:

He may have placeboed himself, but his conclusion was correct. In my setup (emby server on the shield, NAS mounted to the shield) direct access is and always was faster for skipping than http and it's even more noticeable for high bitrate files. Not only that, but something must have happened a couple of months ago and now files with high bit rates (>30mb) hang when using http

I didn't get time yesterday to continue my in-depth testing - but could you just expand/confirm your setup where you have the issue ?

1. NAS > 1Gig Ethernet > Network Switch 

2. Shield Server > 1Gig Ethernet > Network Switch

3. Shield Client

If the above is correct, then it is exactly this sort of configuration where SMB makes complete sense and is a valid use case.

Using HTTP - the data path is : Shield Client > HTTP > Shield Server < SMB < NAS

Using SMB - the data path is : Shield Client < SMB < NAS

As Eric has alluded to above - there are 'many' variables/scenario's that contribute to this technical discussion - putting out statements such as 'mine works' and 'you don't need DFA' is unhelpful without the evidence to back it up.

Now I'm back to Android 9 on my Shield - I should be able to prove where DFA has a valid justification (or not) to be used - from the looks of it @caalin may have identified one scenario, but we need to confirm what is causing the hang. 

[Riggs 300]

@ebr I reported what I found, none is coincidence, what I said was tested in practice and still happens after the update.

@rbjtechThanks for you time to go deep with this.

This: 

Quote

If the above is correct, then it is exactly this sort of configuration where SMB makes complete sense and is a valid use case.

Using HTTP - the data path is : Shield Client > HTTP > Shield Server < SMB < NAS

Using SMB - the data path is : Shield Client < SMB < NAS

My home network and SMB is like that example. The only difference is that I'm over 10GB LAN (1GB compatible) network to avoid bottlenecks when two Shields and another TV device with Android TV are pulling heavy 4K content from my server at the same time.

However pulling one or two movies before the upgrade of my Home Lab never was a problem to play using direct path.

Direct path with Shield works better for me.

[ebr 14913]

8 hours ago, caalin said:

but his conclusion was correct

Based on the only working install with the new firmware we've identified, that conclusion is incorrect.  Direct access is slower in that case.

2 hours ago, rbjtech said:

I should be able to prove where DFA has a valid justification (or not) to be used

You would be "proving" something on an old version of both Shield OS and Android OS.  That won't tell us anything about the state of affairs with Shield 9 and Android 11.

[CBers 6771]

42 minutes ago, ebr said:

Based on the only working install with the new firmware we've identified, that conclusion is incorrect.  Direct access is slower in that case.

You would be "proving" something on an old version of both Shield OS and Android OS.  That won't tell us anything about the state of affairs with Shield 9 and Android 11.

@ebr If you remove the direct file access method, we'll never know if Nvidia fix the problem, if indeed it is an Nvidia issue to fix.

Have you contacted Nvidia to see if it's something they're aware of, or are you just wanting to remove the direct file access method? 

Personally, I have never got DFA to work, so it's no skin off my nose if you keep it or remove it, but others have used it with great success. 

Just my 2 cents. 

Thanks. 

 

[ebr 14913]

9 hours ago, CBers said:

others have used it with great success

Again, that's irrelevant to the current situation.

If anyone can produce a problem with the new firmware using http delivery, then please create a topic.

[CBers 6771]

11 hours ago, ebr said:

Again, that's irrelevant to the current situation.

If anyone can produce a problem with the new firmware using http delivery, then please create a topic.

 

20 hours ago, CBers said:

@ebr

Have you contacted Nvidia to see if it's something they're aware of, or are you just wanting to remove the direct file access method? 

 

[Riggs 300]

On 2/3/2022 at 7:17 PM, FrostByte said:

Take the recent conclusion by a user that file access was quicker for seeking when both tests were actually via http.

Make no mistake, I was the one who put that, but I was talking about before the update, not after.

[rbjtech 4260]

22 hours ago, Riggs said:

My home network and SMB is like that example. The only difference is that I'm over 10GB LAN (1GB compatible) network to avoid bottlenecks when two Shields and another TV device with Android TV are pulling heavy 4K content from my server at the same time.

Thanks. Can you raise as a new topic that http/Android 11 no longer works.

The 10Gig Network should not be required at all - even on a 100Mbit/sec 4K remux, you should be using 1/10th of the capacity of a 1Gig network for a single stream.

There should be ZERO issues streaming 8-9 x 100Mbit/sec streams over a 1 Gig network (800-900Mbit/sec) - over SMB/DFA because Emby is not involved.  The NAS interface will be in Tx mode, the Shield (or whatever) client will be in Rx mode.

When in HTTP mode, Emby server itself is the 'bottleneck' and has to process this extra workload - NAS interface will be in Tx mode, Emby Interface will be in Rx mode AND Tx mode as it then has to send the data to the Shield (in Rx mode) - ie - it was TWICE the workload vs SMB/DFA. 

Many systems should be able to handle this Full Duplex Network I/O - many will not, and that is the crux of the problem - we just need evidence to support it.  

[rbjtech 4260]

21 hours ago, ebr said:

You would be "proving" something on an old version of both Shield OS and Android OS.  That won't tell us anything about the state of affairs with Shield 9 and Android 11.

We will be proving the use case for DFA - if I'm reading it correctly, your tone in this thread suggests you think there is no technical need for it and ultimately want to remove it.  A lot of us beg to differ.

Once we have agreement on the need for it or not,  then, assuming we do -  we can move onto the next stage of why it does not work in Emby on Android 11 when other apps have no issues with it.

It pointless moving onto the next stage if we genuinely have no need for it, but we can only make that decision with use cases.

[Riggs 300]

57 minutes ago, rbjtech said:

Thanks. Can you raise as a new topic that http/Android 11 no longer works.

The 10Gig Network should not be required at all - even on a 100Mbit/sec 4K remux, you should be using 1/10th of the capacity of a 1Gig network for a single stream.

There should be ZERO issues streaming 8-9 x 100Mbit/sec streams over a 1 Gig network (800-900Mbit/sec) - over SMB/DFA because Emby is not involved.  The NAS interface will be in Tx mode, the Shield (or whatever) client will be in Rx mode.

When in HTTP mode, Emby server itself is the 'bottleneck' and has to process this extra workload - NAS interface will be in Tx mode, Emby Interface will be in Rx mode AND Tx mode as it then has to send the data to the Shield (in Rx mode) - ie - it was TWICE the workload vs SMB/DFA. 

Many systems should be able to handle this Full Duplex Network I/O - many will not, and that is the crux of the problem - we just need evidence to support it.  

It's true, you are right, 10GB is not necessary, my point was that the developer knows that I have no limitations in my internal Home Lab. I went to the next level because I also have a 1GB internet connection and everything goes to my pfsense router that I have been using for a few years, which is actually a PC with an i7 processor and 16Gb of RAM that will never have resource routing problems. . I have it with an Intel x540T2 network card, plus it saves me time copying the movies I pull from my personal 4K media which are sometimes over 100GB in size and doing a lot of work in near real time directly on the media server .

1GB with a good cable, in my case, cat6 yes, it was enough as you indicate, but then other things came into play that led me to go to the next level.

I understand well how Rx and Tx work and I agree with what you say, however, we have already had other fights with @ebr in the past so that he understands certain things that he does not want or does not feel like doing. One assumes that @ebr understands this whole I/O, duplex, Tx and Rx stuff, but hey, it's always easier to blame someone else. Doing the SMB/CIFS tests on the Nvidia with the new firmware with other applications, it works fine, it is clear that the problem is the @ebr application (sorry for saying "the ebr application", but the user can no longer even help him without getting angry or avoiding responding)
What I had forgotten is to try the Emby application for android, to see if it gives the same problem, I don't know if ebr develops that application, it seems to me that it is a parallel development of that APP and it is not led by ebr. Maybe that is the way because every version change in the Shield there is a discussion with @ebr for months.

[FrostByte 5049]

1 hour ago, Riggs said:

Make no mistake, I was the one who put that, but I was talking about before the update, not after.

Not sure how you quoted me for that, but ebr is the one who twisted your words.  Not me

There's been enough people who got DFA to work in the past that found it helpful to now say it can be useful for some.  I may be delusional, but I doubt everyone claiming it helps is.

nVidia says Plex reached out to them when Plex had problems after SE9, maybe Emby can do the same.  

 

[ebr 14913]

1 hour ago, FrostByte said:

ebr is the one who twisted your words. 

Did not intend to twist any words.  I asked specifically how he knew that Direct access was "faster" with the new update in place and he responded to that with a statement that he tested it.

Sorry if there was a misunderstanding.

[ebr 14913]

I'm not arguing the theory that direct access is more efficient.  I'm arguing whether or not that extra efficiency is needed in 99% of cases.

This option is an extremely expensive one for us.  This thread is a great example of why.  It is difficult to setup properly, difficult to understand and even more difficult to prove it actually translates to increased performance.  All of that makes it very expensive for us to support so, if it doesn't provide a very real and very needed benefit, then it isn't worth it.

[FrostByte 5049]

HTTP has come a long way since DFA was first added.  Whatever Luke, or whomever, did has helped a lot.  For me, even movies like Gemini Man (90Mbps/60fps) can be streamed now using HTTP.  Couldn't do that before w/o DFA.

[rbjtech 4260]

24 minutes ago, ebr said:

This thread is a great example of why. 

The thread exists and is this active because a feature that many used 'without incident' is no longer working.  If you can fix Emby to use Android 11 like otehr Apps have, then this thread will be totally redundant ...

24 minutes ago, ebr said:

It is difficult to setup properly,..

After my Android 9 recovery, it took less than 30 seconds to configure in the Shield...

24 minutes ago, ebr said:

I'm arguing whether or not that extra efficiency is needed in 99% of cases.

 And we aren't arguing that case either - we agree with you lol - we are simply suggesting that for that 1% of cases that DO need it (for whatever reason), then why take that reason away ?

24 minutes ago, ebr said:

..even more difficult to prove it actually translates to increased performance. 

I'm in the middle of doing this - it is reasonably clear that on decent hardware, it does make no difference and HTTP is fractionally faster.

I've literally just finished an experiment where I have setup a '1Gig NAS' (instead of my production Direct Attached Storage) and had 6 x 100Mbit 4K remux's running over HTTP - zero issues on any of them, BUT the Emby server is my current i7 12700K - and the NAS was on a separate VLAN and physical interface to the Clients. ie Best Case Scenario.

I now want to go the opposite end of the hardware spectrum, and run Emby Server on a low powered CPU - maybe the Shield itself on the same Network (that 99.9% of people will have) and see if I can repeat the exercise.  ie Worst Case (typical?) Scenario.

Edited February 6, 2022 by rbjtech

[rbjtech 4260]

7 minutes ago, FrostByte said:

HTTP has come a long way since DFA was first added.  Whatever Luke, or whomever, did has helped a lot.  For me, even movies like Gemini Man (90Mbps/60fps) can be streamed now using HTTP.  Couldn't do that before w/o DFA.

Agreed - it has, but it doesn't scale well and it will always be a bottleneck from a network perspective - and that is where DFA has a BIG advantage. 

I'm working on it today, if the Shield or a low powered (1-2 core 1Ghz cpu) can stream 6 x 4K remix 70-90Mbit streams over HTTP without issues - then I'm totally happy to hold my hand up and say it's no longer needed for anything but extreme setups.

[ebr 14913]

11 minutes ago, rbjtech said:

After my Android 9 recovery, it took less than 30 seconds to configure in the Shield...

You accept that you are not an "average" user, correct...?  

11 minutes ago, rbjtech said:

why take that reason away ?

I explained that.  It costs us a lot in support time and also can lead users down unnecessary rabbit holes.

[CBers 6771]

56 minutes ago, ebr said:

It costs us a lot in support time and also can lead users down unnecessary rabbit holes.

If you're not going to contact Nvidia for help, why not just remove the option and render this discussion moot? 

 

[rbjtech 4260]

So I'm making progress...

It' appears to be a cpu/thread limitation - I'm not sure if the http engine is multi-threaded, but I'm now running Emby with Direct Attached Storage on an old i3 (12Gb Ram) - reduced to 2 cores (hyperthreading disabled) and reduced to 1.2Ghz (as low as it would go), the 6 x 4K (~400-600Mbit/sec) test streams are now choking / stalling (but not stopping).  Interesting, the CPU is not anywhere near Max and nor are the disks - so something is choking the streams ..    This test simulates Emby Server running on a NAS.

Next is to add SMB read into the mix (ie remove Direct Attached Storage) .. this will simulate Emby running on a Shield using a NAS as the storage.  I think this will be a worst case scenario.. predictions ?

[Riggs 300]

11 minutes ago, rbjtech said:

predictions ?

@ebrdon't wanna look into this