The UI is soooo slow... until it isn't?

[randywest55 0]

I guess I'd be surprised if any of the hardware was sleeping in case 2 from my last post (video already streaming, so all hardware would be engaged on my server). Really though, I'm not quite sure what would be "sleeping" on an always-on NAS. Sure, there's SpeedStep, but that's transparent. The drives never spin down, so far as I'm aware. Perhaps you could clarify. Something I'm missing?

[ebr 14910]

How do you know the drives don't spin down?

[randywest55 0]

WD Red drives don't have a standby function, I believe. Here's a source: https://withblue.ink/2016/07/15/what-i-learnt-from-using-wd-red-disks-to-build-a-home-nas.html. That, and through usage experience: I've never experienced any delay in basic file access after long idle periods.

[OddbOd 21]

Having actually tried standby mode on one of my RED drives I have a hard time believing that page.
 
The following is copied verbatim from my SSH client, no edits have been made:-
 

[admin@TurboStation ~]# hdparm -C /dev/sdf

/dev/sdf:
 drive state is:  active/idle
[admin@TurboStation ~]# hdparm -Y /dev/sdf

/dev/sdf:
 issuing sleep command
[admin@TurboStation ~]# hdparm -C /dev/sdf

/dev/sdf:
 drive state is:  standby
[admin@TurboStation ~]# hdparm -i /dev/sdf

/dev/sdf:

 Model=WDC WD40EFRX-68WT0N0                    , FwRev=80.00A80, SerialNo=     WD-WCC4E1023100
 Config={ HardSect NotMFM HdSw>15uSec SpinMotCtl Fixed DTR>5Mbs FmtGapReq }
 RawCHS=16383/16/63, TrkSize=0, SectSize=0, ECCbytes=0
 BuffType=unknown, BuffSize=0kB, MaxMultSect=16, MultSect=?0?
 CurCHS=16383/16/63, CurSects=16514064, LBA=yes, LBAsects=268435455
 IORDY=on/off, tPIO={min:120,w/IORDY:120}, tDMA={min:120,rec:120}
 PIO modes:  pio0 pio3 pio4
 DMA modes:  mdma0 mdma1 mdma2
 UDMA modes: udma0 udma1 udma2
 AdvancedPM=no WriteCache=enabled
 Drive conforms to: device does not report version:

 * signifies the current active mode

[admin@TurboStation ~]# hdparm -C /dev/sdf

/dev/sdf:
 drive state is:  standby
[admin@TurboStation ~]# hdparm -z /dev/sdf

/dev/sdf:
[admin@TurboStation ~]# hdparm -C /dev/sdf

/dev/sdf:
 drive state is:  active/idle
[admin@TurboStation ~]# hdparm -z /dev/sdf

/dev/sdf:
[admin@TurboStation ~]# hdparm -y /dev/sdf

/dev/sdf:
 issuing standby command
[admin@TurboStation ~]# hdparm -C /dev/sdf

/dev/sdf:
 drive state is:  standby
[admin@TurboStation ~]# hdparm -z /dev/sdf

/dev/sdf:
[admin@TurboStation ~]#

What you can't see in the above is that the first time I issued the -z (re-read partition table) command there was a delay of 5 seconds before hdparm completed, the second time there was no delay and the third time there was also a 5 second delay.

 

From what I can tell it's automatic standby that is not supported on these drives. When I tried setting the spindown timeout to 5 seconds (hdparm -S 1) there was no effect at all.

 

@@ebr The above not withstanding, I don't buy this as a cause of poor performance as I've seen similar problems occur before.

 

About a year ago (it was during the mono based 3.0.x releases I don't recall exactly when) a number of users on the QNAP forum were experiencing very slow initial library scans, additionally their server response times got increasingly longer once the server had been left running for a few days. Luke made a change that had something to do with the database and those problems vanished.

 

I don't mean to imply that the situation here is identical but my gut tells me that randywest55's concerns are valid.

[randywest55 0]

Thanks for the clarification. I certainly haven't configured the drives to sleep when idle, so I doubt something like that would be happening. Regardless, though, I'll note to strengthen argument against sleeping hardware that 1) I can access the file system directly without delay when the system has been idle for a long period, and 2) the minute+ delay I often see in emby doesn't seem to correspond with hdd spinup time.

[sectune 9]

I have also noticed the exact same behavior, also having WD red drives and the i7 qnap TVS-671 so it's not the CPU or ram (16gb). There is no HDD sleep configured and as my previous peers said, the files directly on qnap start always instantly. The cold start (after idle time) of emby takes a long time for the web UI, alternatively if I refresh the web page or the emby app it loads quite fast and snappy. Please let me know how exactly to.debug this or if there is a specific packaged qpkg version with more verbose output out there. Thanks !

[PenkethBoy 2063]

@@randywest55 and @@sectune

 

what qnap firmware version are you on?

which version of the Emby server are you using mono or .net core?

 

Are you leaving the browser open connected to emby and then coming back to it after a period of time and then its slow?

or are you opening a new browser session and then connecting to the server and seeing a slow response?

 

What else are your NAs's doing beside file sharing - do you have lots of apps loaded?

 

Using resource monitor (assuming your are on an up to date firmware) have you checked your IOPS/Memory use/cpu load etc - ideally before you interact with emby after a period of inactivity and then after you try to use it?

 

Do you get high latency values on you volumes?

 

I have a 853A with 8 disk in a raid5 and do not see these slowdowns using the 60.2 .netcore version of emby - i will leave my NAS overnight and see if i can get it to be slow

[randywest55 0]

@@PenkethBoy, I've answered all of your questions above elsewhere in the thread with the exception of QNAP firmware version, for which I'm at 4.3.4.0427 (latest). You'll note in particular in the first post that I established that the application was IOPS bound in the DB on my system. Could be interesting to compare with answers to your questions from @@sectune, though.

[randywest55 0]

@@sectune, I'm glad to hear that this happens on more powerful hardware than mine. If you're not on the .net core version, I'd recommend it. I began the thread on the mono version, and while I'm still having problems, .net core (and the much newer version of SQLite packaged in) did seem to provide a significant boost. As for logging, please turn on debug logging, then once you've experienced the issue, note the time precisely and look for lines with "sqlite" and "(slow)" in them. Also please be ready with your resource monitor open before trying to access emby as @@PenkethBoy suggested. If your issue presents the same way mine does, you'll see no appreciable uptick in resource consumption except for IOPS, which shoot up to the limit of WD Red capabilities.

[randywest55 0]

Bumping the thread...

 

Any progress? Are the devs satisfied that this constitutes a bug? In that case, maybe move it over to GitHub, or however you guys track issues?

[Luke 37058]

We're looking into it, thanks.

[randywest55 0]

Great, just wanted to make sure. Thanks for your work on this, and let me know if there's anything else I can provide to help.

[sectune 9]

Dear all, sorry just wanted to give my sign of life, holidays were hard on me I will replicate the issue I described with the debug log (same issue the others have). For the record, running newest emby server mono testing version, newest qnap OS, not running any hugely demanding apps in parallel, just one VM and some nzb / torrent apps. At any time my NAS has more than 6gb of free RAM and CPU is bored all the time besides when emby is transcoding. Regarding the slow UI start: this happens when I do not use emby for a while and open a new session. I do not have any estimation yet how long it takes but I would guess if I don't use it for 1h or so. If the UI loads slow, I refresh the emby app on mobile or in web (device I am running emby client from does not matter, seems emby server related) it loads the UI faster by manually refreshing, if I wait it can take more than twice as long, approx 15 sec or so to load the UI. Normal UI load time for me is around 2-4s. I will edit this thread with the logs as soon as I will have time and get sober

Happy New year!

Edited January 5, 2018 by sectune

[Luke 37058]

Hi there, can you re-evaluate this on the latest QNAP package:

 

https://emby.media/community/index.php?/topic/51334-new-qnap-package-for-testing/

 

Thanks !

[randywest55 0]

Sure thing. I'm trying it now.

 

I didn't see anything in the release notes that seems to pertain to this, but perhaps they aren't comprehensive? In any event, I'll report back once I've played with it for a while.

[randywest55 0]

The experience seems to be more or less the same. IOPS through the roof and ~2 minute initial load time on average, presumably still bottlenecked by the time taken for the first db query.

 

One thing that I don't think I've mentioned before that might help: Whenever I first try to load the home page, the libraries populate almost immediately. I'm pretty sure this is consistent behavior. However, all of the "next up," "latest," etc. queries take forever to return. So maybe more accurately, the bottleneck is the *second* db query. Assuming the queries are issued in a predictable order, it seems like looking at the difference between the first and second queries, or if not, the difference between the library and other queries, would be a good direction of inquiry. I'm just guessing, but perhaps the library query *doesn't* join with the UserData table, but most of the others do? Remembering that I noticed UserDataKey wasn't indexed, this could provide an explanation.

[Luke 37058]

Great, thanks.

[randywest55 0]

Noticed something weird. In the QNAP AppCenter, my Emby version is shown as 3.2.70.6. However, in the Emby server dashboard, it still says 3.2.50.3 beta, along with the standard new version notice and link, which of course doesn't lead anywhere useful just yet for the new QNAP server. The log likewise indicates 3.2.50.3. So I guess I'm not exactly sure which version I'm running after all. Seems the new install package has some issues.

[Luke 37058]

I haven't heard of that before. Could be related to going back and forth between this and the community package but i'm not sure.

[randywest55 0]

I no longer have the community package installed, nor did I when I installed 3.2.70.6, so that would surprise me. In fact, I don’t think I’ve even started the community version since I installed the .net core version. In any event, this isn’t causing me any problems per se, just wanted to let you know since you’d asked if 3.2.70.6 improved anything.

[Luke 37058]

It's possible that it might improve the situation, yes.

[randywest55 0]

Sorry? I did install 3.2.70.6 already. What I’m telling you is that, internally, emby doesn’t seem to think it’s actually at that version. I’m not sure what else I can do on my end.

[Luke 37058]

Perhaps the install did not complete? 

[randywest55 0]

To make absolutely sure, I installed a second time from https://github.com/MediaBrowser/Emby/releases/download/3.2.70.6/emby-server-qnap_3.2.70.6_x86_64.qpkg. I was actually a bit surprised that it let me reinstall (either there's no version guard, or it's broken in relation to this issue), but regardless, the result is the same. QNAP clearly indicates that the install has completed and displays 3.2.70.6 in the app center menu, but emby internally still seems to think it's at version 3.2.50.3.

[Luke 37058]

Hi there, can you try this again on the latest version? Thanks !