Slow web loading

[Thor.2 1]

I am getting very slow page loading response times from my server, even when I am the only user running.

In the logs I am seeing a lot of over 1000ms http responses, so the interface loads very slow, and any subsequent pages. I recently migrated my setup to stand alone bare metal with a Nvidia P2000 to utilized hardware transcoding. Media playback seems to have no issues and is actually way better than before. Response times drop on a reboot but within 1h they go back to slow again.

I cant see where the bottleneck is coming from.

When I migrated I rsync'd over backup, cache, metadata, recordings, and transcoding-temp. As well as my media folders. Retaining the same paths. Do any of these folders need to be wiped on a migration/restore from backup?

My only concern from the hardware side is potentially the drive, but I would like to make sure that it is the culprit before having to do all that work migrating again....

Specs are;

Intel(R) Xeon(R) CPU E31220 @ 3.10GHz, Quadro P2000, Seagate Pipeline ST3500312CS, Tyan S5510 MoBo, 16 GB EEC Ram, running Emby 4.5.3.0 on Fedora 33 (Server edition).

No other software is loaded onto the machine, and according to the Fedora logs and metrics I have lots of compute available, and drive metrics seem ok as well.

 

[Luke 42078]

Hi there, please attach the emby server log from when this happened. Thanks.

[Spaceboy 2573]

yep, same. server on windows but seeing it onembyserver.zip all clients

Edited December 14, 2020 by Spaceboy
logs

[Thor.2 1]

2020-12-14 00:00:00.030 Info App: Application version: 4.5.3.0
2020-12-14 00:00:00.031 Info App: Emby
    Command line: /opt/emby-server/system/EmbyServer.dll -programdata /var/lib/emby -ffdetect /opt/emby-server/bin/ffdetect -ffmpeg /opt/emby-server/bin/ffmpeg -ffprobe /opt/emby-server/bin/ffprobe -restartexitcode 3 -updatepackage emby-server-rpm_{version}_x86_64.rpm
    Operating system: Linux version 5.9.13-200.fc33.x86_64 (mockbuild@bkernel01.iad2.fedoraproject.org) (gcc (GCC) 10.2.1 20201125 (Red Hat 10.2.1-9), GNU ld version 2.35-1
    Framework: .NET Core 3.1.7
    OS/Process: x64/x64
    Runtime: opt/emby-server/system/System.Private.CoreLib.dll
    Processor count: 4
    Data path: /var/lib/emby
    Application path: /opt/emby-server/system
2020-12-14 00:00:00.031 Info App: Plugins:
    Bluray Folder Support 1.0.0.0
    Cinema Intros 1.0.35.0
    Dlna 1.0.39.0
    Dvd Folder Support 1.0.0.0
    Fanart.tv 1.0.10.0
    MovieDb 1.3.8.0
    MusicBrainz 1.0.13.0
    Nfo Metadata 1.0.34.0
    OMDb 1.0.11.0
    Open Subtitles 1.0.27.0
    Port Mapper 1.0.4.0
    Server Configuration Backup 1.4.5.0
    Studio Images 1.0.3.0
    TheAudioDb 1.0.11.0
    TheTVDB 1.0.35.0
    Webhooks 1.0.13.0

 

Attached is the chunk of the log that pertains to the issue, it seems to have subsided, and I am heavily leaning towards a hard drive issue. Ill try adding another drive as a test and move the recordings and transcoding-temp to the extra drive. I believe at the time, I had one recording going and one transcoding/playing

embyserver.txt

[Luke 42078]

HI there, this is only a small log snippet. Please attach the complete emby server log. thanks.

[Thor.2 1]

Here is my log.

 

Again today with the Fire TV app, at 2020-12-14 @ 19:13-15 refused to load giving me an error on screen. Client was on 10.0.0.175

 

embyserver.txt

[Luke 42078]

Hi, Emby Server 4.5.4 is currently rolling out. Please let us know if that helps with this. Thanks.

[Thor.2 1]

Pushed the update this AM, it seems a little better but still delayed a bit. I'll dig into it this evening.

Thanks!

[Luke 42078]

Thanks for the feedback. Please attach a new log file if there's still an issue.

[Thor.2 1]

New problem, Roku app will not play high bitrate files, and doesn't ask to transcode. I'll send log file shortly

[speechles 2055]

Did you set the Quality higher than your network can support? You can adjust the quality within playback settings of the Roku app using the cog found at the top of the screen. It won't "ask" to transcode. The app tells the server what it supports by creating a profile based on what the device queries return and what you have set in settings. The server follows those directives. If you set the quality absurdly high and your network/equipment cannot support it you will see that result. You can use "Auto" on quality for the best results but sometimes it guesses low. You may have to set it manually and fine tune the best place for it to be.

The log files will show what is happening.

[Thor.2 1]

This is an outside client and usually happens automatically. It is set to auto 

[speechles 2055]

How busy is the HDD activity if you monitor it? Since that isn't ssd/nvme that would be the choke point potentially. Especially if the DB resides there. Put enough sql queries in sequence fetching their data from different platters possibly during the same call and there ya go. The log will tell us how long the responses are taking. You can also try to browse the hard drive and see how fast it populate contents. Maybe that is what is happening.

Edited December 16, 2020 by speechles

[Thor.2 1]

That was my guess as well. Especially when it is pulling images to fill the webpages. I am thinking about adding another drive or a ssd, my previous install was on 8 SAS enterprise drives in raid 6 in a Dell server. The server was faster but was not compatible with the Nvidia P2000.

[speechles 2055]

In those logs above you want to obfuscate your HOST. You left it visible above and some people on these forums harvest those with bots. They try to guess your passwords. You might want to erase your last logs you posted. Then post redacted logs with your HOST changed to [host-goes-here] or similar. Do not just remove it replace it so we understand what was there.

The logs will be the clue that tells us where the problem is. Browse around a bunch. Populate some grids full of images and scroll around forcing it to load more images. That should be a clear indicator in the logs how laggy/slow your responses are. Then you can compare that to how slow your response are when browsing local. The difference between the responses in the log for your remote users versus how long your responses are on the lan. Subtract your lan time from their remote time. That left over will be network lag. If the network lag appears the culprit that would be the problem.

Only the logs will reveal the truth. Make sure to redact them. ^_~

Or is it already redacted? It is hard to tell with those weird hostnames.

Edited December 16, 2020 by speechles

[Thor.2 1]

Haha I did obscure it and my IPTV links... My first thought too Thor.2

[Thor.2 1]

Lost the NFS links that is why the client couldn't play the media. I did an update and restarted to my file server , guess it didn't reconnect after it restarted.

Fully restart on the server and we are back.

[Luke 42078]

Thanks for the feedback.

[Spaceboy 2573]

On 14/12/2020 at 20:20, Spaceboy said:

yep, same. server on windows but seeing it onembyserver.zip all clients

@luke thanks for the response . regardless i notice you fixed it in .09

[Thor.2 1]

Substantially better response with a SSD SATA drive as the primary OS and Emby drive. Kept cache, metadata, transcoding and conversion folders on the SSD locally but all other Media files are on the file server shared by NFS including the Recordings folder. Seems to preform well even when doing conversions.

[Luke 42078]

Great news, thanks for the feedback.