Severe server performance problems - apps and web unresponsive

[knowenoughtobedangerous 6]

I have toyed with Emby for several years but have always come down in favour of Plex, for which I have a lifetime subscription.  However, after some rather awful lifechanging events, I am now in a Nursing Home and I find myself needing to use the remote control feature.  Plex has that, but it needs the 2 clients to be on the same subnetwork.  It works near perfectly if they are, but I can’t guarantee it; I am at the mercy of the building’s DHCP server as to which subnet my devices are declared.  I decided to give Emby another go.

 

As previously, Emby seemed to work well, although I found that it often suffered pauses while playing, which I put down to the fact that I am now accessing my server at home over a WAN and, I assumed, that the WAN connection from the Nursing Home was heavily loaded at times (but see below).  Plex also suffers from occasional jitters while playing streamed content, but I was able to overcome that by downloading content to my player.

 

So, I reckoned that I would need both the remote control and download features of Emby, neither of which are available in the free version, I took out a subscription for Emby Premiere but things have gone downhill from there.

 

The remote control function seems fine except that it (and ‘normal’ playback) is rendered useless by what seem to be quite extreme performance issues in the server.  The problems that I am having are so severe that I’m a bit surprised that no-one else is complaining.  On the other hand, my ‘configuration’ must be quite unusual:

 

1. Emby Server v4.2.1.0 is running on a QNAP TS-453 Pro (quad Intel Celeron 64bit 2GHz processors, 8GB memory, SSD system drive, QTOS v4.4.1.1064) located at my home.  That is ethernet-connected directly to my router.  I don’t have the experience to hack around in UNIX, so you can be sure I’ve not done anything at that level, only via the UIs.
    
2. I have a main music library with 12000+ songs over 1000+ albums, 12000+ photos (mainly .jpg and .dng files) and a paltry 49 video files.  There's a 4th small music library.
    
3. In the Nursing Home just a few miles away I’m using
   
   2a   an iPhone 6 running iOS 13.1 and Emby app 1.8.3
   
           For a variety of reasons, I use that to remote control….
   
   2b   a Lenovo E10 tablet with 32GB internal storage, running Android 8.1.0 and Emby app 3.0.82. 
   
           I’m trying to store downloads on a 256GB high-speed (<=100MB/s) SD card. 
    
4. I have measured actual throughput on the broadband (fttc) connection to my home using the BT Wholesale speed tester from a PC connected to the same router as my QNAP.  That is showing 35Mbps downstream, 9Mbps upstream and 14ms latency.  FYI BT provide the core UK broadband infrastructure across which almost all UK ISPs operate.  It is likely that both broadband connection come from the same local BT exchange.
    
5. The broadband connection to the Nursing Home is also via fttc and is, of course, much more heavily used.  Nevertheless, using the same speed-tester from a laptop connected via the same wi-fi access point as my phone and tablet, I am getting actual throughput around 6-8Mbps downstream, 2-4Mbps upstream with a latency of ~22ms
    

Despite what I would hope are perfectly reasonable network connections (latency?), I am getting quite severe bouts of very slow responses that render Emby useless.

 

1. Server responses grind to a near halt while a library scan is running.  Neither of the apps nor the web interface respond, sometimes even after waiting for minutes.
    
2. This would not be so bad if the scans would complete quickly.  I expected the initial scan to take a ‘long’ time, but subsequent scans seem quite erratic.  A couple of nights ago, the scan completed in a little over a minute.  The next night, even though there were no changes to my libraries that I know of,  it took over 9 hours.
    
3. Responses also stagnate when the Convert Media task is running.  I understand that this is something to do with Download/Sync tasks and, yes, I have some of these.  But they are only of music files so nothing too challenging to convert even if conversion were needed.  I have defined the download quality as ‘original’, ie. “Downloads original files as-is”, so I can see no reason why there is any need to do any media conversion.
    
4. Nevertheless, the Convert Media task runs each night and, as for the Scans, it seems to be a lottery how long it takes.  One night it may only be a very few minutes but the following night it might take over 11 hours (according to ‘Scheduled Tasks’).  That is a mystery in itself because the task is supposedly limited to 10 hours.
    
5. But as well as these tasks apparently clogging up the server, there appears to be something else soaking up Emby Server’s performance.  It seems that I have to restart Emby Server each morning to restore performance to a useable level.  Indeed, once restarted, performance seems quite snappy, but response times drop off gradually during the day to the point where the apps are unusable again.  A couple of days ago I managed to get to the point where I requested my ‘remote controlled’ Android tablet to play an album.  Nothing happened until a few hours later when it suddenly started playing.  It being connected to some quite powerful speakers,  I leaped out of my proverbial skin.
    
6. If it were not bad enough having to restart the server a couple or more times a day, I usually then have to restart the apps on my phone (the controller – an iPhone 6S, IOS 13.1, easy enough), but restarting the Android tablet app (Android 8) is much more of a problem because of my particular circumstances.  And, having restarted the apps, it can often take several attempts before I can connect to Emby Server.

 

And what of the download/sync?  Over the past few days I have managed to get only a single playlist of 94 tracks downloaded, but there are 4 other sync jobs sitting there showing as Queued.  Oddly, the number of items shown for each sync job varies quite considerably as time passes for no reason I can fathom.  For example, one job which should be downloading 4272 tracks is currently showing 22375 items queued.

 

A few observations which may be of use:

 

1. Even when Emby seems totally maxed-out and with the Scan and Convert tasks running, the QNAP box shows CPU utilisation averaging around 50% and RAM usage at around 40%.   Hardly overloaded?  EmbyServer is overwhelming the biggest resource user (over 30% of the total CPU and 1.5GB of the total 8GB).  And, throughout all, a replica configuration but using Plex (on the same QNAP) responds admirably.
    
2. my Emby Server log files seem to grow to a considerable size quite quickly.  Some connection with the slowing down of performance perhaps?  There seems to be a steady stream of data being written to the disks where the log files are stored.  2 or 3 writes  or ~10KB per second with a latency ~12ms.  Not a lot, I guess, but a steady stream, as  I said. 
    
3. I’ve tried running a ‘Rotate log file’ manually, but that had no effect apparent effect having a fresh log file.
    
4. I see a lot of errors being logged but don’t profess to be able to infer anything spefific from most of them.
    
5. I also note a considerable amount of log records of what I assume to be normal HTTP chit-chat between server and apps (I think).  Is that level of detail normal?  Debug-logging is not enabled.
    
6. While my tablet is playing, my iPhone will lock itself.  An interesting event today when I left it on the ‘Remote Control’ screen.  After playing nearly a whole album, I unlocked my iPhone and saw that it resumed at that same screen but still displaying the same track as when I left it.  The display then appeared to show tracks being played at fast speed, the progress bar skimming across the screen in a few seconds for each track until it had caught up with what was actually playing.  It was as though a long queue of status updates had built up un-processed in the background.   

I can’t believe that my problems are in any way common, but I can’t see anything that I’ve done to cause them. Sorry for the long post, but I guess better too much info than too little.
 

So what might it be?  Log files attached.  Can you help, please?

hardware_detection-63705196203.txt

embyserver-63705225600.txt

embyserver.txt

embyserver-63705312000.txt

Edited October 2, 2019 by knowenoughtobedangerous

[Luke 37046]

 

 

1. Responses also stagnate when the Convert Media task is running.  I understand that this is something to do with Download/Sync tasks and, yes, I have some of these.  But they are only of music files so nothing too challenging to convert even if conversion were needed.  I have defined the download quality as ‘original’, ie. “Downloads original files as-is”, so I can see no reason why there is any need to do any media conversion.

    

2. Nevertheless, the Convert Media task runs each night and, as for the Scans, it seems to be a lottery how long it takes.  One night it may only be a very few minutes but the following night it might take over 11 hours (according to ‘Scheduled Tasks’).  That is a mystery in itself because the task is supposedly limited to 10 hours.

 

Hi, Convert Media runs because you created conversion jobs, or download jobs that require conversion. Were you aware of that?

[knowenoughtobedangerous 6]

Hi Luke, Yes, as I said in point 1 that you quoted.  I just can't see what I've created that requires conversion.  And even if I have, why the variation in run times when nothing (AFAIK) has changed? And why so slow?

I've set a start time of midnight and a time limit of 10 hours on the task in an attempt to avoid it running when I'm likely to want to use it,  Even after 10 hours, it might only be, say, 10 or 20% complete.

 

Library scan is currently set to run at 23.00.   

 

regards

 

Ian 

[Luke 37046]

Did you check the download and sync section in the server dashboard?

 

From here you can see conversion jobs that you have created, and cancel them if you wish.

 

Please let us know if this helps. Thanks.

[knowenoughtobedangerous 6]

From here you can see conversion jobs that you have created, and cancel them if you wish.

 

Hi Luke

 

I don't have any conversion jobs.  Did you mean sync jobs?

[Luke 37046]

Yes, that same screen displays them both together, although for the next release they will be separated.

[knowenoughtobedangerous 6]

I deleted all sync jobs and you suggested and started again with some small downloads of music.   All went sweetly to begin with: no apparent performance impact; library scans completing in six or seven minutes and the nighly media converion job completing in a minute or so.  Long enough for it to determine that it had nothing to do, I guess.

 

However, once I introduced several larger downloads (a few thousands of tracks each), much has gone awry again. Last night's library scan took over 6 hours and the convert-media job ran for 10 hours before the timeout guillotine.  And not a single track has been downloaded in nearly 2 days.

 

This is all very frustrating.  I might have persevered but, having learned in another topic that Emby won't even use the dowloaded content as I had expected it would, there is no point in wasting my time.

 

Signing off.  Thanks for your help...... but not many.     

[Luke 37046]

This will all be improved for the next release of the server. Thanks.