Jump to content


Photo

very slow loading of nextup & latest items

nextup latest items slow loading

  • Please log in to reply
349 replies to this topic

#41 curtisghanson OFFLINE  

curtisghanson

    Member

  • Members
  • 17 posts
  • Local time: 06:32 PM
  • LocationPortland, Oregon, USA

Posted 13 April 2016 - 09:35 PM

@swhitmore and @daedalus. Have you guys solved your slowness issues? I'm gonna throw this out there whether it's useful or not.

 

I was experiencing extreme slowness on my server as well and it seemed to only affect Emby. I run Plex, Subsonic, and several other media management applications on my server as well; all of which were not slowed. I ran strace on the Emby processes and found that threads were being held up in a FIFO pipe. When I lsof'd the pipe, I found that another application I installed as a test was creating a great deal of file handles and choking the HttpResponse from Emby. The culprit was my self-hosted Bitbucket server application. After shutting it down, I no longer have any slowness. I'm not saying that your problem is Bitbucket, but maybe another application that's creating a lot of handles and since it's FIFO, your Emby requests are constantly waiting in line to be processed.

$ ps -ef |grep emby
$ sudo strace -p <emby pid> -f -e trace=read
$ sudo lsof|grep <pipe or FIFO or 'nodeid from strace'>

Edit for spelling.


Edited by curtisghanson, 13 April 2016 - 09:37 PM.

  • swhitmore likes this

#42 swhitmore OFFLINE  

swhitmore

    Advanced Member

  • Members
  • 2686 posts
  • Local time: 09:32 AM
  • LocationPerth, Australia

Posted 13 April 2016 - 10:22 PM

Thanks Curtis. How do I run this scan?



#43 Luke OFFLINE  

Luke

    System Architect

  • Administrators
  • 139713 posts
  • Local time: 08:32 PM

Posted 13 April 2016 - 10:58 PM

in the web client go to home -> upcoming. then stay on that page and restart the server. then once the server is back up, refresh the page. is that page slow?



#44 curtisghanson OFFLINE  

curtisghanson

    Member

  • Members
  • 17 posts
  • Local time: 06:32 PM
  • LocationPortland, Oregon, USA

Posted 13 April 2016 - 11:45 PM

I'm on my way home, I'll shoot you my steps as soon as I can.

#45 curtisghanson OFFLINE  

curtisghanson

    Member

  • Members
  • 17 posts
  • Local time: 06:32 PM
  • LocationPortland, Oregon, USA

Posted 14 April 2016 - 02:07 AM

If you wanna skip all my steps, you can simply do this. From a terminal, run the lsof command and pipe it through grep to narrow the results.

$ sudo lsof|grep pipe

or

$ sudo lsof|grep FIFO

Remember to run this as sudo so you get all users (there's probably a flag for all users but I don't know it). The results may take a little bit to run and there will be a lot of files listed. What was important for me was the egregious number of Bitbucket nodes that displayed on this list. There were so many, my terminal's buffer was exceeded.

 

Shutting off the offending application increased my server's response time immediately and immensely.

 

If you still want to know my 'not so precise' method of troubleshooting (and probably not accurate), here is what I did.

 

I first ran the ps command to get the Emby pid.

$ ps -ef |grep emby

You should get two pids, one for mono-sgen and another for emby-server. You can get either, it doesn't matter. You can also run pstree to see the child processes.

$ pstree -p 1344
mono-sgen(1344)─┬─{mono-sgen}(1360)
                ├─{mono-sgen}(1361)
                ├ . . .
                └─{mono-sgen}(22376)

Now run strace on the pid.

$ sudo strace -p1344 -f -etrace=read

The p flag is for the Emby pid, the f flag is to list all child processes, and the e flag with trace expression is to list only system read calls. Optionally, you can output the results to a file using the o flag.

 

I didn't understand anything I was looking at except one line looked funny. Here is some sample output:

[pid 22459] read(3, "\263\246!J", 4)    = 4
[pid 22459] read(3, "\205\6\2523", 4)   = 4
[pid  1406] read(20,  <unfinished ...>
[pid 22535] read(3,  <unfinished ...>
[pid  1406] <... read resumed> "c", 128) = 1
[pid 22535] <... read resumed> "L\342'\252", 4) = 4
[pid  1406] read(20, 0x7f73636fec60, 128) = -1 EAGAIN (Resource temporarily unavailable)
[pid 22459] read(3, "-\274(\250", 4)    = 4
[pid 22535] read(3, "\261\306\22\23", 4) = 4
[pid 22459] read(3, "h\271\325\335", 4) = 4
[pid 22535] read(3, "R`\370\325", 4)    = 4
For pid 1406, the read system call for file descriptor 20 always seems to have a status of EAGAIN (Resource temporarily unavailable). I looked it up and supposedly it isn't really an error, just that no data has been received yet. But since it kept coming up and without a changing buffer position (the second parameter), I suspect something was wrong.

 

With the lsof tool, we can get more information on a file from it's file descriptor. The file descriptor is the 4th column of lsof output. Here's a good article on using strace and lsof: http://mharrytemp.bl...track-down.html.

 

Anyway, I used lsof on that file descriptor to find the file was pipe and the type of FIFO.

$ sudo lsof -d 20

The output from this command produced some Emby nodes but a ton of Bitbucket nodes; ultimately leading me to believe that Bitbucket was the offending spawner.

 

 

Hopefully you'll find something out of these tests to find what is slowing your Emby installation down. 



#46 bfir3 OFFLINE  

bfir3

    Advanced Member

  • Members
  • 412 posts
  • Local time: 02:32 AM

Posted 21 April 2016 - 01:07 PM

I seem to be having a similar issue. Very long load times for Next Up and other library views when accessing them (on local network and remote). Here's a part from my logs, we can see many slow responses: https://paste2.org/hjO05tHU



#47 daedalus OFFLINE  

daedalus

    Advanced Member

  • Members
  • 1175 posts
  • Local time: 02:32 AM

Posted 21 April 2016 - 06:11 PM

little update

the "not restarting workaround" seems to have lost it's magic

i'm back to a 8/10 slow loadings

just discovered another slowness the seriespicker in the automatic organisation takes very long to show all the series and this happens every time even if i just picked another series for another file



#48 bfir3 OFFLINE  

bfir3

    Advanced Member

  • Members
  • 412 posts
  • Local time: 02:32 AM

Posted 21 April 2016 - 07:21 PM

I had been cruising on 3.0.5821 for so long before updating. Thought I'd see some performance improvements with the database upgrades, and since I was so behind I decided why not (I did keep a system backup from before the upgrade), but I'm only seeing the reverse effect so far, lol. I can provide other logs as needed to figure this out. Maybe if I leave the server running for a few more days it will settle down.



#49 curtisghanson OFFLINE  

curtisghanson

    Member

  • Members
  • 17 posts
  • Local time: 06:32 PM
  • LocationPortland, Oregon, USA

Posted 21 April 2016 - 08:05 PM

My troubleshooting was only temporary as well...

 

Based on other research I've done, so far, this only seems to be an I/O wait problem regarding HTTPServer Responses. I'm running Emby with the trace flag on the namespace of the RequestHandler

--trace=N:MediaBrowser.Server.Implementations.HttpServer

Watching the output to see if anything comes up. So far, the only thing I've seen that raises any alarms is the following:

[0x7f27ab5fa700:] EXCEPTION handling: System.IO.IOException: EndRead failure

but I don't know what the means.

 

Maybe someone knows a better place I should be tracing? Maybe tracing on the method itself?

--trace=M:Task:RequestHandler

Just out of curiosity, what are everyone's specs of their servers running Emby? Maybe it isn't I/O on the disk and maybe it's something else like my CPU.

 

CPU: AMD FX-8370 8 Core AM3+ 4300Mhz 125W 16MB

Mem: Ripjaws X 16GB DDR3 SDRAM 1600MHz

HDD: WD Black 1TB 7200RPM Sata 6GB/s 64MB Cache

 

My next attempt will be to compile Emby myself and run it modifying the Http Response. See this thread for more information:

 

http://stackoverflow...-extremely-slow

 

It's old, but maybe still relevant?

 



#50 swhitmore OFFLINE  

swhitmore

    Advanced Member

  • Members
  • 2686 posts
  • Local time: 09:32 AM
  • LocationPerth, Australia

Posted 21 April 2016 - 08:21 PM

little update
the "not restarting workaround" seems to have lost it's magic
i'm back to a 8/10 slow loadings
just discovered another slowness the seriespicker in the automatic organisation takes very long to show all the series and this happens every time even if i just picked another series for another file


I just did a quick check, and it seems mine is still ok without restarting, but I'm going to test further when I can. What version are you on now?
 

in the web client go to home -> upcoming. then stay on that page and restart the server. then once the server is back up, refresh the page. is that page slow?


Sorry Luke, I've been away for work, so haven't tested this yet. I'll try asap.

Edited by swhitmore, 21 April 2016 - 08:23 PM.


#51 bfir3 OFFLINE  

bfir3

    Advanced Member

  • Members
  • 412 posts
  • Local time: 02:32 AM

Posted 22 April 2016 - 12:03 AM

Just out of curiosity, what are everyone's specs of their servers running Emby? Maybe it isn't I/O on the disk and maybe it's something else like my CPU.

 

CPU: AMD FX-8370 8 Core AM3+ 4300Mhz 125W 16MB

Mem: Ripjaws X 16GB DDR3 SDRAM 1600MHz

HDD: WD Black 1TB 7200RPM Sata 6GB/s 64MB Cache

 

My next attempt will be to compile Emby myself and run it modifying the Http Response. See this thread for more information:

 

http://stackoverflow...-extremely-slow

 

It's old, but maybe still relevant?

 

My thoughts were possibly I/O related as well. I'm running a similar machine with less RAM:

 

 

CPU: AMD FX-8370 8 Core AM3+ 4300Mhz 125W 16MB
Mem: Kingston 8GB DDR3 SDRAM 1600MHz
HDD: Like 10+ HDDs and 1 SSD
 
I just wonder why it differs so vastly between calls. Sometimes the response is nearly instant, other times it can take over 20 seconds...what the hell?

Edited by bfir3, 22 April 2016 - 02:00 PM.


#52 daedalus OFFLINE  

daedalus

    Advanced Member

  • Members
  • 1175 posts
  • Local time: 02:32 AM

Posted 22 April 2016 - 06:40 AM

in the web client go to home -> upcoming. then stay on that page and restart the server. then once the server is back up, refresh the page. is that page slow?

 

tested this no difference

 

I just did a quick check, and it seems mine is still ok without restarting, but I'm going to test further when I can. What version are you on now?

 

was on 3.0.5940.0

 

now its 3.0.5942.0 cause i've tested the above thing and had to restart the machine anyways

so i've to wait a couple days to see if it works again


Edited by daedalus, 22 April 2016 - 08:55 AM.


#53 ebr OFFLINE  

ebr

    Chief Bottle Washer

  • Administrators
  • 47409 posts
  • Local time: 08:32 PM

Posted 22 April 2016 - 10:07 AM

Luke asked if the page was slow and you responded with:

 

 

tested this no difference

 

Does that mean it was slow or not...?



#54 daedalus OFFLINE  

daedalus

    Advanced Member

  • Members
  • 1175 posts
  • Local time: 02:32 AM

Posted 22 April 2016 - 10:17 AM

i meant its as fast as before so no it's not slow


Edited by daedalus, 22 April 2016 - 10:34 AM.


#55 Luke OFFLINE  

Luke

    System Architect

  • Administrators
  • 139713 posts
  • Local time: 08:32 PM

Posted 22 April 2016 - 11:44 AM

Well first you said no difference, then you said "no it's not slow". Can you possibly be more specific?



#56 bfir3 OFFLINE  

bfir3

    Advanced Member

  • Members
  • 412 posts
  • Local time: 02:32 AM

Posted 22 April 2016 - 02:04 PM

Here's some of my log from when I was browsing this morning. This is just from the morning, and there were multiple instances of 20s+ of loading. There's a lot of 2-5 second loading as well, and although I think loads shouldn't take much longer than 1 second, that is at least a lot more acceptable than 20s+.

 

https://paste2.org/8Vtv1Xtt


Edited by bfir3, 22 April 2016 - 02:04 PM.


#57 daedalus OFFLINE  

daedalus

    Advanced Member

  • Members
  • 1175 posts
  • Local time: 02:32 AM

Posted 22 April 2016 - 03:36 PM

Well first you said no difference, then you said "no it's not slow". Can you possibly be more specific?

 

you asked for the reloading of the upcoming page after a restart and this was not slow as i tested it

 

--edit--

right now it seems to be pretty fast no comparison to the first days after a restart before

but before cheering lets see what the next days bring up


Edited by daedalus, 22 April 2016 - 03:45 PM.


#58 CBers OFFLINE  

CBers

    Advanced Member

  • Moderators
  • 15023 posts
  • Local time: 01:32 AM
  • LocationKent, England.

Posted 24 April 2016 - 03:57 PM

The latest couple of releases have made the app almost unusable on my Nexus Player.

 

It's slow to load the main UI and when it does, it's very slow to navigate around.

 

Open a details page and it either takes ages to load the details, or it doesn't load anything and I am getting multiple "Volley timeout errors".

 

This is on my Nexus Player mainly, where it is TRANSCODING, but the Shield seems much better, where it Direct Streams.

 

Sometimes when trying to play something and nothing happens, I back out all the way to the device home screen and all of a sudden, the video starts to play !!

 

I will post full server logs when I get a chance, but I have uploaded diagnostic reports from within the app a few times.


Edited by CBers, 24 April 2016 - 04:04 PM.


#59 CBers OFFLINE  

CBers

    Advanced Member

  • Moderators
  • 15023 posts
  • Local time: 01:32 AM
  • LocationKent, England.

Posted 24 April 2016 - 04:09 PM

I was trying to play "The Big Bang Theory - 6x04 - The Re-Entry Minimization.mkv" (timecode in server log:20:31:51.8362), but it kept stopping.

 

Remux log1

 

Remux log2

 

Server log


Edited by CBers, 24 April 2016 - 04:11 PM.


#60 CBers OFFLINE  

CBers

    Advanced Member

  • Moderators
  • 15023 posts
  • Local time: 01:32 AM
  • LocationKent, England.

Posted 24 April 2016 - 04:35 PM

Also having similar problems with the app on my Fire TV (gen1).

Does the caching of all the images a contributory factor, as when opening a library, I often get just green images, until they load, if they ever for.





Also tagged with one or more of these keywords: nextup, latest, items, slow, loading

0 user(s) are reading this topic

0 members, 0 guests, 0 anonymous users