If you wanna skip all my steps, you can simply do this. From a terminal, run the lsof command and pipe it through grep to narrow the results.
$ sudo lsof|grep pipe
$ sudo lsof|grep FIFO
Remember to run this as sudo so you get all users (there's probably a flag for all users but I don't know it). The results may take a little bit to run and there will be a lot of files listed. What was important for me was the egregious number of Bitbucket nodes that displayed on this list. There were so many, my terminal's buffer was exceeded.
Shutting off the offending application increased my server's response time immediately and immensely.
If you still want to know my 'not so precise' method of troubleshooting (and probably not accurate), here is what I did.
I first ran the ps command to get the Emby pid.
$ ps -ef |grep emby
You should get two pids, one for mono-sgen and another for emby-server. You can get either, it doesn't matter. You can also run pstree to see the child processes.
$ pstree -p 1344
├ . . .
Now run strace on the pid.
$ sudo strace -p1344 -f -etrace=read
The p flag is for the Emby pid, the f flag is to list all child processes, and the e flag with trace expression is to list only system read calls. Optionally, you can output the results to a file using the o flag.
I didn't understand anything I was looking at except one line looked funny. Here is some sample output:
[pid 22459] read(3, "\263\246!J", 4) = 4
[pid 22459] read(3, "\205\6\2523", 4) = 4
[pid 1406] read(20, <unfinished ...>
[pid 22535] read(3, <unfinished ...>
[pid 1406] <... read resumed> "c", 128) = 1
[pid 22535] <... read resumed> "L\342'\252", 4) = 4
[pid 1406] read(20, 0x7f73636fec60, 128) = -1 EAGAIN (Resource temporarily unavailable)
[pid 22459] read(3, "-\274(\250", 4) = 4
[pid 22535] read(3, "\261\306\22\23", 4) = 4
[pid 22459] read(3, "h\271\325\335", 4) = 4
[pid 22535] read(3, "R`\370\325", 4) = 4
For pid 1406, the read system call for file descriptor 20 always seems to have a status of EAGAIN (Resource temporarily unavailable). I looked it up and supposedly it isn't really an error, just that no data has been received yet. But since it kept coming up and without a changing buffer position (the second parameter), I suspect something was wrong.
With the lsof tool, we can get more information on a file from it's file descriptor. The file descriptor is the 4th column of lsof output. Here's a good article on using strace and lsof: http://mharrytemp.bl...track-down.html.
Anyway, I used lsof on that file descriptor to find the file was pipe and the type of FIFO.
$ sudo lsof -d 20
The output from this command produced some Emby nodes but a ton of Bitbucket nodes; ultimately leading me to believe that Bitbucket was the offending spawner.
Hopefully you'll find something out of these tests to find what is slowing your Emby installation down.