Thumbnail loading performance problem (unusable in some cases)

[jasonb1 7]

I have a setup with some files on a the same machine as the running server and then another file server elsewhere. I noticed that loading the thumbnails (the 10 second interval bif files) was super slow or "not working" for files on the file server and it also bogged down the entire server for a while (various parts of a web app would have delays loading after up to 10-15 seconds or so). However, the bif files were on the local machine. I recently changed to save the bif files with the media, but there's no perf difference (please read on).

I finally nailed down what the problem is via the Sysinternals Process Monitor. It seems that the server will *for every single frame requested* (both on the items detail page and in a player) query the contents of entire directory the item is in. So, the performance characteristics change depending on the number of files in that directory and whether the file is on a network vs. local (network being slower).

Since it appears to query the contents of the directory for every single frame loaded, basically I get no thumbnails during playback of videos on the server when there's enough files in the directory the video is in. Even when working with local files I'm not satisfied with the load speed of the thumbnails during playback and it's the same reason just not as bad.

Doing some testing with cmd.exe and Process Monitor, I can see that it's not a limitation of Windows to query an entire directory to look for one file. For example if I do: 'dir "sample -320-10.bif"' then it makes an efficient query and not query the contents of the entire directory. Also, it seems that the server shouldn't be making this query to the file system for every single frame (perhaps via short lived limited in-memory cache). By the way, the reason the server bogs down appears to be because the requests to the server for the frames appear to queue up and must complete even though they aren't going anywhere.

 

[jasonb1 7]

BTW, I didn't include the server log in this report since the information contained isn't helpful other than it shows the slow response time upon thumbnail image requests which I described above.

[Luke 42083]

Hi there, can you please attach the emby server log anyway? Thanks.

[jasonb1 7]

Here's a log where you can see some super long times for thumbnail fetching. You can't see the why though from the log, so I recommend using Sysinternals Process Monitor, have it watch events for EmbyServer.exe and as you open a details page of a video with chapters, watch it query the entire directory over and over again. Then, play the video, as you drag (I used the web app) slider and it's trying to load those thumbnails, more and more (and more) full directory file list enumerations happen. It can't keep up and even if you stop playing the video, it's still doing all that directory enumeration and thing's don't return to normal until all that finally completes.

Thanks for looking into it and hopefully it's an easy fix to limit the full directory scanning to a smaller subset via a wildcard (or something).

embyserver-63783553865.txt

[rbjtech 5284]

Hi,

Have you checking the OS file cache is enabled on your emby drive ?

I don't see this behaviour at all.

It loads the single bif file and as you slide through the entire file it loads more of the same file as necessary - once in cache, then you can slide back to the start and no more file is loaded.  My bif files are stored with the media as you can see below - ie they not on fast local I/O.

This looks correct to me ?

...

This is the last read/closefile - after this point, playback continues and I can slide the thumbnail wherever I like, no more bif is loaded as it's in cache.

 

these are your thumbnail responses - ~13 seconds ..

2022-03-22 13:50:53.697 Info Server: http/1.1 Response 500 to 127.0.0.1. Time: 13132ms. http://localhost:8096/emby/Items/8193/Images/Thumbnail?maxWidth=400&tag=c1b20882860e2ad87fc42278d15ac2f2&PositionTicks=7000000000&quality=90
2022-03-22 13:50:53.780 Info Server: http/1.1 Response 500 to 127.0.0.1. Time: 13208ms. http://localhost:8096/emby/Items/8193/Images/Thumbnail?maxWidth=400&tag=c1b20882860e2ad87fc42278d15ac2f2&PositionTicks=7200000000&quality=90
2022-03-22 13:50:53.809 Info Server: http/1.1 Response 500 to 127.0.0.1. Time: 13230ms. http://localhost:8096/emby/Items/8193/Images/Thumbnail?maxWidth=400&tag=c1b20882860e2ad87fc42278d15ac2f2&PositionTicks=7400000000&quality=90
2022-03-22 13:50:53.818 Info Server: http/1.1 Response 500 to 127.0.0.1. Time: 13231ms. http://localhost:8096/emby/Items/8193/Images/Thumbnail?maxWidth=400&tag=c1b20882860e2ad87fc42278d15ac2f2&PositionTicks=7600000000&quality=90
2022-03-22 13:50:53.833 Info Server: http/1.1 Response 500 to 127.0.0.1. Time: 13240ms. http://localhost:8096/emby/Items/8193/Images/Thumbnail?maxWidth=400&tag=c1b20882860e2ad87fc42278d15ac2f2&PositionTicks=7800000000&quality=90

these are mine for the above sysinternal capture .. 24-25 ms ..

2022-03-23 11:32:57.863 Info Server: http/1.1 Response 200 to 192.168.20.16. Time: 25ms. http://emby.rbjtech.lan:8096/Items/664943/Images/Thumbnail?maxWidth=300&tag=239f8a2871b3c3fb73458511544d2ed6&PositionTicks=10800000000&quality=90
2022-03-23 11:32:57.897 Info Server: http/1.1 Response 200 to 192.168.20.16. Time: 24ms. http://emby.rbjtech.lan:8096/Items/664943/Images/Thumbnail?maxWidth=300&tag=239f8a2871b3c3fb73458511544d2ed6&PositionTicks=11100000000&quality=90
2022-03-23 11:32:57.946 Info Server: http/1.1 Response 200 to 192.168.20.16. Time: 24ms. http://emby.rbjtech.lan:8096/Items/664943/Images/Thumbnail?maxWidth=300&tag=239f8a2871b3c3fb73458511544d2ed6&PositionTicks=11400000000&quality=90

 

In summary, I don't think this is an Emby problem but a file system issue.  I would check you have file caching turned on as the first thing to check.

Edited March 23, 2022 by rbjtech

[jasonb1 7]

There is no "cached turned off" feature in Windows that I know of and I certainly wouldn't turn it off if I could. Your example doesn't reproduce the problem since you likely have very few files in the directory compared to me in your test case. Click on the QueryDirectory event and you'll see it listing all the files in the directory and that's only going to be noticed as a problem when you have a lot of files in a single directory and it's over a network. My layout is to have many videos (about 3600 in my repro case but multiply that by 3 due to .bif and .nfo file for each video) in directories and if the server lists all the files (and it does at least on the last version of the open source I can see on github) and if it does it for every frame to load, that will (and does) cause a serious performance problem.

Some more info: I have about 14000 files in a directory on the same machine as the server (so about 4600 videos) and it isn't so terrible at loading thumbnails (like it doesn't seriously bog down), but it is still not fast as a case with a video in a directory with few files. I have a TV show situation like that where there's only 25 files in a directory (on the network, not on the same machine as the server) and it's lightning fast loading thumbnails while seeking.

It all comes down to wasting resources unnecessarily enumerating entire directories for a situation where it shouldn't be.

Edited March 23, 2022 by jasonb1

[Luke 42083]

HI, yes we plan to rework the way these are sent in future updates. Thanks.

[jasonb1 7]

FYI, my server machine is running Server 2008 R2 which has an older version of SMB protocol and apparently SMB3 which came in with Windows 8.1 and Server 2012 has performance improvements. I'm going to try putting Windows 10* on it and see if that helps with the performance while waiting for the fix. One thing I didn't mention also is the server is using StableBit DrivePool to merge multiple drives into one virtual filesystem which may have overhead in this case of listing the contents of a directory since the content likely comes from multiple drives.

 - Jason

* I can't get Windows Server cheap anymore and it's not really necessary given it's just a home server.

[rbjtech 5284]

16 hours ago, jasonb1 said:

There is no "cached turned off" feature in Windows that I know of and I certainly wouldn't turn it off if I could. Your example doesn't reproduce the problem since you likely have very few files in the directory compared to me in your test case. Click on the QueryDirectory event and you'll see it listing all the files in the directory and that's only going to be noticed as a problem when you have a lot of files in a single directory and it's over a network. My layout is to have many videos (about 3600 in my repro case but multiply that by 3 due to .bif and .nfo file for each video) in directories and if the server lists all the files (and it does at least on the last version of the open source I can see on github) and if it does it for every frame to load, that will (and does) cause a serious performance problem.

Some more info: I have about 14000 files in a directory on the same machine as the server (so about 4600 videos) and it isn't so terrible at loading thumbnails (like it doesn't seriously bog down), but it is still not fast as a case with a video in a directory with few files. I have a TV show situation like that where there's only 25 files in a directory (on the network, not on the same machine as the server) and it's lightning fast loading thumbnails while seeking.

It all comes down to wasting resources unnecessarily enumerating entire directories for a situation where it shouldn't be.

I'm not sure why you have all the files in a single directory (not recommended for many reasons..), but I now see why this would cause an issue if the QueryDirectory was called multiple times and you have a slow file system.