Extender performance fix

[paul777 13]

So I finally figure out why the extender performance of MB3-classic was so much worse than MB2 - the host side image resizing and caching had been removed. The protocol used by the extenders sends the images over full size and then renders the mcml on the extender resizing the image to its display area. This means that if you have high resolution posters/banners then they were being sent over at full size which gave significant network lag when rendering into folder pages. The solution (and one we had implemented back in the MB2 days) is to ensure that images are given to the mcml the right size.

 

In my case some of folders would take upwards of 20secs to enter when on wifi, this is now down to c.3secs. When on a wired connection they were taking about 4secs and are now almost instant - the experience of a wired extender if much the same as on the native media center host. On a wifi connection it still pays to turn off FanArt (backdrops) in the theme settings as they are large images and can slow down the experience somewhat.

 

My changes are in a fork here if the devs want to take a look, they are built on top of some other extender stability related changes that I have been talking to ebr about already.

https://github.com/paul-777/MediaBrowser.Classic/

 

[ebr 16169]

Interesting that you would see this much of a difference.  We are already reducing the size of images that are brought over from the server (so nothing is actually full-sized) and that is user-configurable.  What were your settings for this?

[paul777 13]

Settings are at width=350 height=500, although the saved images in the cache seem to be generally 350x525. The resized images for my movies page are at 126x189 which is the one that is slow to load. I run it in thumb view and when debugging I could see it loading probably about 40 of these images at page load time. Some pages like my main start page run at much larger size images but only have a few entries. 

The saved PNGs generally aren't that much smaller but it looks like they are transferred to the extender as bitmaps - judging from the network usage I see during page load.  In the above case a bmp would be a factor of 7 different in size which is approximately the speed up I see (and it would also remove the resizing overhead from the extender although on an xbox that shouldn't be an issue). What I do know is that I have a happy wife

(It is worth mentioning that my image load times from disk on the media center host are negligible as the cache is on an SSD)

[ebr 16169]

Assuming you are talking about the right settings, that is not a width and a height.  It is the max width for Primary images vs. Thumb Images.  Reducing the first number to 130 should have had the same effect as the caching.

 

I removed the specific image size caching because it was unecessary for normal (non-extender) operation and allowed us to not use so much disc space.

[paul777 13]

Sorry, yes I wasn't reading the screen properly that is max primary width=300, max thumb width=500

 

Reducing that setting would in theory work for that folder, but then the images on other pages such as my start page where I have them set much larger would look nasty, as would some of the other folders where I run large images in cover flow mode. Larger images are ok for performance when you are using them larger images on the display as there are obviously fewer displayed on the seen at any give time. It may be that it is sensible to make the caching an option - as you say it uses more disk space, or even trigger it only for when we are using an extender, or both.

[paul777 13]

I have made it so it only affects extenders by default, it seems sensible to have this on all for extenders given the performance improvement.

Also people probably won't adjust the max width settings and doing so would adversely affect the ability to set the size of posters for each view. It is also a (currently hidden) option in the config to enable/disable it by user as I found that on older graphics cards (such as my Radeon x300) it helps with the main media center host as well. 

 

I have also implemented some cache cleaning of images not used for 30days to prevent excessive growth.

 

(Something I noticed it that if your MB3 server changes IP then all the images get re-cached as the IP is part of the source path that gets hashed)

[ebr 16169]

Image cache cleaning is already implemented in the app...

[paul777 13]

Whereabouts? The only function on the ImageCache is ClearCache(string id) that cleans out the images for a given item when that item asks it to. But there are many possibilities for creating orphan images in the cache. A change on the server producing a different image url/id. Images of items that no longer exist. Or the old images if someone changes the max width setting as the setting forms part of the url/id going into the cache.

[ebr 16169]

In the Kernel.  The function actually resides in a helper class that is used for all file clean up (log files too):

                Async.Queue("Image Cache cleanup", () =>
                                             {
                                                 foreach (var directory in Directory.GetDirectories(ApplicationPaths.AppImagePath))
                                                 {
                                                     CacheCleanUp.DeleteCacheFilesFromDirectory(directory, DateTime.UtcNow.AddDays(-(Instance.CommonConfigData.CacheFileRetentionDays)));
                                                 }
                                             });

[paul777 13]

Ah, ok, in the interests on encapsulation it would seem nice to move that to a function on the ImageCache, but small point. Is there anything at the moment touching the files modified date when we use a file? Otherwise files will be being deleted every x days even if used regularly.

[ebr 16169]

No, there is no reliable way to determine last access (even though such a property exists in Windows it is not updated on pure access).  So, they are just cleaned out on a 45 day rolling basis.  Its been working fine for around six or seven months now.

[paul777 13]

The way I would suggest doing it is the touch the modified date when we call GetImagePath to keep the ones we use active. As you say the last access time is unreliable as it is access by anything including antivirus, backup etc.

[ebr 16169]

Actually, in my experience, it is never updated by anything.

 

I don't really want to add another layer of file I/O to a process like image loading.  I've been trying to eliminate as much caching as possible instead of adding more because we have so many layers of it already (the server, http, then mbc and now with multiple sized versions).

[ebr 16169]

Is this new caching stuff already included in your current pull request?  If not, I guess you can add it or submit another one.  I'm probably going to pop out a dev version of MBC just to expose a couple properties some theme devs needed and then I can look at pulling in all your changes.

[paul777 13]

I've just made a new commit to my fork to move the old cache cleaning into the ImageCache class so it is more obvious it is there. I've also left in my touching of the modified dates - it is just a one liner inside the GetImagePath calls and is inconsequential to performance.

 

I'll submit a new pull request (don't know whether than will replace the old with a new one containing everything or just do a new one from the point of the old one, either way hopefully you can see the details of my fork if you want to review each commit at a time). I'll be offline next week so it will give you some time without me bombarding you!

 

Overall I think it is now actually faster than MB2 used to be which is great, having a more structured server behind it helps a lot.

[paul777 13]

Actually it looks like my new commits get added to the pull request as I go (which is kind of scary in some ways). I have made a branch at this point (which I won't touch) and submitted a new pull request from there which should mean you have a stable request to look at I think.