Why is there no HW Acceleration for Image Extraction?

[softworkz 3335]

I'd like to a address a question here that is coming up regularly and sometimes getting into the focus of user desire as a must-have feature with apparent benefits.

 

Obviously it's an evident conception that some graphics acceleration hardware should be able to accelerate graphics related operations like image extraction.

But in this case that doesn't quite apply. The short answer to the subject question is:It wouldn't make a lot of sense.

That answer is not really convincing, so let's get into depth..

 
Most of you will have noticed that we've taken great effort in the area of hardware acceleration for real-time media conversion (transcoding), so from a technical perspective it's not that we couldn't apply that to image extraction. But it doesn't fit well because image extraction is quite a different process.

So, what's different here compared to transcoding? - Quite a few things:

• No Video Encoder Required
  In case you didn't know: In transcoding situations it's always the Encoder doing the hard part of the job. 
  Situations differ, but to name some figures, you may imagine something like from 90%<>10% to 70%<>30% for Encoder<>Decoder processing requirements.
  In case of image extraction, it's just about a few hundred still images (jpg) to be saved - as opposed to thousands of video frames having inter-frame dependencies, so no video encoding needs to be done
  No video encoder also means, that there's no 70, 80,90% percent of the process that could be hardware accelerated.
  .
• There's no hardware acceleration for encoding jpg images
  The encoding of jpg images will always be done by the CPU.
  (Intel and Nvidia have an MJPEG encoder but that would require an additional processing step that might just slow down things altogether)
  .
• Decoding could be done by Hardware
  Right. That's the only part that could in fact be performed by using hardware decoders.
  But the simple question here is: Why? What would be the benefit?
  • Getting the Job done faster?
    We still have the output jpg encoder running on CPU resources which is also limited to a single thread.
    So, that is the limiting factor actually - no matter how fast the GPU might be
    . 
    => Would hardly be faster (at least not in magnitudes that would be of interest - e.g. 5x, 10x, 100x)
    .
  • Saving Resources?
    When the GPU does the decoding, then there's less processing for the CPU, but the CPU will also have some additional load to handle:
    • Copying video stream data to GPU memory
      A CPU based decoder can read the stream from a memory-mapped file without copying
    • Copying decoded uncompressed(!) frames back to system memory
      Uncompressed video frames means huge amounts of data
      We need to copy those back to system memory in full size uncompressed, even though we just want to create rather small images from it
      .
      ​=> Would not save resources (significantly)
  • Emby High-Speed Extraction
    Quite recently we've introduced a new mechanism for image extraction. 
    This procedure requires only a minimal amount of frames to be  decoded, many of them being keyframes.
    Previously we had to decode every single frame of a video for thumbnail extraction. With an extraction interval of 10s and a framerate of 25 fps that means we had to decode 250 frames for a single thumbnail image.
    Now, we're seeking to the position, decode 0-20 frames for accuracy (from keyframe to desired position), might be about 10 frames average. 
    That means, with the new image extraction we need to decode only 4% of all video frames instead of 100%.
    
    => There isn't much work left to do for a hw decoder
    .
• Acceleration Potential
  Test's have shown that using hw acceleration with the new extraction method can achieve performance gains ranging from 1.0 (same speed) to 2.0 (double speed) only.
  But that usually at the cost of high consumption of system resources (CPU for JPG encoding)
  And that leads to the question: Why?
  There's no hurry and no need to get the extraction done a bit faster. Instead, this is meant to be a Background Operation that shouldn't interrupt your Emby Server's availability (and also not occupy your GPU resources)
  .
• Maintenance
  At the time of writing, Emby supports 8 different hardware acceleration variants each supporting a bunch of decoders all with different specifics.
  Making sure that all those are working with image extraction would be a huge effort for no practical benefit at all.