In Transcoding > Advanced  Try changing around the priority of the decoders - ie QuickSync and DX11VA