If I understand what's happening, there is a lower bitrate limit, depending on source resolution, where the transcoder switches the output from 1280x720 to 720x405.
For example: Starting with say a 10 mbit 1080p video, the output resolution will stay at 720p down to 2 mbit or so. This makes sense, as anything below 2 is not really enough for 720.
But when starting with 2160p, the lower limit seems to be set around 5.75 mbit. Anything below this and the output will change to 405p. This