Transcoding doesnt keep up

[ServerGuy 24]

I have a separate thread relating to playback of a 4k file where the current workaround is to force transcoding. Using playback correction lets the video play, however it now seems that transcoding cant keep up. It doesnt seem hardware related but Ill list specs to give you an idea of what I'm working with. I also checked system performance metrics during transcoding and I dont see any red flags saying that my hardware is overtaxed. During playback I essentially have to "buffer" every 15 seconds or so, I can see the transcoding buffer tick away through the dashboard and when i hit around 4 seconds of buffer left, my playback goes to a loading bar. I sometimes pause it to let it buffer to the 2 minute max, but playback eats away at it until i go to loading again. Any help is appreciated

 

System Specs:

PowerEdge R720XD

2x - Intel® Xeon® CPU E5-2650 v2 @ 2.60GHz, 2600 Mhz, 8 Core(s), 16 Logical Processor(s)

OS/Emby Server Files/Cache/Transcoding Volume - RAID 1 - 1TB SSD 

Media Volume - RAID 6 - 12 Disk - 36TB 

64GB Memory

 

 

Performance during transcoding:

All volumes show less than 1% active time and sub 0.10 queue length

CPU shows less than ~13% utilization (ffmpeg is using the displayed ~13%)

Less than 8GB of total system memory utilization

 

 

 

[Luke 42080]

Hi there, let's look at an example. Please attach the information requested in how to report a media playback issue. thanks.

[Happy2Play 9782]

Can you post a ffmpeg log.  As it could be buffering do to bitrate instead of actual processing.  If it is the same media you pm'd, it has a h265 bitrate of 69620422, so that usually gets doubled when converting to h264, depending on quality settings.  So you could be trying to play a item with a bitrate of almost 140Mbps.

[ServerGuy 24]

Can you post a ffmpeg log.  As it could be buffering do to bitrate instead of actual processing.  If it is the same media you pm'd, it has a h265 bitrate of 69620422, so that usually gets doubled when converting to h264, depending on quality settings.  So you could be trying to play a item with a bitrate of almost 140Mbps.

 

 

Here you are!

 

During this playback it buffered 3 times, the first one was a small, less than 5 second loading screen, the second was around a 10 second buffer, the last one lasted around a minute and buffered me to around 55 seconds, then began to count down as it lost traction.

Transcoding4k.txt

Edited November 3, 2019 by ServerGuy

[Happy2Play 9782]

Looking at the fps the processor is not keeping up with playback.

elapsed=00:00:14.36 frame=  277 fps= 19 q=28.0 size=    5497kB time=00:13:54.31 bitrate=  54.0kbits/s throttle=off speed=58.1x 

But I do not get how the speed is being processed here as you are not processing at 58x normal playback.

 

Also you are converting a item to a maxbirate of 110Mbs.  No device really supports that bitrate as I don't believe any tv have gigabit ethernet,

[ServerGuy 24]

Am I hitting some upper limit on ffmpeg? Processor never goes over a total of 15% usage

[Happy2Play 9782]

Am I hitting some upper limit on ffmpeg? Processor never goes over a total of 15% usage

 

@@Luke or @@softworkz will have to comment on that.

[Luke 42080]

What is your G: drive?

[ServerGuy 24]

12 disk, NLSAS, 7200 RPM, RAID 6 running thru a PERC H710P Mini.

 

All firmware and drivers are up to date, Windows Server 2016

[softworkz 5070]

elapsed=00:00:14.36 frame=  277 fps= 19 q=28.0 size=    5497kB time=00:13:54.31 bitrate=  54.0kbits/s throttle=off speed=58.1x 

But I do not get how the speed is being processed here as you are not processing at 58x normal playback.

 

There's a bug in ffmpeg calculating the speed value in case when transcoding does not start from the beginning (here: " -ss 00:13:42.000")

[softworkz 5070]

@@ServerGuy - This source video has a rate of 70 Mbit/s. Same as @@Luke, my primary suspicion would be source file reading performance.

[ServerGuy 24]

@@ServerGuy - This source video has a rate of 70 Mbit/s. Same as @@Luke, my primary suspicion would be source file reading performance.

 

My current read performance for the G drive is below. No other streams were playing at the time and the server was pretty idle. I know i have a bottleneck of my controller and am currently looking at an H730P, but I wouldnt think a single transcoding would saturate the server. 

 

 

 

I also ran this against the SSD RAID 1 and got similar numbers, however the SSD write performed at 311 MB/s.

 

The transcoding logs you have are after i changed the transcoding location from the RAID6 (G) to the RAID1 SSDs ©. I did see a small improvement in disk active times and queue length after doing so, but only a marginal amount. (With transcoding on G, active time was around 1.2% and disk queue was around 1, normally below. With transcoding on C, it improved to the numbers i gave in my initial post) With transcoding in either location, I still see the issue.

[softworkz 5070]

Could you please post a transcoding log with the video file located on the C: drive (= local SSD if I understood correctly) and playback from the start (not resume)?

[ServerGuy 24]

I created a temporary library and copied the file (which copied at the below screenshot rate) to the C drive (RAID1 SSD) and began playback, because I had to use playback correction, i wasnt sure if this started FFMPEG from the beginning of the file or not so I included all logs from that point in time. It takes 2 instances of selecting playback correction for playback to work correctly. Sadly I had the same result, started around 10 seconds of buffer and ate away to 5 seconds where I got a loading screen for it to buffer again for a good while.

 

 

ffmpeg-transcode-17e978ae-d9a8-4e4e-a981-e1ce3211c7d3_1.txt

ffmpeg-remux-dc891cab-fbc3-4f4c-9c87-a7c41b77042e_1.txt

ffmpeg-remux-d91859ab-e64f-41b4-9236-a626cfa78c57_1.txt

ffmpeg-remux-afdf1d64-6541-4d5d-876d-aff915e7545a_1.txt

[softworkz 5070]

Thanks for the logs!

 

You are right. It's not a matter of disk IO.

 

Could you please open a command prompt and navigate to:

C:\Users\Administrator\AppData\Roaming\Emby-Server\system\

 

Then try the following command lines:

 

CMD1

ffmpeg.exe -f matroska -i file:"C:\TempLibrary\Harry Potter and the Prisoner of Azkaban (2004) - 4k.mkv" -threads 0 -map 0:0 -map 0:1 -sn -c:v:0 libx264 -pix_fmt yuv420p  -maxrate 109616000 -bufsize 219232000 -preset veryfast -profile:v:0 high -level:v:0 4.1 -crf 23 -x264opts:0 subme=0:me_range=4:rc_lookahead=10:me=dia:no_chroma_me:8x8dct=0:partitions=none -g:v:0 72 -keyint_min:v:0 72 -sc_threshold:v:0 0  -copyts -vsync -1 -codec:a:0 ac3 -metadata:s:a:0 language=eng -disposition:a:0 default -ac:a:0 6 -ab:a:0 384000 -ar:a:0 48000  -f segment -max_delay 5000000 -avoid_negative_ts disabled -map_metadata -1 -map_chapters -1 -start_at_zero -segment_time 3  -individual_header_trailer 0 -segment_format mpegts -segment_write_temp 1 -segment_list_type m3u8 -segment_start_number 0 -segment_list "C:\Transcoding\transcoding-temp\efa1cf782b41490d871227915afb5148.m3u8" -y "C:\Transcoding\transcoding-temp\efa1cf782b41490d871227915afb5148%d.ts"

CMD2

ffmpeg.exe -f matroska -i file:"C:\TempLibrary\Harry Potter and the Prisoner of Azkaban (2004) - 4k.mkv" -threads 0 -map 0:0 -map 0:1 -sn -c:v:0 libx264 -pix_fmt yuv420p  -maxrate 10961600 -bufsize 21923200 -preset veryfast -profile:v:0 high -level:v:0 4.1 -crf 23 -x264opts:0 subme=0:me_range=4:rc_lookahead=10:me=dia:no_chroma_me:8x8dct=0:partitions=none -g:v:0 72 -keyint_min:v:0 72 -sc_threshold:v:0 0  -copyts -vsync -1 -codec:a:0 ac3 -metadata:s:a:0 language=eng -disposition:a:0 default -ac:a:0 6 -ab:a:0 384000 -ar:a:0 48000  -f segment -max_delay 5000000 -avoid_negative_ts disabled -map_metadata -1 -map_chapters -1 -start_at_zero -segment_time 3  -individual_header_trailer 0 -segment_format mpegts -segment_write_temp 1 -segment_list_type m3u8 -segment_start_number 0 -segment_list "C:\Transcoding\transcoding-temp\efa1cf782b41490d871227915afb5148.m3u8" -y "C:\Transcoding\transcoding-temp\efa1cf782b41490d871227915afb5148%d.ts"

CMD3

ffmpeg.exe -f matroska -i file:"C:\TempLibrary\Harry Potter and the Prisoner of Azkaban (2004) - 4k.mkv" -map 0:0 -map 0:1 -sn -c:v:0 libx264 -pix_fmt yuv420p  -maxrate 10961600 -bufsize 21923200 -preset veryfast -profile:v:0 high -level:v:0 4.1 -crf 23 -g:v:0 72 -keyint_min:v:0 72 -sc_threshold:v:0 0 -copyts -vsync -1 -an -f segment -max_delay 5000000 -avoid_negative_ts disabled -map_metadata -1 -map_chapters -1 -start_at_zero -segment_time 3  -individual_header_trailer 0 -segment_format mpegts -segment_write_temp 1 -segment_list_type m3u8 -segment_start_number 0 -segment_list "C:\Transcoding\transcoding-temp\efa1cf782b41490d871227915afb5148.m3u8" -y "C:\Transcoding\transcoding-temp\efa1cf782b41490d871227915afb5148%d.ts"

take note of the fps= and speed= values in each case
 

Edited November 4, 2019 by softworkz

[ServerGuy 24]

Here are logs for each command, let me know if you need anything else.

 

Thanks!

CMD1.txt

CMD2.txt

CMD3.txt

[softworkz 5070]

One last round:

CMD4

 ffmpeg.exe -f matroska -i file:"C:\TempLibrary\Harry Potter and the Prisoner of Azkaban (2004) - 4k.mkv" -threads 6 -map 0:0 -map 0:1 -sn -c:v:0 libx264 -pix_fmt yuv420p  -maxrate 10961600 -bufsize 21923200 -preset veryfast -profile:v:0 high -level:v:0 5 -crf 23 -x264opts:0 subme=0:me_range=4:rc_lookahead=10:me=dia:no_chroma_me:8x8dct=0:partitions=none -g:v:0 72 -keyint_min:v:0 72 -sc_threshold:v:0 0  -copyts -vsync -1 -codec:a:0 ac3 -metadata:s:a:0 language=eng -disposition:a:0 default -ac:a:0 6 -ab:a:0 384000 -ar:a:0 48000  -f segment -max_delay 5000000 -avoid_negative_ts disabled -map_metadata -1 -map_chapters -1 -start_at_zero -segment_time 3  -individual_header_trailer 0 -segment_format mpegts -segment_write_temp 1 -segment_list_type m3u8 -segment_start_number 0 -segment_list "C:\Transcoding\transcoding-temp\efa1cf782b41490d871227915afb5148.m3u8" -y "C:\Transcoding\transcoding-temp\efa1cf782b41490d871227915afb5148%d.ts"

CMD5

 ffmpeg.exe -f matroska -i file:"C:\TempLibrary\Harry Potter and the Prisoner of Azkaban (2004) - 4k.mkv" -threads 0 -map 0:0 -map 0:1 -sn -c:v:0 libx264 -pix_fmt yuv420p  -maxrate 1096160 -bufsize 2192320 -preset veryfast -profile:v:0 high -level:v:0 4.1 -crf 23 -x264opts:0 subme=0:me_range=4:rc_lookahead=10:me=dia:no_chroma_me:8x8dct=0:partitions=none -g:v:0 72 -keyint_min:v:0 72 -sc_threshold:v:0 0  -copyts -vsync -1 -codec:a:0 ac3 -metadata:s:a:0 language=eng -disposition:a:0 default -ac:a:0 6 -ab:a:0 384000 -ar:a:0 48000  -f segment -max_delay 5000000 -avoid_negative_ts disabled -map_metadata -1 -map_chapters -1 -start_at_zero -segment_time 3  -individual_header_trailer 0 -segment_format mpegts -segment_write_temp 1 -segment_list_type m3u8 -segment_start_number 0 -segment_list "C:\Transcoding\transcoding-temp\efa1cf782b41490d871227915afb5148.m3u8" -y "C:\Transcoding\transcoding-temp\efa1cf782b41490d871227915afb5148%d.ts"

Edited November 4, 2019 by softworkz

[ServerGuy 24]

 

One last round:

CMD4

 ffmpeg.exe -f matroska -i file:"C:\TempLibrary\Harry Potter and the Prisoner of Azkaban (2004) - 4k.mkv" -threads 6 -map 0:0 -map 0:1 -sn -c:v:0 libx264 -pix_fmt yuv420p  -maxrate 10961600 -bufsize 21923200 -preset veryfast -profile:v:0 high -level:v:0 5 -crf 23 -x264opts:0 subme=0:me_range=4:rc_lookahead=10:me=dia:no_chroma_me:8x8dct=0:partitions=none -g:v:0 72 -keyint_min:v:0 72 -sc_threshold:v:0 0  -copyts -vsync -1 -codec:a:0 ac3 -metadata:s:a:0 language=eng -disposition:a:0 default -ac:a:0 6 -ab:a:0 384000 -ar:a:0 48000  -f segment -max_delay 5000000 -avoid_negative_ts disabled -map_metadata -1 -map_chapters -1 -start_at_zero -segment_time 3  -individual_header_trailer 0 -segment_format mpegts -segment_write_temp 1 -segment_list_type m3u8 -segment_start_number 0 -segment_list "C:\Transcoding\transcoding-temp\efa1cf782b41490d871227915afb5148.m3u8" -y "C:\Transcoding\transcoding-temp\efa1cf782b41490d871227915afb5148%d.ts"

CMD5

 ffmpeg.exe -f matroska -i file:"C:\TempLibrary\Harry Potter and the Prisoner of Azkaban (2004) - 4k.mkv" -threads 0 -map 0:0 -map 0:1 -sn -c:v:0 libx264 -pix_fmt yuv420p  -maxrate 1096160 -bufsize 2192320 -preset veryfast -profile:v:0 high -level:v:0 4.1 -crf 23 -x264opts:0 subme=0:me_range=4:rc_lookahead=10:me=dia:no_chroma_me:8x8dct=0:partitions=none -g:v:0 72 -keyint_min:v:0 72 -sc_threshold:v:0 0  -copyts -vsync -1 -codec:a:0 ac3 -metadata:s:a:0 language=eng -disposition:a:0 default -ac:a:0 6 -ab:a:0 384000 -ar:a:0 48000  -f segment -max_delay 5000000 -avoid_negative_ts disabled -map_metadata -1 -map_chapters -1 -start_at_zero -segment_time 3  -individual_header_trailer 0 -segment_format mpegts -segment_write_temp 1 -segment_list_type m3u8 -segment_start_number 0 -segment_list "C:\Transcoding\transcoding-temp\efa1cf782b41490d871227915afb5148.m3u8" -y "C:\Transcoding\transcoding-temp\efa1cf782b41490d871227915afb5148%d.ts"

 

 

Attached

CMD4.txt

CMD5.txt

[softworkz 5070]

Thanks again. With CMD4 it was possible to achieve a transcoding speed that should be sufficient.

But it's just a moderate improvement. It seems that a significant amount of processing is already required for the HEVC decoding, because even minimal encoder bitrate settings didn't make this run faster.

Though, we must not forget that 4k processing involves moving around and processing 4 times the data compared to full hd. What adds to this is color conversion from 10bit to 8bit.

 

What kind of CPU do you have?

[ServerGuy 24]

2 - Intel® Xeon® CPU E5-2650 v2 @ 2.60GHz, 2600 Mhz, 8 Core(s), 16 Logical Processor(s) Each

Edited November 4, 2019 by ServerGuy

[softworkz 5070]

Not everything in ffmpeg can run multi-threaded.

The bottleneck is probably either the HEVC decoder or the color conversion where you are hitting the limit of single-core performance of your CPU which is rather let's say - moderate.

 

In case of CMD4, there were two major changes:

• Set explicit thread count
  You can try to set this to 6 or 8 in the Emby transcoding settings (instead of auto)
  .
• Reduced bitrate
  You'll need to set this in the Roku settings.
  Start with something very low like 1 Mbps

Hope that will get you above 1.0x speed..