Jump to content


Photo

FFMPEG hung up


  • Please log in to reply
13 replies to this topic

#1 pir8radio OFFLINE  

pir8radio

    Advanced Member

  • Members
  • 2581 posts
  • Local time: 09:05 AM

Posted 04 February 2019 - 11:00 PM

I had an issue yesterday that I have not seen in a while...  It looked like two roku users's ffmpeg was hung up.   Is that still an issue with roku if they quit in some weird way?



#2 Luke OFFLINE  

Luke

    System Architect

  • Administrators
  • 119080 posts
  • Local time: 10:05 AM

Posted 05 February 2019 - 12:06 AM

Depending on how they exit the app it could take a few extra minutes to shut down, yes.



#3 speechles OFFLINE  

speechles

    Advanced Member

  • App Developer
  • 3518 posts
  • Local time: 07:05 AM

Posted 05 February 2019 - 12:15 AM

If they pause the app and then press the home button the server might not notice to kill ffmpeg. I know if the person is playing an item and hits the home button the server knows since no more progress updates are incoming to kill the ffmpeg process. But if the user pauses and then presses home, is the server smart enough to still kill ffmpeg eventually? @Luke


Edited by speechles, 05 February 2019 - 12:15 AM.


#4 ebr OFFLINE  

ebr

    Chief Bottle Washer

  • Administrators
  • 42241 posts
  • Local time: 10:05 AM

Posted 05 February 2019 - 12:52 PM

If they pause the app and then press the home button the server might not notice to kill ffmpeg. I know if the person is playing an item and hits the home button the server knows since no more progress updates are incoming to kill the ffmpeg process. But if the user pauses and then presses home, is the server smart enough to still kill ffmpeg eventually? @Luke

 

Yes because progress updates are still sent when paused.



#5 Luke OFFLINE  

Luke

    System Architect

  • Administrators
  • 119080 posts
  • Local time: 10:05 AM

Posted 05 February 2019 - 02:34 PM

Yes it will eventually kill the process but it could take a few minutes.



#6 pir8radio OFFLINE  

pir8radio

    Advanced Member

  • Members
  • 2581 posts
  • Local time: 09:05 AM

Posted 11 February 2019 - 09:40 AM

well i have been having intermittent ffmpeg hangs, that last until i force kill them...   had two yesterday, might not have any more for days...   Looks like im going to have to re-install process lasso and kill any tasks that have been running for longer than 6 hours..  had this on my last server setup, was hoping it was fixed...  not sure what is causing it to be honest..   can pm logs if needed...  but i think it will be hard to spot...



#7 Luke OFFLINE  

Luke

    System Architect

  • Administrators
  • 119080 posts
  • Local time: 10:05 AM

Posted 11 February 2019 - 02:51 PM

Why do you feel they were hangs?



#8 pir8radio OFFLINE  

pir8radio

    Advanced Member

  • Members
  • 2581 posts
  • Local time: 09:05 AM

Posted 12 February 2019 - 10:01 PM

Why do you feel they were hangs?

 

No users on server, no recordings, no streams, FFMPEG running and consuming 0% CPU, no scheduled tasks running, task stays running as long as emby is running.   

 

 

Below I have 3 legit streams going right now, look at ffmpeg instance #2, its been running for almost 6 hours, and only has 500ms of cpu time...  it has no average CPU usage.   Its like emby started it and never put it to use, but left it open...    Its also the only instance not using any of my drives. 

 

 

hungffmpeg.png


Edited by pir8radio, 12 February 2019 - 10:19 PM.


#9 Luke OFFLINE  

Luke

    System Architect

  • Administrators
  • 119080 posts
  • Local time: 10:05 AM

Posted 13 February 2019 - 10:47 PM

How did they stop playback?



#10 speechles OFFLINE  

speechles

    Advanced Member

  • App Developer
  • 3518 posts
  • Local time: 07:05 AM

Posted 13 February 2019 - 10:54 PM

Every time the video player exits normally a stop transcode api call (/Videos/ActiveEncodings DELETE) is made. One is also made every time an error causes playback fallback or the user uses playback correction.

 

The only thing I can think of is pausing. Are they pausing the stream and then while paused pressing home? The most important variable is as luke mentioned. How are they exit the app which leaves the ffmpeg process orphaned? Knowing this it would help tremendously in spotting a root cause here. Thanks. :)


Edited by speechles, 13 February 2019 - 11:01 PM.


#11 pir8radio OFFLINE  

pir8radio

    Advanced Member

  • Members
  • 2581 posts
  • Local time: 09:05 AM

Posted 14 February 2019 - 01:22 PM

I don't think that is the case...     That ffmpeg instance only has 1/2 a second of CPU time..     no video actually played on that instance of ffmpeg.   Its like emby started it and didnt do anything with it.  



#12 speechles OFFLINE  

speechles

    Advanced Member

  • App Developer
  • 3518 posts
  • Local time: 07:05 AM

Posted 14 February 2019 - 05:43 PM

If it happens again and you can isolate which ffmpeg log belongs to it that should give us an indication of what is happening. There should be something in the logs even if ffmpeg decided to orphan itself. It would be interesting to see a root cause for this as others have reported similar and we have never been able to spot it in the wild ourselves. Having the logs is like having the footprints of it. With those can put it to rest finally. :)


Edited by speechles, 14 February 2019 - 05:44 PM.


#13 pir8radio OFFLINE  

pir8radio

    Advanced Member

  • Members
  • 2581 posts
  • Local time: 09:05 AM

Posted 14 February 2019 - 05:48 PM

Ill try to watch it.. the only pattern so far seems to be after i notice my only roku user has been watching live TV or a movie...   


Edited by pir8radio, 14 February 2019 - 05:48 PM.


#14 pir8radio OFFLINE  

pir8radio

    Advanced Member

  • Members
  • 2581 posts
  • Local time: 09:05 AM

Posted 20 February 2019 - 09:41 PM

If it happens again and you can isolate which ffmpeg log belongs to it that should give us an indication of what is happening. There should be something in the logs even if ffmpeg decided to orphan itself. It would be interesting to see a root cause for this as others have reported similar and we have never been able to spot it in the wild ourselves. Having the logs is like having the footprints of it. With those can put it to rest finally. :)

 

Here is the ffmpeg log from the last hang.   I stripped the parameters off the beginning of the log, because i hate sanitizing it.   Looks like ffmpeg did its analyze just fine, then when it tried to play below it got a 403 for whatever reason... but ffmpeg just sits there?       Pretty sure this is the log from the frozen ffmpeg.  Im going to keep looking.

ffmpeg version 4.0.3-emby_2019_02_18-g5abbb1430d+380 Copyright (c) 2000-2018 the FFmpeg developers
  built with gcc 8.2.1 (Rev1, Built by MSYS2 project) 20181214
  configuration:  --disable-autodetect --enable-amf --enable-bzlib --enable-cuda --enable-cuvid --enable-d3d11va --enable-dxva2 --enable-iconv --enable-lzma --enable-nvenc --enable-zlib --enable-sdl2 --disable-debug --enable-ffnvcodec --enable-nvdec --enable-gmp --enable-libmp3lame --enable-libopus --enable-libvorbis --enable-libvpx --enable-libx264 --enable-libx265 --enable-fontconfig --enable-libass --enable-libbluray --enable-libfreetype --enable-libmfx --enable-libmysofa --enable-libopencore-amrnb --enable-libopencore-amrwb --enable-libopenjpeg --enable-libsnappy --enable-libsoxr --enable-libspeex --enable-libtheora --enable-libtwolame --enable-libvidstab --enable-libvo-amrwbenc --enable-libwavpack --enable-libwebp --enable-libxml2 --enable-libzimg --enable-libshine --enable-gpl --enable-avisynth --enable-libxvid --enable-libaom --enable-version3 --enable-libzvbi --enable-gnutls --extra-cflags=-DLIBTWOLAME_STATIC --extra-libs=-lstdc++ --extra-cflags=-DLIBXML_STATIC --extra-libs=-liconv
  libavutil      56. 14.100 / 56. 14.100
  libavcodec     58. 18.100 / 58. 18.100
  libavformat    58. 12.100 / 58. 12.100
  libavdevice    58.  3.100 / 58.  3.100
  libavfilter     7. 16.100 /  7. 16.100
  libswscale      5.  1.100 /  5.  1.100
  libswresample   3.  1.100 /  3.  1.100
  libpostproc    55.  1.100 / 55.  1.100
[http @ 000002303a641700] HTTP error 403 Forbidden
http://iptv.com/play/2435.ts?token=eyJifQ==: Server returned 403 Forbidden (access denied)






0 user(s) are reading this topic

0 members, 0 guests, 0 anonymous users