Recording ended early

[iiiJoe 55]

On 7/3/2023 at 7:23 PM, iiiJoe said:

@Brian_Mthanks for joining the conversation! @BoomerGamer62found an excellent article on solving these issues: https://medium.com/intrasonics/robust-continuous-audio-recording-c1948895bb49

I believe @Lukeis trying to implement these changes in beta but I don’t believe we’ve seen any difference so far. Would you mind sharing how you did your testing? Wouldn’t mind trying this myself. Were you able to keep your recordings alive?

@Carlo@Luke, any updates on making these changes in beta?

[iiiJoe 55]

Update: was doing a livestream and Emby/ ffmpeg tried to recover from an interruption(http error 404) 3 times for a total of 4 seconds. Looks like the delay max setting is currently at 4. Can this please be changed to 127?

[iiiJoe 55]

Any updates here? @Carlo@Luke

Edited July 20, 2023 by iiiJoe

[Brian_M 4]

I apologize for the delay in getting back on this thread @iiiJoeeven though I'm not much help.  I thought I had the issues fixed but I ended up still having them.  So, I've been working on a shell script/wrapper for ffmpeg that checks the log file to see if the recording failed, and if so, restarts ffmpeg and appends to the existing .ts recording.  It's mostly working but I will see if I can adapt it to Windows.  I just can't understand why Emby simply doesn't restart the recording again when it fails.  Instead, it just says it's done and never even tries again.

I think for some of our issues we are using the same IPTV provider, and it seems that once a 404, 410, or 500 error appear you pretty much have to restart ffmpeg entirely to get it to ever really work correctly again.

[iiiJoe 55]

1 hour ago, Brian_M said:

I apologize for the delay in getting back on this thread @iiiJoeeven though I'm not much help.  I thought I had the issues fixed but I ended up still having them.  So, I've been working on a shell script/wrapper for ffmpeg that checks the log file to see if the recording failed, and if so, restarts ffmpeg and appends to the existing .ts recording.  It's mostly working but I will see if I can adapt it to Windows.  I just can't understand why Emby simply doesn't restart the recording again when it fails.  Instead, it just says it's done and never even tries again.

I think for some of our issues we are using the same IPTV provider, and it seems that once a 404, 410, or 500 error appear you pretty much have to restart ffmpeg entirely to get it to ever really work correctly again.

Hey, Brian! We’re all super busy, lol. I did install Beta and had one recording that ffmpeg tried to re-start (3) times after the initial failure but the entire process lasted less than 5 seconds. I believe this is due to the delay max setting being at 4 instead of 127 as prescribed in documentation. Would really like to see this changed as it doesn’t seem complicated and would likely only require a few key strokes. Wish I could help on the script/wrapper for ffmpeg, sounds very promising. I’m running windows 11 and certainly wouldn’t mind beta testing. What would you say your success rate is in re-starting failed recordings with your script?

Edited July 20, 2023 by iiiJoe

[BoomerGamer62 18]

My beta testing has went well.  Certainly seeing a reduction is recordings ending early.  @Luke,  any idea when this will get out of beta and into production?

[Luke 37166]

On 7/25/2023 at 10:01 AM, BoomerGamer62 said:

My beta testing has went well.  Certainly seeing a reduction is recordings ending early.  @Luke,  any idea when this will get out of beta and into production?

Hi, no ETA but we're working hard on it.

[joekingcool 26]

@Luke

after doing some tests i've came to the conclusion thus far. vpn reconnects and causes interruption of recordings and gives "Seek to start failed". but emby doesn't restart recording. im assuming the time it takes for vpn to auto reconnect is longer than few seconds and ffmpeg gives up. 

if you goto  "event viewer \ application and service logs \ microsoft \ windows \ networkprofile " in windows machine. you will see a history of network changes. ive seen a pattern with timestamp of "Seek to start failed" in emby logs with the timestamps in event viewer. 

if this is true then we might of found the problem to some of these  "Seek to start failed" failed recordings. but now how do we get emby to restart the recording?

 

ps, i tried the recent beta version. and i had a problem where all my scheduled recordings deleted at a later point. im not sure what happened, it could of been a mistake in the transition from regular to beta version. although after i transitioned they were in there but later they weren't. im not sure if anyone else had this trouble, if not then it's probably a mistake on my part. just not wanting to risk again at this point, of no recordings. 

Edited November 29, 2023 by joekingcool

[joekingcool 26]

@Luke

follow-up on last post. i did further testing and found my hard drive was causing most the interruptions, but on a rare occasion it was vpn reconnecting. both caused same error to stop recordings. which emby didn't restart. 

also i tested the beta version again, and seems great no issues at all. i like how they designed the sidebar. also if i play livetv it dont stop randomly and have to start again. although it occasionally jumps forward by a sec or 2 . i assume that's the changes in keeping the connection longer. hopefully now that i have a better drive even that won't happen. 

ps. i used crystaldiskinfo  app. it showed caution on that drive and said its power on time was over 5 yrs. so i moved to another drive. also tested doing 13 recordings at one time and non failed. then tried another 13 recordings and 2 failed, but not from vpn. although this drive is close to 5 yrs old but no warnings so far. next week i have new drive coming and can confirm more then. 

[iiiJoe 55]

26 minutes ago, joekingcool said:

@Luke

follow-up on last post. i did further testing and found my hard drive was causing most the interruptions, but on a rare occasion it was vpn reconnecting. both caused same error to stop recordings. which emby didn't restart. 

also i tested the beta version again, and seems great no issues at all. i like how they designed the sidebar. also if i play livetv it dont stop randomly and have to start again. although it occasionally jumps forward by a sec or 2 . i assume that's the changes in keeping the connection longer. hopefully now that i have a better drive even that won't happen. 

ps. i used crystaldiskinfo  app. it showed caution on that drive and said its power on time was over 5 yrs. so i moved to another drive. also tested doing 13 recordings at one time and non failed. then tried another 13 recordings and 2 failed, but not from vpn. although this drive is close to 5 yrs old but no warnings so far. next week i have new drive coming and can confirm more then. 

The hard drive aspect is really interesting. I’m using less than 1yr old solid state. Kinda wondering if the delay max setting on Beta has been changed, it was at 3 which provided 3 seconds of re-tries. As a side-note, I upgraded my modem and that did seem to cut down on failed recordings though they still happen too often. I have someone working on a script to monitor recordings and re-start failures, so I’m interested to see if that will be the silver bullet. Question for @Lukeand anyone else who might know: when the “delay max”setting is being utilized in Beta is it opening a new header?

[joekingcool 26]

@iiiJoe

so you record and have emby installed on ssd?

have you checked the event log like i suggested for network changes?

you would think that emby would keep track of recordings.... check every min or so if any recordings are "done" but didn't reach its "end time", if so then start recording again. its difficult to design something that monitors from outside embys software to catch and reschedule. unless i can find some cmdline for emby to give info and reschedule. then i might create a script. if it still happens with new seagate nas drive next week, then ill probably start working on an app to monitor it. also i tried using udp vpn connection as well, but not sure if it helped. in theory it should. 

 

 

[iiiJoe 55]

4 minutes ago, joekingcool said:

@iiiJoe

so you record and have emby installed on ssd?

have you checked the event log like i suggested for network changes?

you would think that emby would keep track of recordings.... check every min or so if any recordings are "done" but didn't reach its "end time", if so then start recording again. its difficult to design something that monitors from outside embys software to catch and reschedule. unless i can find some cmdline for emby to give info and reschedule. then i might create a script. if it still happens with new seagate nas drive next week, then ill probably start working on an app to monitor it. also i tried using udp vpn connection as well, but not sure if it helped. in theory it should. 

 

 

Yep. Everything on SSD. Haven’t checked the event log. I’ll do that on next failed recording. Yeah, I don’t understand at all why Emby doesn’t monitor for a re-start since this would make the most sense and would plug the final hole in the platform, imo. I’m happy with Emby except for this issue, which unfortunately is kinda big. I’ve used Express VPN and IP Vanish which actually led to an increase in failed recordings. I think the VPN only helps if your ISP tries to block your connection. 

[joekingcool 26]

@iiiJoe

i've been testing some more. when I'm transferring large files on and off the drive its recording too. it seems to trigger it fail recordings. so I'm thinking the answer is even simpler. it preferably needs its own drive for recordings. if you install emby on that drive like me. when it updates metadata it can buffer read/writes to the drive. but i also use same drive for security cameras. which is maybe ok until i transfer them to the new drive..... so while i've been testing and moving to new drives, i also had allot of other big files that needed transferred as well working in background. so I'm sure that has caused some of my issues because as soon as i started transfering few minutes later over half the recordings stopped. 

and in your case of using everything on onedrive. i would say same issue. any windows updates, anti-virus, metadata updates, etc. would buffer the read/writes to the drive long enough to cause ffmpeg to fail. even though you have an ssd with really high read/writes i assume. 

[iiiJoe 55]

4 hours ago, joekingcool said:

@iiiJoe

i've been testing some more. when I'm transferring large files on and off the drive its recording too. it seems to trigger it fail recordings. so I'm thinking the answer is even simpler. it preferably needs its own drive for recordings. if you install emby on that drive like me. when it updates metadata it can buffer read/writes to the drive. but i also use same drive for security cameras. which is maybe ok until i transfer them to the new drive..... so while i've been testing and moving to new drives, i also had allot of other big files that needed transferred as well working in background. so I'm sure that has caused some of my issues because as soon as i started transfering few minutes later over half the recordings stopped. 

and in your case of using everything on onedrive. i would say same issue. any windows updates, anti-virus, metadata updates, etc. would buffer the read/writes to the drive long enough to cause ffmpeg to fail. even though you have an ssd with really high read/writes i assume. 

This is REALLY interesting. So background or other processes may be causing the disruption. I actually have a spare SSD that I could install for testing. Emby lets you specify a custom path for recordings. I may just direct my recordings to the unused spare drive

Edited December 1, 2023 by iiiJoe

[joekingcool 26]

@iiiJoe

also ive noticed my timers.json which is where all the recordings are stored. need a lot of cleanup. mostly from me changing the recording folder over time, delete videos outside of emby, etc. but if everything stays the same and do everything thru emby, it does cleanup the list of recordings.

so im working on a app to help cleanup. wish me luck !

[joekingcool 26]

@Luke @iiiJoe

i did a fresh install of windows. fresh build from scratch of emby beta. with new hard drive.

still getting over half recordings failed, with same error "Seek to start failed". tried figuring out a pattern by looking threw the logs allot. i cant seem to find a reason. other than the timeline of the video is getting messed up. maybe windows time changes when it updates a little and throws off timestamps. its not because when ip address changes of the stream ether.

but if i reboot pc, it will restart recording. makes no since unless timeline of video is getting mixed up. even so if timeline is messed up to continue. it should stop and start recording again. 

i did make an app to easily view recording set and delete old ones. if anyone would like me to post. 

i did make an app/script to read threw the logs and tell you what the name are of failed recordings. i run on a schedule. then send text msg to phone. can post as well if anyone like. 

so hopefully luke can help with this failed recordings, because its becoming useless for livetv recordings. generally only have a few recordings a day. but ive tested multiple times with over a dozen channels recording at same time. and i cant seem to find a pattern of what makes them fail. most of our shows are VOD but few shows we have to get from livetv. only few shows a day, and over half are failing. 

 

[joekingcool 26]

On 7/20/2023 at 3:39 PM, Brian_M said:

So, I've been working on a shell script/wrapper for ffmpeg that checks the log file to see if the recording failed, and if so, restarts ffmpeg and appends to the existing .ts recording.  It's mostly working but I will see if I can adapt it to Windows. 

@Brian_M@iiiJoe @Luke

im thinking of writing a script that does same thing on windows. scan log file for failed recording then extract the cmd from log file and rerun. although i need some help on changing the cmd because if i rerun exactly it dont write the new log file correctly and other questions i have. 

[iiiJoe 55]

10 hours ago, joekingcool said:

@Brian_M@iiiJoe @Luke

im thinking of writing a script that does same thing on windows. scan log file for failed recording then extract the cmd from log file and rerun. although i need some help on changing the cmd because if i rerun exactly it dont write the new log file correctly and other questions i have. 

So, I believe for a script to effectively restart failed recordings it would have to create a new header, otherwise Emby will just be recording “dead air”. I’m not smart enough to write one but I have talked to some other ppl who say this is how it would need to be to actually work. 

[iiiJoe 55]

Not sure why Emby can’t be programmed to restart the recording when “seek to start” fails.

[iiiJoe 55]

@Lukeany chance you could provide access to the file log so a script could read it in real time?

[Luke 37166]

1 hour ago, iiiJoe said:

@Lukeany chance you could provide access to the file log so a script could read it in real time?

What file log do you mean?

[iiiJoe 55]

1 hour ago, Luke said:

What file log do you mean?

[iiiJoe 55]

Here’s what I’m being told from a friend:

“When Emby starts the log file it seems to become the administrator of the file and it locks it. I keep getting a “File Access Not Allowed” error trying to read the file from the script. Which is a bit strange because I can read the file in notepad in near real time.”

@Lukeis there a solution or work around for this?

[Brian_M 4]

Here is my scripts I wrote that I use on Linux to solve this problem.  While I know this may not help you all out, maybe it will give you some ideas.

Here's my ffmpeg script.  The real ffmpeg was renamed to ffmpeg.bin

#!/bin/bash

APP_DIR=/opt/emby-server

export AMDGPU_IDS=$APP_DIR/share/libdrm/amdgpu.ids
export FONTCONFIG_PATH=$APP_DIR/etc/fonts
export LD_LIBRARY_PATH=$APP_DIR/lib:$APP_DIR/extra/lib
export LIBVA_DRIVERS_PATH=$APP_DIR/extra/lib/dri
export OCL_ICD_VENDORS=$APP_DIR/extra/etc/OpenCL/vendors
export PCI_IDS_PATH=$APP_DIR/share/hwdata/pci.ids
export SSL_CERT_FILE=$APP_DIR/etc/ssl/certs/ca-certificates.crt

$APP_DIR/bin/ffmpeg.bin "$@"

if echo $5 | grep recording; then
  LOGFILE=`echo $5|sed s/graph//`

  for ((restart = 1; restart <= 10; restart++)); do
    sleep 5
    if ! grep "\[q\] command" $LOGFILE; then
      echo "*** "`date`" - FAIL - RESTART #"$restart" - ""${52}" - log $LOGFILE >> /var/lib/emby/logs/failed-recordings
      echo
      echo "**** recording failed, attempting restart #$restart ****"
      echo
      $APP_DIR/bin/ffmpeg.bin $1 $2 $3 $4 "$5" $6 $7 $8 $9 ${10} ${11} ${12} ${13} ${14} ${15} ${16} ${17} ${18} ${19} "${20}" ${21} ${22} ${23} ${24} ${25} ${26} ${27} ${28} ${29} ${30} ${31} 15 ${33} "${34}" ${35} ${36} ${37} ${38} ${39} ${40} ${41} ${42} ${43} ${44} ${45} ${46} ${47} ${48} ${49} ${50} ${51} -f mpegts - >> "${52}"
    else
      break
    fi
  done
fi

Basically all it does is check the log file for "[q] command" and if this doesn't exist, the recording has failed, so it will restart the recording for up to 10 times.  After that many restarts, I've found it pointless to try anymore.  The recording is just appended to the one that failed.  I think I had to add the sleep timer in there to wait until the logs flushed. 

I've always had to "recopy" IPTV recordings to make comskip work in the first place, but this may be needed to make these files playable after that fact.  Here's my postprocessing script.  I should probably use a temporary directory to do the recopy in so that Emby doesn't detect it and make .nfo and thumb files for the copy while it's being processed.

#!/bin/bash
LD_LIBRARY_PATH=
ffmpeg -i "$1" -codec copy "$1-fixed.ts" 2> "$1.fix.log"
ls -l "$1" >> "$1.fix.log"
ls -l "$1-fixed.ts" >> "$1.fix.log"
mv "$1-fixed.ts" "$1"
rm "$1-fixed.nfo"
rm "$1-fixed-thumb.jpg"
/usr/bin/comskip --ini=/usr/local/etc/comskip.ini "$1" > "$1-comskip.log"

I know this is the Windows forum, so I apologize I'm not much help, but maybe some ideas can be had from this.  I'm just very surprised that all of this is needed in the first place really, if I just straight up kill ffmpeg Emby will automatically restart the recording, but it will not do so for these kinds of failures.

[iiiJoe 55]

46 minutes ago, Brian_M said:

Here is my scripts I wrote that I use on Linux to solve this problem.  While I know this may not help you all out, maybe it will give you some ideas.

Here's my ffmpeg script.  The real ffmpeg was renamed to ffmpeg.bin

#!/bin/bash

APP_DIR=/opt/emby-server

export AMDGPU_IDS=$APP_DIR/share/libdrm/amdgpu.ids
export FONTCONFIG_PATH=$APP_DIR/etc/fonts
export LD_LIBRARY_PATH=$APP_DIR/lib:$APP_DIR/extra/lib
export LIBVA_DRIVERS_PATH=$APP_DIR/extra/lib/dri
export OCL_ICD_VENDORS=$APP_DIR/extra/etc/OpenCL/vendors
export PCI_IDS_PATH=$APP_DIR/share/hwdata/pci.ids
export SSL_CERT_FILE=$APP_DIR/etc/ssl/certs/ca-certificates.crt

$APP_DIR/bin/ffmpeg.bin "$@"

if echo $5 | grep recording; then
  LOGFILE=`echo $5|sed s/graph//`

  for ((restart = 1; restart <= 10; restart++)); do
    sleep 5
    if ! grep "\[q\] command" $LOGFILE; then
      echo "*** "`date`" - FAIL - RESTART #"$restart" - ""${52}" - log $LOGFILE >> /var/lib/emby/logs/failed-recordings
      echo
      echo "**** recording failed, attempting restart #$restart ****"
      echo
      $APP_DIR/bin/ffmpeg.bin $1 $2 $3 $4 "$5" $6 $7 $8 $9 ${10} ${11} ${12} ${13} ${14} ${15} ${16} ${17} ${18} ${19} "${20}" ${21} ${22} ${23} ${24} ${25} ${26} ${27} ${28} ${29} ${30} ${31} 15 ${33} "${34}" ${35} ${36} ${37} ${38} ${39} ${40} ${41} ${42} ${43} ${44} ${45} ${46} ${47} ${48} ${49} ${50} ${51} -f mpegts - >> "${52}"
    else
      break
    fi
  done
fi

Basically all it does is check the log file for "[q] command" and if this doesn't exist, the recording has failed, so it will restart the recording for up to 10 times.  After that many restarts, I've found it pointless to try anymore.  The recording is just appended to the one that failed.  I think I had to add the sleep timer in there to wait until the logs flushed. 

I've always had to "recopy" IPTV recordings to make comskip work in the first place, but this may be needed to make these files playable after that fact.  Here's my postprocessing script.  I should probably use a temporary directory to do the recopy in so that Emby doesn't detect it and make .nfo and thumb files for the copy while it's being processed.

#!/bin/bash
LD_LIBRARY_PATH=
ffmpeg -i "$1" -codec copy "$1-fixed.ts" 2> "$1.fix.log"
ls -l "$1" >> "$1.fix.log"
ls -l "$1-fixed.ts" >> "$1.fix.log"
mv "$1-fixed.ts" "$1"
rm "$1-fixed.nfo"
rm "$1-fixed-thumb.jpg"
/usr/bin/comskip --ini=/usr/local/etc/comskip.ini "$1" > "$1-comskip.log"

I know this is the Windows forum, so I apologize I'm not much help, but maybe some ideas can be had from this.  I'm just very surprised that all of this is needed in the first place really, if I just straight up kill ffmpeg Emby will automatically restart the recording, but it will not do so for these kinds of failures.

Thank you for this. I will pass it along to my friend. Question: does your script start a new recording or does it continue the previous? Also, does the script stagger the restarts with like 30 second (or some other interval) pauses?