Recording ended early

[iiiJoe 55]

5 minutes ago, Luke said:

Both, but all of the ones that are bundled with Emby Server are built by us.

How complex do you think it would be to make a plug-in that would restart failed recordings? I'm total novice on this, btw. Lol

[Luke 37068]

6 minutes ago, iiiJoe said:

How complex do you think it would be to make a plug-in that would restart failed recordings? I'm total novice on this, btw. Lol

The server already restarts failed recordings.

[iiiJoe 55]

10 minutes ago, Luke said:

The server already restarts failed recordings.

This hasn't been my experience. Is there a particular option that needs to enabled?

[BoomerGamer62 18]

On 5/30/2023 at 1:49 PM, Luke said:

The server already restarts failed recordings.

Have to say, this hasnt been my experience either -- which is how this thread originally started.  The logs show ffmpeg throwing error "Seek to start failed" and "operation not permitted.  There is no restart of the recording.

Here's are example logs.  The first is the recording log and the second is the server log  Its saying the reecording completed with it did not.:

14:08:29.749 [https @ 000001fd68bdfc40] Opening 'https://redacted.servers.tips:5052/live11/0btzL1DJVAYkm0fvO5r6RQ/689039/1685570341/3021064.m3u8' for reading
14:08:29.802 [hls @ 000001fd680ff3c0] Skip ('#EXT-X-VERSION:3')
14:08:29.802 [https @ 000001fd68936f00] Opening 'https://redacted.servers.tips:5052/live11/0btzL1DJVAYkm0fvO5r6RQ/689039/1685570341/3021064_646.ts' for reading
14:08:29.857 elapsed=00:09:23.05 frame=17280 fps= 31 q=-1.0 size=  270080kB time=00:09:24.77 bitrate=3917.5kbits/s throttle=off speed=   1x    
14:08:30.103 Seek to start failed.
14:08:30.103 https://38-68-205-41.servers.tips:5052/live11/0btzL1DJVAYkm0fvO5r6RQ/689039/1685570341/3021064.m3u8: Operation not permitted
14:08:30.103 14:08:30.106 elapsed=00:09:23.30 frame=17520 fps= 31 q=-1.0 Lsize=  273920kB time=00:09:32.81 bitrate=3917.4kbits/s throttle=off speed=1.02x    
14:08:30.106 video:249751kB audio:13702kB subtitle:0kB other streams:0kB global headers:0kB muxing overhead: 3.973063%    Last message repeated 1 times
14:08:30.117 EXIT
14:08:30.130 

------------------------------

2023-05-31 14:08:30.146 Info LiveTV: Recording completed to file G:\DVR\filenameredacted.ts
2023-05-31 14:08:30.147 Info LiveTV: Recording completed: G:\DVR\filenameredacted.ts
2023-05-31 14:08:30.147 Info MediaSourceManager: Live stream eaca7478d8b821f315c3ee45ecb4debb consumer count is now 0
2023-05-31 14:08:30.147 Info MediaSourceManager: Closing live stream 06044cf0e6f93cdae5f285c9ecfaaeb4_01413a525b3a9622ce6fdf19f7dde354_eaca7478d8b821f315c3ee45ecb4debb
2023-05-31 14:08:30.147 Info SharedHttpPipelineSource: Closing SharedHttpPipelineSource
2023-05-31 14:08:30.147 Info SharedHttpPipelineSource: Deleting temp files F:\temp\transcoding-temp\59208b1d57a44f199e5ec9928c87fb47.ts
2023-05-31 14:08:30.147 Info MediaSourceManager: Live stream 06044cf0e6f93cdae5f285c9ecfaaeb4_01413a525b3a9622ce6fdf19f7dde354_eaca7478d8b821f315c3ee45ecb4debb closed successfully
2023-05-31 14:08:30.172 Info LiveTV: Running recording post processor E:\comskip\EmbySkipAdswithSRT.bat "G:\DVR\filenameredacted.ts"
2023-05-31 14:08:30.197 Info LiveTV: Triggering refresh on G:\DVR\filenameredacted.ts
2023-05-31 14:08:30.198 Info LiveTV: Refreshing recording parent G:\DVR\folderredacted

[BoomerGamer62 18]

@Lukeplease see previous post.  Can a correction be made to pick up on "Seek to start failed" and.or "Operation not permitted" in order to fail the recording at that point and restart it?  It is being marked as completed, when it is not.

Edited June 5, 2023 by BoomerGamer62
Tag added

[Luke 37068]

Yea we’ll have to see how we can detect that, anyway the command line changes are in the latest beta build. Thanks.

[iiiJoe 55]

20 minutes ago, Luke said:

Yea we’ll have to see how we can detect that, anyway the command line changes are in the latest beta build. Thanks.

Hi, @Luke Are you talking about changes to the “delay_max” setting?

[Luke 37068]

7 minutes ago, iiiJoe said:

Hi, @Luke Are you talking about changes to the “delay_max” setting?

All of the ones we went over on the previous page.

[iiiJoe 55]

9 minutes ago, Luke said:

All of the ones we went over on the previous page.

Outstanding! Is that beta build available yet?

[iiiJoe 55]

Also, any chance the “delay_max” setting can be made user adjustable?

[Luke 37068]

5 minutes ago, iiiJoe said:

Also, any chance the “delay_max” setting can be made user adjustable?

It’s possible for future updates, but first I would try out the new changes.

[iiiJoe 55]

1 minute ago, Luke said:

It’s possible for future updates, but first I would try out the new changes.

Logical approach, Luke. Ty

[Carlo 4330]

On 5/30/2023 at 1:49 PM, Luke said:

The server already restarts failed recordings.

Hardware, yes. Can't say I've ever seen a stream from m3u file restarted. It should keep retrying right up until the recording is scheduled to finish.

On 5/29/2023 at 5:52 PM, Luke said:

That sounds a little extreme, no?

Yes, it probably is, see below.

On 5/25/2023 at 12:16 PM, BoomerGamer62 said:

Ive been digging into this a lot because I am running into this issue of my recordings stopping early, with similar messages showing in the logs.

First, the message "operation not permitted" is being thrown by ffmpeg, so this indicates ffmpeg is failing. Although it may be an issue with the source, ffmpeg should be able to recover and not fail.  In the log, I see the following as part of the ffmpeg command line:

--reconnect_delay_max 2

This is the maximum delay in seconds after which to give up reconnecting.  Two seconds seems too low, and this may be why ffmpeg is failing.  Maybe trying more seconds before failing might save recordings that are failing now.

Also, there are two command line options that might help that I dont see being used. 

reconnect_on_network_error - Reconnect automatically in case of TCP/TLS errors during connect.

reconnect_on_http_error -A comma separated list of HTTP status codes to reconnect on. The list can include specific status codes (e.g. ’503’) or the strings ’4xx’ / ’5xx’.

For more information, please see an excellent writeup at https://medium.com/intrasonics/robust-continuous-audio-recording-c1948895bb49

I hope this was helpful and I hope some of these suggestions can be incorporated in the future.  Im the meantime, is it possible for me to change the ffmpeg command line in Emby for live tv recording?  Ive looked pretty extensively and cant find it -- maybe its in a .dll or something.  Anyone know where this is located?

Based on that article

At present we have a setting of --reconnect_delay_max 2. Reconnect itself was never activated.
The setting of 2 only allowed the immediate retry, followed by another 1 second later.

ffmpeg will retry up to 127 seconds but not any longer than that. However, it's important to understand this means we've retried at 1 second, 3 seconds, 7 seconds, 15, 31 & 63 so the total pattern looks like this: 1, 3, 7, 15, 31, 63, 127 which took 247 seconds (4 minutes 7 seconds) in total with 7 additional retries.

A max value of 15 would get 4 retries over 26 seconds.
A max value of 31 would get 5 retries over 57 seconds which seems about right to me as a starting point.

We should beef up the parameters used as some only cover http(s) and some only the connection stage while others are used after the connection is made.  Others are for network.

-reconnect 1 This is what turns the feature on so must be present.

-reconnect_on_network_error 1 covers issues at the network level

-reconnect_on_http_error 4xx,5xx is used only for the initialization of the stream.

-reconnect_streamed 1 is needed to restart on http(s) issues after the initial connection

We should be able to use  reconnect_at_eof as well since we don't process .mpd files from dash streams. If set then eof is treated like an error and causes reconnection which is useful for live / endless streams.

-reconnect_delay_max 31 is what I'd suggest we use

Based on article the command line should look similar to this:

-reconnect 1 -reconnect_on_http_error 4xx,5xx -reconnect_streamed 1 -reconnect_at_eof 1 -reconnect_on_network_error 1 -reconnect_delay_max 31

[iiiJoe 55]

13 hours ago, Carlo said:

Hardware, yes. Can't say I've ever seen a stream from m3u file restarted. It should keep retrying right up until the recording is scheduled to finish.

Yes, it probably is, see below.

Based on that article

At present we have a setting of --reconnect_delay_max 2. Reconnect itself was never activated.
The setting of 2 only allowed the immediate retry, followed by another 1 second later.

ffmpeg will retry up to 127 seconds but not any longer than that. However, it's important to understand this means we've retried at 1 second, 3 seconds, 7 seconds, 15, 31 & 63 so the total pattern looks like this: 1, 3, 7, 15, 31, 63, 127 which took 247 seconds (4 minutes 7 seconds) in total with 7 additional retries.

A max value of 15 would get 4 retries over 26 seconds.
A max value of 31 would get 5 retries over 57 seconds which seems about right to me as a starting point.

We should beef up the parameters used as some only cover http(s) and some only the connection stage while others are used after the connection is made.  Others are for network.

-reconnect 1 This is what turns the feature on so must be present.

-reconnect_on_network_error 1 covers issues at the network level

-reconnect_on_http_error 4xx,5xx is used only for the initialization of the stream.

-reconnect_streamed 1 is needed to restart on http(s) issues after the initial connection

We should be able to use  reconnect_at_eof as well since we don't process .mpd files from dash streams. If set then eof is treated like an error and causes reconnection which is useful for live / endless streams.

-reconnect_delay_max 31 is what I'd suggest we use

Based on article the command line should look similar to this:

-reconnect 1 -reconnect_on_http_error 4xx,5xx -reconnect_streamed 1 -reconnect_at_eof 1 -reconnect_on_network_error 1 -reconnect_delay_max 31

@CarloThank you very much for this detailed explanation! Question: are you saying that “127” would be the limit for ffmpeg and that it can’t be set any higher? 

[Carlo 4330]

According to the article linked above that's what I got from it.  There is very little documentation about this anywhere!

I've not looked at the source code for this, but that would be way to see exactly how this works.

[iiiJoe 55]

1 hour ago, Carlo said:

According to the article linked above that's what I got from it.  There is very little documentation about this anywhere!

I've not looked at the source code for this, but that would be way to see exactly how this works.

I would LOVE to see the delay max setting be user adjustable, but I’m sure this would take some doing. In lieu of this, could we please adjust it to the max of 127? Here’s my thinking: I obviously would like to get 100% of whatever I’m trying to record. But, I don’t mind missing some minutes of whatever I’m recording via a longer reconnection sequence, especially when a recording fails after 5, 10 or 15 minutes (the point of failure varies ofc). Using the max setting of 127 could help with discerning which provider has more desirable streams. For instance, if you have one provider that has 10 disruptions on a 1 hr daily news program but reconnects within 60 seconds vs another provider that allows reconnection within a couple of minutes but only has one disruption, the latter would be preferable, imo. 

Thoughts?

Edited June 6, 2023 by iiiJoe

[Carlo 4330]

Keep in mind the 127 setting would actually be over 4 minutes of retries.
My thinking was to use something around 1 minute to handle what I'd consider minor issues.  If a stream is down for over a minute, there is probably something more serious going on.
Regardless, I still think we should keep trying and record as much as possible for the duration of the movie/show/time slot.  The difference is that after a minute, it might be better to hand control back to Emby Server to retry.

It could be the provider switched the stream to a new location and the ffmpeg retry won't be trying that.  But if Emby Server was to try the parent stream again it could pick up a redirect to a new location and continue recording. 

Keep in mind some entries in m3u point to m3u8 files which might be set up with multiple quality streams.  Emby might have picked the highest quality stream the first time but could try a different stream the second time. I'm not sure exactly how the code handles this at present but something like that could certainly be done.

Does that make sense to try and leverage the "best of both" to do what's needed to get the recording?

[iiiJoe 55]

17 minutes ago, Carlo said:

Keep in mind the 127 setting would actually be over 4 minutes of retries.
My thinking was to use something around 1 minute to handle what I'd consider minor issues.  If a stream is down for over a minute, there is probably something more serious going on.
Regardless, I still think we should keep trying and record as much as possible for the duration of the movie/show/time slot.  The difference is that after a minute, it might be better to hand control back to Emby Server to retry.

It could be the provider switched the stream to a new location and the ffmpeg retry won't be trying that.  But if Emby Server was to try the parent stream again it could pick up a redirect to a new location and continue recording. 

Keep in mind some entries in m3u point to m3u8 files which might be set up with multiple quality streams.  Emby might have picked the highest quality stream the first time but could try a different stream the second time. I'm not sure exactly how the code handles this at present but something like that could certainly be done.

Does that make sense to try and leverage the "best of both" to do what's needed to get the recording?

I’m def for whatever gets us success and a multi pronged approach makes total sense. I only record a few shows a week via m3u but successful recordings have become my “white whale”, lol. If the m3u is pointing to m3u8 files this should be in the log, yes?

[BoomerGamer62 18]

18 hours ago, Carlo said:

-reconnect 1 This is what turns the feature on so must be present.

@Carlo, my thanks for looking into this as well.  Glad the article was helpful.

Since the -reconnect 1 has never been included in the ffmpeg options, it will be interesting to see how that addition may improve things.  It's my understanding that the other "reconnect" options currently would not do anything since this has been missing from the start.

I agree about 1 minute of retrying would be a good middle ground.

[iiiJoe 55]

11 minutes ago, BoomerGamer62 said:

@Carlo, my thanks for looking into this as well.  Glad the article was helpful.

Since the -reconnect 1 has never been included in the ffmpeg options, it will be interesting to see how that addition may improve things.  It's my understanding that the other "reconnect" options currently would not do anything since this has been missing from the start.

I agree about 1 minute of retrying would be a good middle ground.

@BoomerGamer62finding that article and your input on this has been gold! I’m curious, why limit the ability of ffmpeg to reconnect at 1 min? Why not use the max setting of 127 and let it work to full capacity?

[BoomerGamer62 18]

31 minutes ago, iiiJoe said:

@BoomerGamer62finding that article and your input on this has been gold! I’m curious, why limit the ability of ffmpeg to reconnect at 1 min? Why not use the max setting of 127 and let it work to full capacity?

My only thought on that is that once you have a gap of more than a minute in your recording, then the recording itsself is pretty much shot.  Id rather know it didnt finish than have a 5 minute hole in the recoeding.

That said, Im hoping that errors can be picked up upon and restart the recording fresh.  According to @Carlo(and I agree), only hardware errors have caused restarts.....havent seen m3u recordings be restarted because of an error.

Im hoping to see some improvement in the reconnect option changes that are now in beta.

[iiiJoe 55]

3 minutes ago, BoomerGamer62 said:

My only thought on that is that once you have a gap of more than a minute in your recording, then the recording itsself is pretty much shot.  Id rather know it didnt finish than have a 5 minute hole in the recoeding.

That said, Im hoping that errors can be picked up upon and restart the recording fresh.  According to @Carlo(and I agree), only hardware errors have caused restarts.....havent seen m3u recordings be restarted because of an error.

Im hoping to see some improvement in the reconnect option changes that are now in beta.

Totally see where you’re coming from. Would be killer to make these two options user adjustable. 

[iiiJoe 55]

@Carloi wouldn’t mind being a tester for these options, if needed

Edited June 6, 2023 by iiiJoe

[Spaceboy 2493]

21 minutes ago, BoomerGamer62 said:

My only thought on that is that once you have a gap of more than a minute in your recording, then the recording itsself is pretty much shot.  Id rather know it didnt finish than have a 5 minute hole in the recoeding.

That said, Im hoping that errors can be picked up upon and restart the recording fresh.  According to @Carlo(and I agree), only hardware errors have caused restarts.....havent seen m3u recordings be restarted because of an error.

Im hoping to see some improvement in the reconnect option changes that are now in beta.

i disagree with this. i want it to keep retrying for the whole length of the original recording. imagine i'm recording sports game. i'd rather miss 5mins of action in the middle of the game and have the end recorded than emby give up and miss everything. there's no downside to that.

yes the user should be notified, plus if another broadcast is available at a later date then emby should automatically schedule that for recording too

edit - but you guys should get better iptv providers. i almost never have failed recordings

Edited June 6, 2023 by Spaceboy

[Carlo 4330]

Spaceboy, I think we all want it to keep trying.  All I was suggesting is about a minute for ffmpeg to try and reconnect then give up.  At that point it's back to Emby Server to try and re-establish the connection.  My thought is that there could be additional streams available, could have been a stream redirect so the new location of the stream is different, etc.

Oh yea, another, when we get stacked channel support (ie one channel with different resolutions) Emby could switch streams to continue if needed. Lots of people get multiple versions of the same channel isuch as 4K, 1080, 720, SD so these could be stacked and used in cases like this.