Recording ended early

[iiiJoe 55]

Hello to all! Emby working really well but I did have a small issue tonight. I began watching a recorded program while it was still being recorded. Playback was working great with no issues as I fast forwarded. When I was FF for the final time it settled on the point where i stopped FF and then jumped back to the point when I began the FF. At the moment it jumped back Emby stopped recording the program. In the log the error is "Seek To Start Failed". Reocrding was being done on PC while playback was on my Shield. Emby server and recording logs attached. Any help would be greatly appreciated! Thanks in advance guys!!

embyserver(2).txt recording-conversion-b84baca6-3711-41d1-acce-3ea2f6999fac_1.txt

Edited November 22, 2022 by iiiJoe

[iiiJoe 55]

Hi, guys. Everything has been working pretty good but I have had some recordings ending early with an error message I don't recognize.

"Error in the pull function"

"IO error: Error number -10054 occurred"

Would appreciate any input. Thanks in advance!

embyserver-63805881600.txt recording-conversion-516e6aac-1ef8-41df-a011-98f681f68b34_1.txt

[Luke 37067]

HI, it's hard to answer this because it looks like the stream just stopped and the url became invalid. The best way to try and get to the bottom of this would be to immediately try starting the recording again after it fails or try to play the channel.

[iiiJoe 55]

Luke, when this has happened before I was in front of my pc and tried to restart the recording almost immediately after it stopped. Here’s what happened: went to the guide, clicked on the program and hit record. The guide would then have the active red dot indicating the show was now being recorded and would even say the show was recording (in the guide area where you can press play, record series, record show, cancel the series). However, the log would not show that a new recording had been triggered and the program would not show in the active recordings section of the Home Screen. It appears part of Emby thought the program was being recorded (but it really wasn’t). At the exact same time I can live play the program with no problem on pc or any client device. The issue seems to be with recording only. 

[iiiJoe 55]

Here's a similiar example, although I wasn't able to catch it in time to see if I could just play the channel right after the recording stopped. I know these m3u's can be enigmatic. Let me know what you think.

embyserver-63806054400.txt recording-conversion-4a821356-ca43-4a77-b487-fc694271dccd_1.txt

[iiiJoe 55]

Luke, here's another one that I'm trying to decipher. Recording ended after 15 min and the error message said operation not permitted. What do you think? Stream ended prematurely? Or, perhaps provider changed the stream so it couldn't be recorded? Thanks in advance

embyserver.txt recording-conversion-6adf6626-564e-48d4-b93e-44056e5a80ed_1(1).txt

[iiiJoe 55]

Hello everyone. Just wondering if anyone knows what this error message means? My recording ended after 7 minutes and was supposed to be 1 hr. Been recording this same program all week with no issues. Any help would be greatly appreciated. Thanks!

embyserver(1).txt recording-conversion-11dc90f4-ceec-4afc-a552-dcc665784074_1.txt

[Luke 37067]

Hi, the server hosting the m3u sent back an error response. More information will be needed from them.

[iiiJoe 55]

Hi, Luke. Apologies it took so long to get back. Been a long week, lol. So, it would be safe to say this is a provider issue. I have another recording that stopped prematurely that I'd like to get your input on, if you don't mind. "HTTP: error 410 gone". There are some error messages at the beginning too. I'm making a list of these errors in the hopes I can make some tweaks and avoid them in the future. Thank you again!

recording-conversion-8227703e-44e1-4335-96df-c452ee3f55a4_1.txt embyserver.txt

[Luke 37067]

Hi, yes that looks like the same thing. The recording URL became invalid abruptly in the middle of it.

[iiiJoe 55]

Any suggestions on software to monitor my network to check for packet loss? Would like to determine if this an ISP or provider issue. I’m leaning toward provider but would like to be sure…

[iiiJoe 55]

Hello to all. So I was recording very early this morning and the recording ended prematurely with this error message "operation not permitted". Any ideas on what this means? Maybe some kind of provider issue? Thanks in advance!

recording-conversion-7cf0f410-56e1-47e4-97b0-887a235a6e53_1.txt embyserver-63808646400.txt embyserver-63808646400.txt

[Luke 37067]

@iiiJoe I've merged your new topic into our existing ongoing discussion.

[iiiJoe 55]

@Luke, thank you. Wasn’t sure what to do since it was a new error message

[iiiJoe 55]

Guys, any thoughts on this?

[Luke 37067]

I think it is a problem with the source stream. I see you have another topic about retries that we'll respond to. Thanks.

[BoomerGamer62 18]

Ive been digging into this a lot because I am running into this issue of my recordings stopping early, with similar messages showing in the logs.

First, the message "operation not permitted" is being thrown by ffmpeg, so this indicates ffmpeg is failing. Although it may be an issue with the source, ffmpeg should be able to recover and not fail.  In the log, I see the following as part of the ffmpeg command line:

--reconnect_delay_max 2

This is the maximum delay in seconds after which to give up reconnecting.  Two seconds seems too low, and this may be why ffmpeg is failing.  Maybe trying more seconds before failing might save recordings that are failing now.

Also, there are two command line options that might help that I dont see being used. 

reconnect_on_network_error - Reconnect automatically in case of TCP/TLS errors during connect.

reconnect_on_http_error -A comma separated list of HTTP status codes to reconnect on. The list can include specific status codes (e.g. ’503’) or the strings ’4xx’ / ’5xx’.

For more information, please see an excellent writeup at https://medium.com/intrasonics/robust-continuous-audio-recording-c1948895bb49

I hope this was helpful and I hope some of these suggestions can be incorporated in the future.  Im the meantime, is it possible for me to change the ffmpeg command line in Emby for live tv recording?  Ive looked pretty extensively and cant find it -- maybe its in a .dll or something.  Anyone know where this is located?

[iiiJoe 55]

@BoomerGamer62 really appreciate you diving into this! Would you mind posting some screen shots of where I might find these command lines? 

[BoomerGamer62 18]

2 hours ago, iiiJoe said:

@BoomerGamer62 really appreciate you diving into this! Would you mind posting some screen shots of where I might find these command lines? 

If you look in the logs folder, you should see a bunch of files that all start with "recording".  There should be one file for each recording that took place.  You can find the correct log by matching up the date modified time of the log file with what time your recording ended prematurely.  Search this file for ffmpeg and you will see the entire command line that actually does the recording.  It's a very large string of characters.  Here is an example from one of my logs:

ffmpeg.exe -loglevel +timing -y -print_graphs_file "C:\Users\xxx\AppData\Roaming\Emby-Server\programdata\logs\recording-conversion-3da74b5a-81f0-4f74-a1fc-be4bae402b0d_1graph.txt" -copyts -start_at_zero -fflags +genpts -analyzeduration 3000000 -f hls -stream_loop -1 -c:v:0 h264 -user_agent "VLC/3.0.0-git LibVLC/3.0.0-git" -multiple_requests 1 -reconnect_streamed 1 -reconnect_delay_max 2 -i "https://0-0-0-0.servers.tips:5052/live11/URL.m3u8" -map 0:0 -map 0:1 -sn -c:v:0 copy -c:a:0 copy -disposition:a:0 default -avoid_negative_ts disabled -map_metadata -1 -map_chapters -1 "E:\DVR\outputfile.ts"

Also, if your recording stopped before you expected it to, you should see Operation not permitted in your log, near the end.  That's coming from ffmpeg saying it failed to complete.

Here's my problem.  Although I have a couple suggestions to try to improve the ffmpeg setup (see previous post), I have NO idea where or if I can change that ffmpeg command.  Ive poked around and not come up with anything.

I HAVE tested the options in a command line outside of Emby, and it records fine (or at least it hasnt failed yet).  I plan on setting a long recording in Emby and the same recording in a command line outside of Emby and see what happens.

Im hoping @Lukeor @ebrmight give me a clue, or perhaps change the ffmpeg call in Emby the way I suggested.

 

[Luke 37067]

9 minutes ago, BoomerGamer62 said:

If you look in the logs folder, you should see a bunch of files that all start with "recording".  There should be one file for each recording that took place.  You can find the correct log by matching up the date modified time of the log file with what time your recording ended prematurely.  Search this file for ffmpeg and you will see the entire command line that actually does the recording.  It's a very large string of characters.  Here is an example from one of my logs:

ffmpeg.exe -loglevel +timing -y -print_graphs_file "C:\Users\xxx\AppData\Roaming\Emby-Server\programdata\logs\recording-conversion-3da74b5a-81f0-4f74-a1fc-be4bae402b0d_1graph.txt" -copyts -start_at_zero -fflags +genpts -analyzeduration 3000000 -f hls -stream_loop -1 -c:v:0 h264 -user_agent "VLC/3.0.0-git LibVLC/3.0.0-git" -multiple_requests 1 -reconnect_streamed 1 -reconnect_delay_max 2 -i "https://0-0-0-0.servers.tips:5052/live11/URL.m3u8" -map 0:0 -map 0:1 -sn -c:v:0 copy -c:a:0 copy -disposition:a:0 default -avoid_negative_ts disabled -map_metadata -1 -map_chapters -1 "E:\DVR\outputfile.ts"

Also, if your recording stopped before you expected it to, you should see Operation not permitted in your log, near the end.  That's coming from ffmpeg saying it failed to complete.

Here's my problem.  Although I have a couple suggestions to try to improve the ffmpeg setup (see previous post), I have NO idea where or if I can change that ffmpeg command.  Ive poked around and not come up with anything.

I HAVE tested the options in a command line outside of Emby, and it records fine (or at least it hasnt failed yet).  I plan on setting a long recording in Emby and the same recording in a command line outside of Emby and see what happens.

Im hoping @Lukeor @ebrmight give me a clue, or perhaps change the ffmpeg call in Emby the way I suggested.

 

You can't during recording, but the diagnostics plugin actually has a command line search and replace feature for transcoding playback, so you could actually use that to experiment. Just make sure to play the live stream from the channel, rather than starting a recording and playing the recording.

That article is actually very helpful though and it sounds like we want to increase the value of reconnect_delay_max.

[Luke 37067]

@BoomerGamer62if you do try this out please let me know if any specific changes have an impact. Thanks !

[iiiJoe 55]

14 hours ago, BoomerGamer62 said:

If you look in the logs folder, you should see a bunch of files that all start with "recording".  There should be one file for each recording that took place.  You can find the correct log by matching up the date modified time of the log file with what time your recording ended prematurely.  Search this file for ffmpeg and you will see the entire command line that actually does the recording.  It's a very large string of characters.  Here is an example from one of my logs:

ffmpeg.exe -loglevel +timing -y -print_graphs_file "C:\Users\xxx\AppData\Roaming\Emby-Server\programdata\logs\recording-conversion-3da74b5a-81f0-4f74-a1fc-be4bae402b0d_1graph.txt" -copyts -start_at_zero -fflags +genpts -analyzeduration 3000000 -f hls -stream_loop -1 -c:v:0 h264 -user_agent "VLC/3.0.0-git LibVLC/3.0.0-git" -multiple_requests 1 -reconnect_streamed 1 -reconnect_delay_max 2 -i "https://0-0-0-0.servers.tips:5052/live11/URL.m3u8" -map 0:0 -map 0:1 -sn -c:v:0 copy -c:a:0 copy -disposition:a:0 default -avoid_negative_ts disabled -map_metadata -1 -map_chapters -1 "E:\DVR\outputfile.ts"

Also, if your recording stopped before you expected it to, you should see Operation not permitted in your log, near the end.  That's coming from ffmpeg saying it failed to complete.

Here's my problem.  Although I have a couple suggestions to try to improve the ffmpeg setup (see previous post), I have NO idea where or if I can change that ffmpeg command.  Ive poked around and not come up with anything.

I HAVE tested the options in a command line outside of Emby, and it records fine (or at least it hasnt failed yet).  I plan on setting a long recording in Emby and the same recording in a command line outside of Emby and see what happens.

Im hoping @Lukeor @ebrmight give me a clue, or perhaps change the ffmpeg call in Emby the way I suggested.

 

@BoomerGamer62the premature ending of recordings with the “operation not permitted”  error happens occasionally on some recordings but happens virtually every time I try to record on one specific channel. Please let me know the procedure to change the reconnect_delay_max value. Also, can you please elaborate on testing the options in a command line outside of Emby? Programming noob here, lol. 

[BoomerGamer62 18]

4 hours ago, iiiJoe said:

@BoomerGamer62the premature ending of recordings with the “operation not permitted”  error happens occasionally on some recordings but happens virtually every time I try to record on one specific channel. Please let me know the procedure to change the reconnect_delay_max value. Also, can you please elaborate on testing the options in a command line outside of Emby? Programming noob here, lol. 

@iiiJoeaccording to what Luke said, there unfortunately isnt a way to change the reconnect_delay_max value in Emby.  Im doing testing now, and I hope to convince everyone at Emby to change the command string to keep the recording alive longer.

If you want to test it outside of Emby, I would set up a fresh folder with nothing else in it so you dont accidentally affect anything.  Then copy the ffmpeg.exe file into that (its in the system folder where your Emby system is, like the logs are).  Ive attached a sample bat file you can run to test it, but you will have to edit it to replace the sample URL I have (URLSTREAM.m3u8).  I put a pause in the file so you can see the results in the command window.  Once you edit the bat file, just run it.

By the way, one suggestion that no one has mentioned yet:  check the speed of you internet connection (I use speedtest.net), and if you use a VPN, check that the load on the VPN isnt maxed out at 100%.  A slow connection can cause timeouts and I/O errors and lead to your recording failing.

Hope this helps.

simple.bat

[BoomerGamer62 18]

17 hours ago, Luke said:

@BoomerGamer62if you do try this out please let me know if any specific changes have an impact. Thanks !

@LukeThanks for taking a look at this.  Ive completed testing.  I did 10 one-hour recordings. Ten were with the command string exactly as it shows in the Emby log, and ten recordings with the modified command string.  The same URL/channel was used each time, at the same time of day.  The results were that the unmodified command string failed two times, but the modified command string did not fail at all.  I did not see any difference in performance or system resource use on the server.

Therefore, I think it would benefit everyone to change the ffmpeg command string in Emby.  My suggestion is to replace

 -reconnect_streamed 1 -reconnect_delay_max 2

with

-reconnect 1 -reconnect_streamed 1 -reconnect_delay_max 127 -reconnect_on_http_error 5xx,4xx

Ive attached the bat file I tested with for reference (input URL is edited).

These changes are based on the article I quoted earlier (which mentioned you need -reconnect 1 so I included that too).  Hope you will consider making this change in a future release.  If I can help further let me know.

[BoomerGamer62 18]

Forgot to attach the bat file.

fulltest0526.bat