Recording ended early

[iiiJoe 62]

Just wanted to update everyone on the Monitor program. Been using it now for 4 months and it has been a REALLY nice solution for failed recordings. Interested in anyone else’s experiences……

[Ugnaughts 53]

i started tinkering with the monitor recently and founds some more info.

i found that it seems the STS (seek to start error) happens only on HLS streams. if the channel uses mp4 as the container instead of HLS. it has a self healing within about 30-45 seconds of the stream being blocked.  this might explain previous discussions about changing the cmd that emby uses to record the stream. it might have been helping but only for the MP4 streams not the HLS streams. 

--Example HLS stream CMD emby uses--

E:\Emby-Server\system\ffmpeg.exe -loglevel +timing -y -print_graphs_file "E:\Emby-Server\programdata\logs\recording-conversiongraph.txt" -copyts -start_at_zero -f mp4 -stream_loop -1 -c:v:0 h264 -noautorotate -user_agent "VLC/3.0.0-git LibVLC/3.0.0-git" -multiple_requests 1 -reconnect 1 -reconnect_on_network_error 1 -reconnect_on_http_error 5xx,4xx -reconnect_streamed 1 -reconnect_delay_max 8 -i "https://somestream.m3u8" -map 0:0 -an -sn -c:v:0 copy -avoid_negative_ts disabled -map_metadata -1 -map_chapters -1 "E:\Emby-Server\recordings\someshow.ts"

--Example MP4 stream CMD emby uses--

E:\Emby-Server\system\ffmpeg.exe -loglevel +timing -y -print_graphs_file "E:\Emby-Server\programdata\logs\recording-conversiongraph.txt" -copyts -start_at_zero -fflags +genpts -analyzeduration 3000000 -f hls -stream_loop -1 -c:v:0 h264 -noautorotate -user_agent "VLC/3.0.0-git LibVLC/3.0.0-git" -multiple_requests 1 -reconnect 1 -reconnect_on_network_error 1 -reconnect_on_http_error 5xx,4xx -reconnect_streamed 1 -reconnect_delay_max 8 -i "https://somestream.m3u8" -map 0:0 -map 0:1 -sn -c:v:0 copy -c:a:0 copy -disposition:a:0 default -avoid_negative_ts disabled -map_metadata -1 -map_chapters -1 "E:\Emby-Server\recordings\someshow.ts"
 

--emby runs ffmprob and gets this for the HLS example--

{"Protocol":"Http","Id":"5ae2670600c435a11b148611727f3a51","Path":"https://somestream.m3u8","Type":"Default","Container":"hls","Size":257,"IsRemote":true,"HasMixedProtocols":false,"ContainerStartTimeTicks":132386319560,"SupportsTranscoding":true,"SupportsDirectStream":false,"SupportsDirectPlay":false,"IsInfiniteStream":true,"RequiresOpening":true,"RequiresClosing":true,"LiveStreamId":"somestreamid","RequiresLooping":true,"SupportsProbing":false,"MediaStreams":[{"Codec":"h264","ColorTransfer":"bt709","ColorPrimaries":"bt709","ColorSpace":"bt709","StreamStartTimeTicks":132386573670,"TimeBase":"1/90000","VideoRange":"SDR","DisplayTitle":"720p H264","NalLengthSize":"0","IsInterlaced":false,"BitRate":8000000,"BitDepth":8,"RefFrames":1,"IsDefault":false,"IsForced":false,"IsHearingImpaired":false,"Height":720,"Width":1280,"AverageFrameRate":59.94006,"RealFrameRate":59.94006,"Profile":"Main","Type":"Video","AspectRatio":"16:9","Index":0,"IsExternal":false,"IsTextSubtitleStream":false,"SupportsExternalStream":false,"Protocol":"File","PixelFormat":"yuv420p","Level":40,"IsAnamorphic":false,"ExtendedVideoType":"None","ExtendedVideoSubType":"None","ExtendedVideoSubTypeDescription":"None"},{"Codec":"aac","StreamStartTimeTicks":132386319560,"TimeBase":"1/90000","DisplayTitle":"AAC stereo","IsInterlaced":false,"ChannelLayout":"stereo","BitRate":192000,"Channels":2,"SampleRate":48000,"IsDefault":false,"IsForced":false,"IsHearingImpaired":false,"Profile":"HE-AAC","Type":"Audio","Index":1,"IsExternal":false,"IsTextSubtitleStream":false,"SupportsExternalStream":false,"Protocol":"File","ExtendedVideoType":"None","ExtendedVideoSubType":"None","ExtendedVideoSubTypeDescription":"None"},{"Codec":"eia_608","StreamStartTimeTicks":132386573670,"TimeBase":"1/90000","Title":"Closed Captions 1","DisplayTitle":"(Default EIA_608)","IsInterlaced":false,"IsDefault":true,"IsForced":false,"IsHearingImpaired":false,"Type":"Subtitle","Index":100,"IsExternal":false,"IsTextSubtitleStream":true,"SupportsExternalStream":false,"Protocol":"File","ExtendedVideoType":"None","ExtendedVideoSubType":"None","ExtendedVideoSubTypeDescription":"None","SubtitleLocationType":"VideoSideData"}],"Formats":[],"Bitrate":8192000,"RequiredHttpHeaders":{"User-Agent":"VLC/3.0.0-git LibVLC/3.0.0-git","Referrer":"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/94.0.4606.61 Safari/537.36"},"AddApiKeyToDirectStreamUrl":false,"ReadAtNativeFramerate":false,"DefaultAudioStreamIndex":1}

--emby runs ffmprob and gets this for the MP4 example--

{"Protocol":"Http","Id":"cc2200acb4c26a296e3c424c8e69b90a","Path":"https://somestream.m3u8","Type":"Default","Container":"mp4","Size":78503,"IsRemote":true,"HasMixedProtocols":false,"RunTimeTicks":150000000,"ContainerStartTimeTicks":0,"SupportsTranscoding":true,"SupportsDirectStream":false,"SupportsDirectPlay":false,"IsInfiniteStream":false,"RequiresOpening":true,"RequiresClosing":true,"LiveStreamId":"somestreamid","RequiresLooping":true,"SupportsProbing":false,"MediaStreams":[{"Codec":"h264","CodecTag":"avc1","Language":"und","StreamStartTimeTicks":0,"TimeBase":"1/12800","VideoRange":"SDR","DisplayTitle":"1080p H264","NalLengthSize":"4","IsInterlaced":false,"BitRate":39024,"BitDepth":8,"RefFrames":1,"IsDefault":true,"IsForced":false,"IsHearingImpaired":false,"Height":1080,"Width":1920,"AverageFrameRate":25,"RealFrameRate":25,"Profile":"High","Type":"Video","AspectRatio":"16:9","Index":0,"IsExternal":false,"IsTextSubtitleStream":false,"SupportsExternalStream":false,"Protocol":"File","PixelFormat":"yuv420p","Level":40,"ExtendedVideoType":"None","ExtendedVideoSubType":"None","ExtendedVideoSubTypeDescription":"None"}],"Formats":[],"Bitrate":39024,"RequiredHttpHeaders":{"User-Agent":"VLC/3.0.0-git LibVLC/3.0.0-git","Referrer":"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/94.0.4606.61 Safari/537.36"},"AddApiKeyToDirectStreamUrl":false,"ReadAtNativeFramerate":false}

hoping this might help out the emby team which have far more expertise in these cmd line switches  

@Luke

[iiiJoe 62]

18 hours ago, joekingcool said:

i started tinkering with the monitor recently and founds some more info.

i found that it seems the STS (seek to start error) happens only on HLS streams. if the channel uses mp4 as the container instead of HLS. it has a self healing within about 30-45 seconds of the stream being blocked.  this might explain previous discussions about changing the cmd that emby uses to record the stream. it might have been helping but only for the MP4 streams not the HLS streams. 

--Example HLS stream CMD emby uses--

E:\Emby-Server\system\ffmpeg.exe -loglevel +timing -y -print_graphs_file "E:\Emby-Server\programdata\logs\recording-conversiongraph.txt" -copyts -start_at_zero -f mp4 -stream_loop -1 -c:v:0 h264 -noautorotate -user_agent "VLC/3.0.0-git LibVLC/3.0.0-git" -multiple_requests 1 -reconnect 1 -reconnect_on_network_error 1 -reconnect_on_http_error 5xx,4xx -reconnect_streamed 1 -reconnect_delay_max 8 -i "https://somestream.m3u8" -map 0:0 -an -sn -c:v:0 copy -avoid_negative_ts disabled -map_metadata -1 -map_chapters -1 "E:\Emby-Server\recordings\someshow.ts"

--Example MP4 stream CMD emby uses--

E:\Emby-Server\system\ffmpeg.exe -loglevel +timing -y -print_graphs_file "E:\Emby-Server\programdata\logs\recording-conversiongraph.txt" -copyts -start_at_zero -fflags +genpts -analyzeduration 3000000 -f hls -stream_loop -1 -c:v:0 h264 -noautorotate -user_agent "VLC/3.0.0-git LibVLC/3.0.0-git" -multiple_requests 1 -reconnect 1 -reconnect_on_network_error 1 -reconnect_on_http_error 5xx,4xx -reconnect_streamed 1 -reconnect_delay_max 8 -i "https://somestream.m3u8" -map 0:0 -map 0:1 -sn -c:v:0 copy -c:a:0 copy -disposition:a:0 default -avoid_negative_ts disabled -map_metadata -1 -map_chapters -1 "E:\Emby-Server\recordings\someshow.ts"
 

--emby runs ffmprob and gets this for the HLS example--

{"Protocol":"Http","Id":"5ae2670600c435a11b148611727f3a51","Path":"https://somestream.m3u8","Type":"Default","Container":"hls","Size":257,"IsRemote":true,"HasMixedProtocols":false,"ContainerStartTimeTicks":132386319560,"SupportsTranscoding":true,"SupportsDirectStream":false,"SupportsDirectPlay":false,"IsInfiniteStream":true,"RequiresOpening":true,"RequiresClosing":true,"LiveStreamId":"somestreamid","RequiresLooping":true,"SupportsProbing":false,"MediaStreams":[{"Codec":"h264","ColorTransfer":"bt709","ColorPrimaries":"bt709","ColorSpace":"bt709","StreamStartTimeTicks":132386573670,"TimeBase":"1/90000","VideoRange":"SDR","DisplayTitle":"720p H264","NalLengthSize":"0","IsInterlaced":false,"BitRate":8000000,"BitDepth":8,"RefFrames":1,"IsDefault":false,"IsForced":false,"IsHearingImpaired":false,"Height":720,"Width":1280,"AverageFrameRate":59.94006,"RealFrameRate":59.94006,"Profile":"Main","Type":"Video","AspectRatio":"16:9","Index":0,"IsExternal":false,"IsTextSubtitleStream":false,"SupportsExternalStream":false,"Protocol":"File","PixelFormat":"yuv420p","Level":40,"IsAnamorphic":false,"ExtendedVideoType":"None","ExtendedVideoSubType":"None","ExtendedVideoSubTypeDescription":"None"},{"Codec":"aac","StreamStartTimeTicks":132386319560,"TimeBase":"1/90000","DisplayTitle":"AAC stereo","IsInterlaced":false,"ChannelLayout":"stereo","BitRate":192000,"Channels":2,"SampleRate":48000,"IsDefault":false,"IsForced":false,"IsHearingImpaired":false,"Profile":"HE-AAC","Type":"Audio","Index":1,"IsExternal":false,"IsTextSubtitleStream":false,"SupportsExternalStream":false,"Protocol":"File","ExtendedVideoType":"None","ExtendedVideoSubType":"None","ExtendedVideoSubTypeDescription":"None"},{"Codec":"eia_608","StreamStartTimeTicks":132386573670,"TimeBase":"1/90000","Title":"Closed Captions 1","DisplayTitle":"(Default EIA_608)","IsInterlaced":false,"IsDefault":true,"IsForced":false,"IsHearingImpaired":false,"Type":"Subtitle","Index":100,"IsExternal":false,"IsTextSubtitleStream":true,"SupportsExternalStream":false,"Protocol":"File","ExtendedVideoType":"None","ExtendedVideoSubType":"None","ExtendedVideoSubTypeDescription":"None","SubtitleLocationType":"VideoSideData"}],"Formats":[],"Bitrate":8192000,"RequiredHttpHeaders":{"User-Agent":"VLC/3.0.0-git LibVLC/3.0.0-git","Referrer":"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/94.0.4606.61 Safari/537.36"},"AddApiKeyToDirectStreamUrl":false,"ReadAtNativeFramerate":false,"DefaultAudioStreamIndex":1}

--emby runs ffmprob and gets this for the MP4 example--

{"Protocol":"Http","Id":"cc2200acb4c26a296e3c424c8e69b90a","Path":"https://somestream.m3u8","Type":"Default","Container":"mp4","Size":78503,"IsRemote":true,"HasMixedProtocols":false,"RunTimeTicks":150000000,"ContainerStartTimeTicks":0,"SupportsTranscoding":true,"SupportsDirectStream":false,"SupportsDirectPlay":false,"IsInfiniteStream":false,"RequiresOpening":true,"RequiresClosing":true,"LiveStreamId":"somestreamid","RequiresLooping":true,"SupportsProbing":false,"MediaStreams":[{"Codec":"h264","CodecTag":"avc1","Language":"und","StreamStartTimeTicks":0,"TimeBase":"1/12800","VideoRange":"SDR","DisplayTitle":"1080p H264","NalLengthSize":"4","IsInterlaced":false,"BitRate":39024,"BitDepth":8,"RefFrames":1,"IsDefault":true,"IsForced":false,"IsHearingImpaired":false,"Height":1080,"Width":1920,"AverageFrameRate":25,"RealFrameRate":25,"Profile":"High","Type":"Video","AspectRatio":"16:9","Index":0,"IsExternal":false,"IsTextSubtitleStream":false,"SupportsExternalStream":false,"Protocol":"File","PixelFormat":"yuv420p","Level":40,"ExtendedVideoType":"None","ExtendedVideoSubType":"None","ExtendedVideoSubTypeDescription":"None"}],"Formats":[],"Bitrate":39024,"RequiredHttpHeaders":{"User-Agent":"VLC/3.0.0-git LibVLC/3.0.0-git","Referrer":"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/94.0.4606.61 Safari/537.36"},"AddApiKeyToDirectStreamUrl":false,"ReadAtNativeFramerate":false}

hoping this might help out the emby team which have far more expertise in these cmd line switches  

@Luke

@joekingcooloutstanding research! So, what about making the container user adjustable? @Lukepossible?

[Ugnaughts 53]

ive been fine tuning the app. got rid of few memory leaks. still have a small one that im still working on. i added 2 variables in the config file. 1 for time to recheck logs 2 time to delay before emby starts back up. just so we can tinker with what works best for us.

also im removing the msc error portion for now.  ive rarely had an example of this, and when i did it never saved a recording. so i figured why have it restart emby and possible interrupt other recordings. 

but if there is other common errors or warnings that the recording is not going to be very good or most likly fail. then i can add those errors and have it shutdown emby and wait so many seconds then start back up and it will get a different streaming server from what ive witnessed. usually a restart of emby is all it takes for it to get a new server for the recording stream to get better, except for the sts error because it tells emby its completed. but all other issues with streams im hoping can be resolved with a restart of emby. 

if you have some examples of recordings that got messed up. like skipping allot, replaying same clip over and over, or stopping early, or not starting on time. just group them by topic, and no more than few examples of each for now.  ill try to implement so we can test on those issues as well.

for example i was getting "maybe the m3u8 list sequence have been wraped." then later "HTTP error 500 Internal Server Error" but after i restarted emby it got a new stream from different server and didnt get those (what i call errors) . although some times with those errors it will continue the record, but not sure of the quality of that recording. but sometimes it just stops. 

also i found out that the majority of streams are HLS. im not sure if we should be even tring to force ffmpeg to keep the stream any more than what it does. but maybe switch focus to having emby pick up on these failing streams, then stop that ffmpeg stream, then try again. it should be the same as restarting emby and get a new server stream.  I'm assuming it will be the same, maybe @luke can correct me?

[iiiJoe 62]

13 hours ago, joekingcool said:

maybe the m3u8 list sequence have been wraped." then later "HTTP error 500 Internal Server Error"

I’ve gotten this error as well, usually results in stopped recordings. 

 

13 hours ago, joekingcool said:

2 time to delay before emby starts back up. just so we can tinker with what works best for us.

Just wondering what the current delay is for restart? When the monitor program reacts to an error I’ve found it usually takes about 45 seconds from shutdown to recording restart, as long as the stream is stable enough to start recording again. I actually had one tonight that had to restart 3 different times starting about 15 min in. I still got 57 min of a 60 min program. Huge improvement over pre-monitor. 

[Ugnaughts 53]

it can vary on the time on the old monitor because . it checks the logs every 15 seconds and depending on how many logs it has to go threw at the time. i changed this to every 5 seconds to check the log and have it move the un-necessary logs to backup folder prior to searching the logs. also i went threw all the code and adjusted any 1-5 second pause for certain cmds to fully complete. then just left one variable to the user for when its done to delay starting back up emby. mostly because like you pointed out the stream might just be offline on all servers and needs more time before they are adjusted by the provider. im not sure if this will help or not. it might end up as a variable as well. we'll have to test it out.

but to answer question, it is about that time maybe quicker now, but you can now extend that time. 

also i found that when the monitor recreated the timer.json file it was creating extra data. this could have been causing possible issues down the roar. although that is fixed mostly. there is an occasion when the description of a show uses special characters it adds a couple more special characters. but this is contained within the comments section of that show. so it shouldn't effect any functionality. at it resolves its elf each time the guide is refreshed anyway. 

i fixed when it shuts down emby that it makes sure there is no rogue ffmpeg process as well, that could be left running and when it starts back up recording that show again it seems to mess that show up. 

the monitor itself uses very little memory, but it does gradually increase over time. i changed it so that when ever it finds something to fix, it reloaded the app and then memory is at its lowest. but if its a long time for you have an error to fix then this could possibly grow. but not sure if it will grow to a point that is noticeable. although it still bothers me, so im still looking for why. 

maybe pm me when you get failed recordings that are not fixed by the monitor. so i can see if there is other things we can trouble shoot. 

 

[Ugnaughts 53]

here is all apps updated.

the monitor now checks for 3 errors. then verifys that no matter what that recording status is set to inprogress so it trys again when emby starts back up.

1. "Media sequence changed unexpectedly" = MSC

2. "maybe the m3u8 list sequence have been wraped" = SHBW

3. "Seek to start failed" = STS

the recording cleanup, now refresh's the guide after it starts back up

the config file has a few changes, so make sure you update your current config file.

let me know if these work or any fail

Recording_Monitor.exe Recording_Cleanup.exe Recording_Folder_Cleanup.exe Recording_Joiner.exe config.txt

[iiiJoe 62]

10 hours ago, joekingcool said:

here is all apps updated.

the monitor now checks for 3 errors. then verifys that no matter what that recording status is set to inprogress so it trys again when emby starts back up.

1. "Media sequence changed unexpectedly" = MSC

2. "maybe the m3u8 list sequence have been wraped" = SHBW

3. "Seek to start failed" = STS

the recording cleanup, now refresh's the guide after it starts back up

the config file has a few changes, so make sure you update your current config file.

let me know if these work or any fail

Recording_Monitor.exe 2.91 MB · 0 downloads Recording_Cleanup.exe 2.9 MB · 0 downloads Recording_Folder_Cleanup.exe 2.89 MB · 0 downloads Recording_Joiner.exe 2.89 MB · 0 downloads config.txt 301 B · 0 downloads

Awesome! I’ll try to install today

[Ugnaughts 53]

i just added one other feature this evening. 

send an email each time it finds an error that its working on. send the name of show and error. you would need a gmail email for notifications. and generate an app password. then put in config file. to make app password https://myaccount.google.com/u/1/apppasswords

 

if it seems like its not fixing the other 2 errors, but more of an interruption, then i can just remove them, and get you one with only the sts error.

 

Recording_Monitor.exe config.txt

[Nonnos 2]

@joekingcoollooks like you done a great job! Unfortunately not able to teat since running my server on a linux machine.

Hoping that the emby team take’s your analysis and improve emby based on your findings. @Luke ?

 

[Ugnaughts 53]

if i could find a program language like ahk that was universal between windows linux and mac. and fairly basic language like ahk. i would be willing to learn it maybe to make apps.  i have tried a little with python, java, but a little tough on the users getting dependency's sometimes. also the ahk community is awesome, they are always there to help when i get stuck on some function. not sure if you could use wine in linux with my app? maybe figure a way with dockers or containers? 

 

[Ugnaughts 53]

well i found that the joiner app has been on occasion running while recordings were happening. i had about 6 lines of code to check before the joiner started. but now i have about 130 lines of code to check before the joiner starts. and i think i covered all possibilities(hopefully). now there shouldn't be any extra long recordings or repeated clips at the end of recordings. i changed the versions of both apps effected to 2.01. if you dont want to use the mail notifications just put 0 and if you want it on put 1. also embys scheduled recordings are in universal time. for example i live east coast so i have to subtract 4 hours from the time they have, so i put -4 in the config file. 

so the joiner currently filters all scheduled recordings. makes list of inprogress and new. it verfies they are not within the next 15 minutes. then it continues to check very 30 seconds for any recordings in progress for 6 minutes. then it joins the files. it only takes a few minutes to join the files. and only takes about 6 minutes to start the joining after all recordings and streams have stopped. 

hopefully this is the last thing i find for while, Good Luck

Recording_Joiner.exe config.txt Recording_Monitor.exe

[Ugnaughts 53]

i updated all apps again.

(Joiner) i noticed that the joiner seems to always find a way of running when recordings are going on. so i removed it from auto starting from the monitor app for now. the joiner works perfectly fine. so you just need to verify that no recordings are currently going on or will start in next 5 mins or so for it to complete. i did update the amount of files it can join from 4 clips to 8 clips of a movie. 

(recording cleanup) refresh's the guide automatically afterwards. which is needed for the series timers to be added back in as well.

(recording folder cleanup) it deletes all metadata in recording folder. then deletes all empty folders in recording folder. and now if you add the parent id of your recording folder to the config. it will automatically refresh meta data for the recording folder only. 

(monitor) this i redid from the ground up. its much faster, but also depends on the size of your recordings json file. if it hasn't been cleaned up in while, then it has allot more to search threw to fix the status. also found that if you get the 501 error // maybe the m3u8 list sequence have been wraped. nothing fixes it. it will get a different server but still gets the same bad stream until it says offline. so i will ignore that error for now. the Seek to start failed error, seems to work flawlessly.  you might want to set a delay of 20 seconds or set to 1. ive had some that were fixed in one quick fix. and others that did 2-3 quick fixes. onced there merged together it dont seem noticeable.

another big change to the monitor is. it finds and fixes the issue, then shuts down, replaces timers.json file, then starts back up. so that means if you have people watching emby when it restarts they might not be interrupted. because its a quick reboot of emby. but you can extend that time before it restarts emby. but the restart time of emby is the same. hopefully that makes sense. 

 

 

config.txt 4.Recording_Monitor.exe 1.Recording_Cleanup.exe 2.Recording_Folder_Cleanup.exe 3.Recording_Joiner.exe